747 results about "Speech generation" patented technology

The invention provides an intelligent real-time interactive question-answering system based on a virtual digital human, and belongs to the technical field of voice signal processing and voice recognition, and the system comprises a data acquisition module which receives a voice or text interaction request input by a user, collects the expression dynamic parameter sequence and limb movement sequence data of the user in real time, and transmits the data to a user interaction module; obtaining a standardized voice feature vector and structured text data; the cross-modal fusion module is used for constructing an interactive feature matrix; the behavior decision module outputs a decision instruction set; the knowledge retrieval module is used for generating an answer text with emotional adaptability and voice features; and the voice generation module is used for generating a mouth shape animation key frame, a micro expression parameter sequence and a limb action track of the virtual digital human, generating a voice response in combination with the answer text and the voice characteristics, and pushing the voice response to the user terminal. According to the method, the interaction experience and adaptability of the virtual digital human are remarkably improved.

The invention discloses a digital human real-time interaction system and method. The interaction system comprises terminal equipment and a cloud server. The terminal device receives voice input by a user and transmits the voice to the cloud server. The cloud server comprises a video acquisition and processing module, an intelligent question and answer and voice generation module, an audio processing module and an image rendering module. A video capture and processing module captures original video data indicative of an object of a digital figure and generates successive image frames. The intelligent question and answer and voice generation module converts the voice into a text; performing semantic analysis on the text, understanding the intention of the user, and generating a corresponding reply text; the reply text is also converted into a reply voice. The audio processing module processes the reply voice and the continuous image frames to generate first video data. And the image rendering module fuses the first video data with the virtual background, and performs real-time processing on the fused video to generate a response video. And the terminal equipment outputs the response video.

Disclosed systems and methods convert user-supplied textual content into animated videos. Textual input is received and processed to identify narrative elements such as characters, settings, and events. These elements are then transformed into visual scene components. Still images generated based on these components are subsequently animated in line with the narrative context. The system can automatically implement storytelling techniques adapted to incorporate neuroscience principles. The system also synthesizes speech for dialogues or narrations using voice synthesis technology that considers emotional markers, tone, and pace. Generated media and metadata are stored in a data storage system that maintains data integrity and enables efficient retrieval. Users can interact with an export interface to choose video resolution, format, and sharing options. A feedback system employing machine learning algorithms collects and analyzes user feedback for real-time adjustments to the generated animated video.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of medical health, financial science and technology and the like, and discloses a voice generation method which comprises the following steps: constructing a multi-language voice synthesis model, obtaining plain text data and paired voice text data, and constructing an expansion vocabulary; updating a language perception embedding layer and model parameters, and converting an input text into a mark sequence; and the encoder extracts context semantic features, extracts pronunciation rule features, and the decoder fuses the features to generate an acoustic feature sequence, and converts the acoustic feature sequence into target voice data. According to the invention, the multi-language speech synthesis model is combined with the language perception embedding layer, so that the speech generation capability of a low-resource language is improved; the text conversion accuracy is improved by expanding the vocabulary, the target language learning ability is enhanced by unsupervised training, the low data environment adaptability is optimized by supervised training, and the speech naturalness and fluency are improved by feature fusion.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of medical health, financial science and technology, cultural transmission and the like, and discloses a voice generation method based on multi-modal fusion, which comprises the following steps: collecting audio data to extract timbre features, and training a field feature timbre generation model; analyzing text semantic recognition emotion information, adjusting speech synthesis parameters, combining personalized information to construct a parameter mapping table, fusing to generate a synthesis control parameter sequence, aligning the synthesis control parameter sequence with character labels, visual elements and background music data, driving a domain feature timbre generation model, and generating synthesis data of synchronous speech, text, vision and music. Domain timbres are generated through timbre feature training, speech expression is optimized by combining semantic analysis and emotion recognition, user requirements are matched based on personalized information, and time alignment is performed by fusing text, vision and music data, so that the synthesized speech has domain features, emotion adaptability and personalization, and the speech quality is improved. And the voice immersion and the information transmission capability are improved.

Methods, systems, and computer programs are presented for audio-visual speech generation with multi-modal latent diffusion models. One method includes encoding raw audio signals and video frames into respective latent spaces using audio and visual autoencoders. A text transcript is processed into phoneme sequences using a text transcript processor. The audio and visual latent spaces are conditioned using the text transcript and a conditioning variable. Joint distributions of the visual and audio latent spaces, text transcript, and conditioning variable are learned using a multi-modal latent diffusion model. The model adds noise to the latent audio-visual representations and predicts the noise through denoising neural networks. An inverted diffusion process is utilized to generate diverse speech content and speaker characteristics, resulting in realistic audio-visual speech. The technology presented provides a novel approach to conditional speech generation with potential applications in speech synthesis, voice conversion, and speech recognition.

The invention relates to the technical field of speech synthesis, can be applied to business scenes such as financial science and technology, medical health and the like, and discloses a speech generation method, device, equipment and medium. And inputting a text speech language model to generate an intermediate code in combination with a speech code extracted based on a codebook generation mode, extracting a speaker feature vector in the prompt speech, decoding the intermediate code and the speaker feature vector by a generative adversarial decoder, and outputting a target speech. According to the method, fine-grained text representation is established by fusing character semantics and pinyin pronunciation information, and voice cloning is completed through unified voice language modeling and an adversarial generation mechanism in combination with codebook-driven acoustic coding and speaker personality characteristics, so that the naturalness, similarity and pronunciation accuracy of generated voices are improved.

The invention relates to the technical field of voice processing, can be applied to business scenes in the fields of medical health, financial science and technology, barrier-free service and the like, and discloses a text-to-voice method based on knowledge distillation, which comprises the following steps: carrying out standardization processing on an input text to generate a standard text sequence; the lightweight text encoder encodes the standard text sequence to generate a text implicit vector; the non-autoregressive acoustic feature prediction module maps the text implicit vector into a student acoustic feature sequence, and calculates alignment loss through knowledge distillation; and performing structured pruning and parameter quantization based on alignment loss, generating an acoustic feature sequence by the optimized model, and converting the acoustic feature sequence into a voice waveform by a vocoder. According to the method, through knowledge distillation, pruning optimization and parameter quantification, the reasoning speed and the cross-equipment adaptability are improved while the model size and the calculation requirement are reduced, so that the TTS system can realize efficient, low-delay and low-power-consumption voice generation in a resource-constrained environment.

The present application provides training and speech generation methods and apparatuses for a speech generation model, an electronic device, a computer-readable storage medium, and a computer program product. The method comprises: obtaining a first speech generation model; obtaining sample data of a plurality of modalities; on the basis of a prompt image sequence and speech text, respectively calling a plurality of encoders to perform encoding, so as to obtain a multi-modal encoding vector sequence; on the basis of the multi-modal encoding vector sequence, calling a decoder to perform decoding, so as to obtain decoded text; determining a probability distribution for the decoded text and the sample data of the plurality of modalities, and determining a target loss on the basis of the probability distribution; and on the basis of the target loss, updating parameters of the decoder and at least one of the encoders, wherein the updated decoder and the plurality of updated encoders are configured to form a second speech generation model, and the second speech generation model is used to generate target speech text corresponding to a prompt image sequence of a target object.

Disclosed are systems, methods and other implementations for speech generation, including a method that includes obtaining a speech sample for a target speaker, processing, using a trained encoder, the speech sample to produce a parametric representation of the speech sample for the target speaker, receiving configuration data for a speech synthesis system that accepts as an input the parametric representation, and adapting the configuration data for the speech synthesis system according to an input comprising the parametric representation, and a time-domain representation for the speech sample, to generate adapted configuration data for the speech synthesis system. The method further includes causing configuration of the speech synthesis system according to the adapted configuration data, with the speech synthesis system being implemented to generate synthesized speech output data with estimated voice and time-domain speech characteristics approximating actual voice and time-domain speech characteristics for the target speaker.

This disclosure describes a speech-to-video system that automatically generates enhanced short-form videos from speech. For example, the speech-to-video system utilizes different speech processing models to analyze speech in audio input and determine contextual features. Additionally, the speech-to-video system utilizes various video generation models to create enhanced short-form videos using text summaries, audio contexts, user information, video parameter inputs, and / or other user contexts. In some cases, the speech-to-video system leverages components of a mobile core network to efficiently generate and deliver these features to mobile devices.

The invention relates to the technical field of voice processing, can be applied to business scenes such as financial science and technology and medical health, and discloses an environmental acoustics simulation voice generation method, device and equipment and a medium, and the method comprises the steps: obtaining and separating mixed voice data, and generating original voice content and original environmental acoustics information; converting the original voice content into first text information; determining a target environment acoustic tag in combination with the original environment acoustic tag, the first text information and the target geographical location information; acquiring target environmental acoustic information from a preset sound data set based on the tag, and adjusting the amplitude characteristic of the target environmental acoustic information to match the original environmental acoustic information; and synthesizing the adjusted target environment acoustic information and the original voice content into simulated voice data. According to the method, the target geographic position information is introduced to participate in acoustic feature determination and amplitude adjustment, so that the generated simulated voice data is more consistent in geographic semantics and acoustic performance, and the authenticity and concealment of voice environment disguise are effectively improved.

The invention relates to the technical field of speech synthesis, can be applied to business scenes of medical health, financial science and technology and the like, and discloses a hierarchical emotional speech generation method, which comprises the following steps: acquiring an input text and an emotional speech sample, extracting a text embedding feature from the input text, extracting a Mel spectrum feature from the emotional speech sample, and obtaining a text embedding feature; the method comprises the following steps: extracting phoneme-level, word-level and statement-level sentiment distribution characteristics through a hierarchical sentiment distribution extraction module, carrying out time dimension alignment, generating a multi-level sentiment guidance matrix, and inputting the multi-level sentiment guidance matrix and text embedding characteristics into a sentiment synthesis module to generate a target Mel spectrum; and finally, converting the target Mel spectrum into target voice through a vocoder. The emotion control is expanded from the statement level to the fine-grained level of phonemes, words and the like, and the multi-level emotion guidance matrix is combined for generation, so that the generated speech is finer and more abundant in emotion expression and conforms to the context, and the naturalness of speech generation and the accuracy of emotion transmission are improved.

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a voice generation method, which comprises the following steps of: acquiring a to-be-converted text and a reference voice sample, extracting voice features of reference voice by using a pre-trained self-supervised learning encoder, converting the to-be-converted text into voice features through a text-to-voice feature model, and generating a voice feature. And analyzing the similarity of the two groups of features, determining the voice features of the target speaker from the two groups of features, fusing the two groups of features to generate final fused voice features, and decoding by using a pre-trained vocoder to generate voice signals. According to the method, feature matching and fusion are carried out under the small sample or zero sample condition, the voice similar to the target voice can be generated without a large amount of voice data, the voice synthesis efficiency and quality are effectively improved, the multi-scene application requirement is met, and the method is particularly suitable for voice generation of unseen speakers.

The invention relates to the technical field of data processing, and particularly discloses a digital human voice generation system based on a large language model, which is used for solving the existing problems and comprises a semantic analysis module, a voice generation module, a multi-mode synchronization module, a data transmission module and a user feedback module, the semantic analysis module performs semantic analysis and sentiment analysis on voice and text data input by a user by using a large language model, extracts voice information and sentiment metadata for describing context, and dynamically constructs a semantic structure and a sentiment label adaptive to an education scene according to input content based on a metadata automatic derivation technology; through the dynamic time warping algorithm and the educational knowledge graph, semantic and emotion changes are accurately captured, the speech speed and tone are flexibly adjusted, and the naturalness of digital human speech generation and the adaptability of educational scenes are improved.

The invention discloses a traffic domain service voice adaptive generation method and system based on a thinking chain fine-tuning large model, and the method comprises the steps: converting an input voice signal into a high-dimensional voice feature signal through a voice encoder, and generating a text output signal according to the high-dimensional voice feature signal through a text decoder and a pinyin decoder; according to the invention, a variational de-quantization joint data enhancement mechanism, a dual-channel multi-mode discriminator architecture and a layered composite loss function are adopted to carry out high-fidelity emotional speech generation and robustness synthesis in a complex noise environment, and semantic-driven dynamic rhythm optimization and professional term precise pronunciation are synchronously supported; in an application scene in the traffic field, a multi-task speech recognition method can be adopted to realize efficient linkage of character-level recognition, audio-to-pinyin conversion and sentence-level accent classification modules, so that complex accent, more noise and traffic term polyphone challenges are effectively handled, and the method is suitable for being widely popularized and used.

The invention belongs to the technical field of three-dimensional reconstruction, and discloses a three-dimensional digital human generation method and system capable of voice interaction. According to the invention, brand new speaking audios in different languages are automatically generated according to different languages of the input target text and the sampled human voice audios; the sequential stability and detail reduction capability of three-dimensional human motion are guaranteed by using multi-model joint estimation and a sequential loss function, and facial expression details and hand postures in the image can be accurately estimated. After the high-precision three-dimensional human body model is obtained through estimation, human body action and expression generation is carried out based on voice driving, accurate synchronization of actions and expressions generated through voice is achieved, and facial expression movement and body posture movement, namely a whole-body three-dimensional human body model, conforming to brand-new speaking audio are accurately generated; and finally, rendering the whole-body three-dimensional human body model into a real digital human capable of voice interaction by using a three-dimensional neural rendering model. According to the invention, the realization of single person picture input, high-precision three-dimensional digital person generation and voice interaction is facilitated.

The invention relates to the technical field of artificial intelligence, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses an emotional voice generation method, device, equipment and medium, and the method comprises the steps: obtaining a target text of a to-be-generated voice and a target emotional prompt text based on natural language description; inputting the target emotion prompt text into a pre-trained emotion encoder to obtain an emotion embedding vector; inputting the target text into a pre-trained text encoder to obtain a semantic embedding vector; inputting the emotion embedding vector and the semantic embedding vector into a pre-trained joint speech generation model, performing fused speech feature prediction on the emotion embedding vector and the semantic embedding vector, and generating a speech feature with an expected emotion; and decoding the voice features to generate target emotional voice corresponding to the target text. The speech emotion is controlled through the emotion prompt text described by the natural language, and the flexibility and synthesis effect of emotion speech generation interaction are improved.

The invention relates to the technical field of speech synthesis, can be applied to service system platforms of medical health, financial science and technology and the like, and discloses a speech generation method, device, equipment and medium based on rhythm characteristics, and the method comprises the steps: obtaining an input text and a corresponding reference speech, extracting a Mel spectrum, and generating speech representation; extracting overall rhythm features through a multi-head attention mechanism, combining with a text embedding feature input condition flow matching module, generating intermediate rhythm features, and executing threshold adjustment processing to obtain a fine rhythm feature set; and fusing the fine rhythm feature set with the text embedded features to generate an acoustic parameter sequence, and converting the acoustic parameter sequence into a voice waveform through a pre-trained neural vocoder. According to the method, rhythm feature generation is optimized through condition flow matching, the flexibility of rhythm expression is enhanced in combination with threshold adjustment, and the matching degree of the text and the rhythm is improved through cross-modal attention fusion, so that the generated voice is more natural, diversified and controllable.

The invention relates to the technical field of artificial intelligence, and discloses a speech synthesis method and device, equipment and a medium, and the method comprises the steps: obtaining a to-be-synthesized text and a reference audio; dividing the to-be-synthesized text into a plurality of text segments by adopting a sliding window algorithm; generating an audio waveform corresponding to each text segment based on the reference audio and a pre-trained speech synthesis model; and performing speech synthesis on the audio waveform corresponding to each text segment to obtain a target speech. Through the mode, the input text is segmented by adopting the sliding window algorithm to form a plurality of smaller and easier-to-manage text segments, and the plurality of text segments and the parameter audio are input into the speech synthesis model, so that the speech synthesis model generates the audio sentence by sentence based on the shorter text segments; therefore, the model can focus on accurate voice and intonation details of each segment, the accuracy and natural fluency of audio output are ensured, the degradation problem caused by the fact that the text length exceeds a typical length parameter is solved, and the stability of voice generation is improved.

The invention discloses a privacy information desensitization method and system for a voice generation type large model, and relates to the technical field of artificial intelligence. Input voice data is discretized, Gaussian noise disturbance sensitive features are injected, and a cross-modal voice generation model is constructed in combination with three-stage training; and meanwhile, a cross-modal privacy enhancement mechanism is applied to detect and fuzzify sensitive information in real time in an output stage. According to the invention, the adaptability of a large-scale voice generation model in a privacy protection scene is improved, and comprehensive protection of user privacy is realized. And moreover, the capability of extracting user privacy information by an adversarial attacker is effectively limited, and the risk of sensitive information leakage in the data transmission, storage and generation process of the voice generation model is reduced. On the premise that privacy is ensured, the voice generation model can still keep high-quality generation performance, generated voice output has high naturalness and accuracy, and actual application requirements are met.

Techniques for determining when speech is directed at another individual of a dialog, and storing a representation of such user-directed speech for use as context when processing subsequently-received system-directed speech are described. A system receives audio data and / or video data and determines therefrom that speech in the audio data is user-directed. Based on this, the system determine whether the speech is able to be used to perform an action by the system. If the speech is able to be used to perform an action, the system stores a natural language representation of the speech. Thereafter, when the system receives system-directed speech, the system generates a rewrite of a natural language representation of the system-directed speech based on the previously-received user-directed speech. The system then determines output data responsive to the system-directed speech using the rewritten natural language representation.

The embodiment of the invention discloses an improvement method of a speech synthesis system, electronic equipment and a storage medium, the method is used for a VALL-E model, and comprises the following steps: on the basis of the VALL-E model, combining an advanced speculation decoding strategy and a tolerance mechanism, and using a plurality of prediction headers to predict more marks, the advanced speculation decoding strategy comprises an effective lossless speculation decoding method based on an autoregression Transform model; and inference sampling is ensured to be consistent with the VALL-E model through a verification mechanism. According to the embodiment of the invention, the original VALL-E model is completely reserved on the whole, more tokens are predicted by using more prediction heads in combination with speculation sampling and a tolerance mechanism, and reasoning sampling is ensured to be consistent with the original model through a verification mechanism. In this way, the original model performance can be ensured, and the stability and the generation effect of the model are optimized by predicting farther tokens through multiple heads. Through the proposal of a'tolerance 'mechanism, more head prediction results are accepted, the acceleration effect is obviously enhanced, and meanwhile, the voice generation performance is improved.

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a multi-module collaborative voice generation method, which comprises the following steps: acquiring a reference text and reference voice, converting the reference text into a phoneme sequence and generating potential features, converting the reference voice into a Mel spectrogram and extracting style features, and extracting fundamental frequency information. A phoneme sequence and a Mel spectrogram are input into a text alignment module to generate alignment information, voice duration is predicted based on the alignment information and potential features, fundamental frequency of voice is predicted through fundamental frequency information, the predicted duration, fundamental frequency and style features are input into a decoder to generate a reconstructed Mel spectrogram, and finally the reconstructed Mel spectrogram is converted into voice audio data through a vocoder. According to the method, the generated voice can better reflect the natural rhythm and tone change of the text content by combining accurate control on the voice duration, the fundamental frequency and the style characteristics, and the rhythm and expressive force of the voice are enhanced, so that the naturalness and the fidelity of voice synthesis are improved.

The present disclosure provides a speech synthesis model training method. The method can comprise: acquiring initial training data; selecting a plurality of first text segments, second text segments, and third text segments from coherent text, and acquiring first audio segments, second audio segments, and third audio segments from coherent audio; detecting whether the plurality of audio segments and text segments meet a stitching condition; on the basis of the text sequence, stitching, by alternatively arranging text and audio, the plurality of text segments and audio segments meeting the stitching condition to generate combined training data, so as to generate a combined training data set; on the basis of the combined training data set, training the initial speech generation model to obtain a trained speech synthesis model, wherein during the training, prosodic, tonal, and / or emotional features from the second text segments and / or the third text segments are extracted. Further disclosed is a speech synthesis method. The present disclosure allows for use of the context of the text and audio to realize text comprehension to capture prosody, tonality, and emotion in the context.

The invention relates to the technical field of robot control, and discloses a spiking neural network-based robot control method, which comprises the following steps of: preprocessing collected multi-modal data to obtain an emotion pulse signal; inputting the emotion pulse signal into a pre-constructed emotion pulse neural network, and outputting a comprehensive emotion pulse; inputting the comprehensive emotion pulse into a central pattern generator, and outputting a behavior rhythm; acquiring an environment feedback signal generated by executing the behavior rhythm, and adjusting a connection weight according to the environment feedback signal; optimizing the comprehensive emotion pulse according to the connection weight, and converting the optimized comprehensive emotion pulse into emotion interaction voice information; and adjusting the emotion intensity according to the optimized comprehensive emotion pulse and the multi-modal data. According to the method, data are collected through the multi-mode sensor to generate emotion pulses, the emotion pulses are input into the central mode generator to generate behavior rhythms after SNN processing, weight optimization, emotional speech generation and emotional steady-state control setting are combined with the STDP algorithm, and the efficiency of a robot service scene is improved.

The invention belongs to the technical field of artificial intelligence, and particularly relates to a voice generation method, a voice generation device, a computer readable medium, electronic equipment and a computer program product. The method comprises the following steps: acquiring a natural language instruction, wherein the natural language instruction is used for describing a presentation effect of voice in a natural language; a sample pair having a similar voice presentation effect with the natural language instruction is obtained, the sample pair comprises an audio sample and a text sample, and the text sample is used for describing the voice presentation effect of the audio sample; combining the natural language instruction with the sample pair to obtain a multi-modal cue word; and generating voice according to the multi-mode prompt word. According to the invention, the control precision and flexibility of voice generation can be improved.

Systems and techniques described herein relate to a diffusion-based model for generating converted speech from a source speech based on target speech. For example, a device may extract first prosody data from input data and may generate a content embedding based on the input data. The device may extract second prosody data from target speech, generate a speaker embedding from the target speech, and generate a prosody embedding from the second prosody data. The device may generate, based on the first prosody data and the prosody embedding, converted prosody data. The device may then generate a converted spectrogram based on the converted prosody data, the speaker embedding, and the content embedding.

The invention relates to the technical field of smart home, and discloses a device control method, a storage medium and an electronic device.The method comprises the steps that in the process that a user sends out a voice instruction to expect to control a certain device, the user can send out gesture information used for determining a target device expected to be controlled through gesture operation within a preset period of time; therefore, the voice instruction is executed by the target device expected to be controlled. Specifically, the equipment control method comprises the following steps: after a voice instruction sent by a user is recognized, performing image recognition on image data in a preset time period in an environment based on voice generation time when the user sends the voice instruction, and obtaining gesture information used for determining target equipment expected to be controlled by the user through an image recognition result, and finally, outputting a control instruction through the gesture information and the voice instruction to control target equipment expected to be controlled by the user to execute the voice instruction sent by the user.

The invention relates to the field of artificial intelligence, and discloses a speech generation method and device based on a conditional flow matching model, computer equipment and a storage medium, and the method comprises the steps: obtaining a training text of a generated speech, and taking the speech as a training speech; sampling noise data, and inputting the training text, the training voice and the noise data into a conditional flow matching model for training; reasoning the conditional flow matching model by using a variable step size to obtain a vector field about noise and voice; and constructing a voice generation model based on the conditional flow matching model and the vector field, and performing reasoning prediction on a target text of a to-be-generated voice by using the voice generation model. According to the embodiment of the invention, the text of the generated voice is utilized to train the conditional flow matching model so as to construct the voice generation model, and in the training process of the conditional flow matching model, the variable step size is adopted for reasoning and solving, so that the voice can be quickly generated, and the voice generation quality can be improved.