2213 results about "Speech synthesis" patented technology

The invention relates to dialect speech synthesis, in particular to a semi-supervised dialect emotion speech synthesis system based on hybrid experts, which comprises a text analysis module for preprocessing an input dialect text and generating a text representation vector through feature extraction and feature fusion; the mixed expert module is used for acquiring dialect acoustic features, rhythm features, emotional features and general acoustic features; the dynamic routing module is used for realizing intelligent cooperation among experts through a task-aware soft routing algorithm; the semi-supervised learning module is used for training supervised learning by using the marked dialect emotion voice data and training self-supervised learning by using the unmarked dialect emotion voice data; the acoustic parameter generation module is used for integrating the output of each expert to generate a complete acoustic parameter set; the neural vocoder is used for converting the acoustic parameter set into final dialect emotion voice; the method can effectively overcome the defect that the dialect emotional voice is difficult to accurately synthesize under the condition of lack of sample resources.

The invention provides a multi-modal intelligent question-answering and recommendation system supporting emotional speech output, and relates to the technical field of visual question-answering, and the system specifically comprises a multi-modal question-answering module, a multi-modal recommendation module, a speech recognition module, and a speech synthesis module. Wherein the multi-mode question answering module converts an image input by a user into a natural language form, and final question answering is completed in combination with a comet knowledge base COMET; the voice recognition module is used for recognizing voice data input into the multi-mode intelligent question answering and recommending system and processing the voice data into a text form; the multi-modal recommendation module identifies whether modal missing exists in input data or not, generates representation of the missing modal, obtains vector representation of the user and the article through a graph convolutional neural network and learning according to an interaction matrix of the user and the article, carries out reliability calculation, and finally sends a multi-modal recommendation result to the user and the article. Sending back to the multi-mode question and answer module; and the voice synthesis module outputs the voice to the user.

An apparatus for end-to-end text-to-speech synthesis is provided. The apparatus comprises input interface circuitry configured to receive first input data indicative of a phoneme and second input data indicative of a first target duration for the phoneme. The apparatus further comprises processing circuitry configured to, using a trained machine-learning model, map the phoneme to a state using an encoder sub-model of the trained machine-learning model, estimate a second target duration for the phoneme based on the state and determine an attention weight based on the first target duration and the second target duration. The processing circuitry is further configured to map the state to audio data based on the attention weight using a decoder sub-model of the trained machine-learning model, wherein the audio data are indicative of an audio waveform representing speech.

The invention relates to the technical field of computer vision and human-computer interaction, and discloses an AI digital human intelligent interaction method, system and equipment based on multiple modalities, and the method comprises the steps: pre-awakening a digital human when a human face is detected, and further thoroughly awakening the digital human based on recognized preset voice information or preset gesture information; voice and video information of a user in the interaction process is obtained, a keyword extraction result, a gesture recognition result and an emotional state tag are generated, a pre-constructed knowledge base is utilized to retrieve related information, a big language generation model module is combined to generate an answer text, and the answer text is input into a preset voice synthesis model to generate emotional voice output. And based on the current emotional state label of the user, driving the digital human animation to be output in an emotional manner. According to the method and the system, the digital human for understanding the emotion of the user, generating personalized answers, providing voices with rich emotions and displaying natural expressions and actions can be created, better interaction with the user can be realized, and more humanized and effective services can be provided.

The invention discloses a construction method of a real-time streaming voice intelligent question and answer service system. The construction method comprises the following steps: receiving input voice data; calling an independent voice recognition service to convert the voice data into an input text; the method comprises the following steps: constructing cue words of a large language model based on an input text, initiating an LLM streaming request to an independent large language model service, and obtaining streaming text answers generated by LLM in real time; segmenting the flow type text answer in real time through a dynamic sentence segmentation device to generate a plurality of clauses; calling an independent speech synthesis service in parallel for each clause, and converting the text into an audio data block; and combining the audio data blocks into streaming audio data according to the generation sequence, and returning the streaming audio data to a client for playing in real time. According to the invention, by decoupling ASR, LLM and TTS services and combining the asynchronous streaming framework and the strategy of "wanting while speaking", the real-time performance and flexibility of the voice question-answering system are significantly improved.

The invention provides a video generation control method and a computer readable storage medium, and is applied to the technical field of computers. According to the method, the structural defect of a split script is solved through a large split model, the consistency of split image features is ensured through image generation based on reference image feature constraint, audio and picture synchronization is realized in combination with emotional speech synthesis and dynamic mouth shape alignment, the video fidelity is improved through a split video cue word guiding multi-subject generation algorithm, and the user experience is improved. Therefore, the technical problems of missing emotion logic, poor consistency of split image features, difficulty in audio synchronization, insufficient controllability of video generation and the like in film and television creation in the existing AIGC technology are effectively solved, and the efficiency and the quality of video content production are improved.

The invention relates to the technical field of data processing, and discloses a multi-modal digital human generation method and system based on AIGC and a storage medium. The method comprises the steps of performing modeling and dynamic weight adjustment processing on multi-modal feature data to obtain dynamic personalized knowledge data; performing intelligent matching processing on the target scene information and the dynamic personalized knowledge data to obtain situational behavior instruction data; performing hierarchical action sequence generation, adaptive resolution optimization and speech synthesis processing on the initial digital human to obtain multi-modal digital human performance data; and behavior deviation identification is carried out on the multi-modal digital human performance data to obtain behavior deviation data and performance feedback data, and adaptive optimization processing is carried out on the multi-modal digital human performance data and the performance feedback data to obtain a target digital human. According to the invention, the efficiency and accuracy of multi-modal digital human generation based on AIGC are improved.

The invention discloses a personalized digital human generation method based on a single video, and relates to the field of virtual digital human modeling and driving, and the method aims at the video and voice data of a target person, through introducing a multi-modal alignment constrained voice driving synchronization mechanism and combining semantic understanding and an emotion label expression generation model, a personalized digital human model is generated. High synchronization, nature and vividness of digital human facial expressions and voice contents are realized. According to the method, a self-supervised style consistency constraint is added in model training to ensure that the style of a generated character image is stable, and a 3D semantic mask is added in an image fusion stage to improve the fusion precision and realistic effect of a synthetic facial expression and a reference face. According to the method, the digital human can be rapidly cloned and driven to synthesize the expression through the voice only through a single video sample, the generation process is efficient, and the obtained digital human video has excellent sense of reality and interactivity.

The invention belongs to the technical field of vehicle state detection and early warning, and particularly relates to a vehicle state monitoring and early warning method and system.The monitoring and early warning method comprises the steps that a sensor obtains real-time data, an anomaly detection algorithm is applied, and an abnormal event is marked; calculating an information priority according to the abnormal severity and the driving scene, and distributing the information priority to a high-priority queue; multi-mode early warning is generated, and high-frequency sound and vibration are used during high-speed driving; extracting a voice prompt, and generating voice waveform data through a voice synthesis module; voice input of a driver is recognized, intention is analyzed, and abnormal information feedback is provided; according to the scene and the abnormal state, interactive output is optimized, and detailed information is displayed during low-speed congestion; dynamically adjusting the interface of the instrument panel, and amplifying the key area in case of abnormal severity; and integrating the image data and the voice data, generating a multi-mode signal, and transmitting the multi-mode signal after rendering processing. The vehicle abnormal information can be timely and accurately transmitted to a driver, and the driving safety is effectively improved.

An automated conversational multi-player gaming platform is described. A machine-learning model is trained to use a natural language processor and a speech synthesizer as an automated dealer communicating with players of a game of chance. The trained machine-learning model receives requests from people to become players of a game of chance, outputs private game content associated with the game of chance to the players' private user interfaces, and outputs public game content associated with the game of chance to the players' public user interface. The trained machine-learning model receives selections of game playing options and wager values associated with the game of chance from the players via their private user interfaces, and outputs information identifying one player as a winner of the game of chance and a credit of the winner's wager value to the players' public user interface and / or the winner's private user interface.

The invention provides a multi-mode voice interaction method based on a large model, electronic equipment and a storage medium, and the method comprises the steps: carrying out the voice recognition of the voice data of a first language inputted by a user, and obtaining a first language text; performing language translation on the first language text to translate the first language text into a second language text; performing semantic analysis on the second language text to obtain a semantic analysis result; performing multi-modal knowledge graph construction on the semantic analysis result based on a preset large model to obtain an enhanced semantic understanding result; wherein the multi-modal information related to the second language text is fused in the multi-modal knowledge graph; generating a dialogue strategy of the enhanced semantic understanding result; and performing voice synthesis based on the dialogue strategy to generate reply voice data expressed in the first language. According to the method and the device, multi-language interaction is realized, and meanwhile, various modal information is fused.

The invention provides a speech synthesis method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence, and the method comprises the steps: obtaining a to-be-synthesized text and a tone reference audio; inputting the to-be-synthesized text into a pre-training decoding model to enable the pre-training decoding model to determine voice content and voice style corresponding to the to-be-synthesized text, and obtaining semantic features corresponding to the to-be-synthesized text; wherein the pre-training decoding model is trained by using a training text and a corresponding training semantic feature, the training semantic feature is generated by using a training voice audio corresponding to the training text, and the training voice audio has a preset voice style; inputting the semantic features, the to-be-synthesized text and the timbre reference audio into a pre-trained speech synthesis model for speech synthesis to obtain a synthesized speech audio; according to the invention, speech synthesis can be carried out together with the text to be synthesized and the timbre reference audio by using the semantic features containing the speech content and the speech style, so that the timbre style decoupling is realized.

According to the home assistant system based on the large language model agent, a voice interaction module receives voice input of a user and converts the voice input into a text, multi-language recognition and voice synthesis are achieved through natural language processing, and natural and smooth two-way voice interaction is supported. The WeChat interaction module receives and processes text or voice information sent by a user through WeChat, and remote instruction input and information feedback are achieved. The core of the system is a plurality of functional Agent modules. The task analysis Agent analyzes semantics input by a user based on a large language model and dispatches a corresponding functional Agent to execute tasks, and the functions of information query, smart home control, information transmission, mobile navigation, monitoring and the like are covered. Personalized learning and optimization can be carried out according to a user-defined information base, use habits and historical records, the task execution precision and efficiency are improved, and the method is suitable for multi-scene application in the field of smart home.

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a speech synthesis method based on a gated attention mechanism, and the method comprises the steps: obtaining text data and speech data, converting the text data into a text symbol sequence through a text encoder, speech features of the speech data are extracted through a speech encoder trained based on a self-supervised learning model, and the speech features are quantized into a discrete speech symbol sequence; then, respectively organizing the text symbol sequence and the voice symbol sequence into a text sequence and a voice sequence; the corresponding relation between the text symbols and the voice symbols is established through preliminary alignment processing, the matching degree between the text symbols and the voice symbols is dynamically adjusted through a gating attention mechanism, and finally a final voice signal is generated through a decoder. According to the invention, through the voice encoder trained by the self-supervised learning model, the extraction capability of voice features is effectively improved, and effective feature representation can still be learned from unlabeled data especially in a scene lacking a large amount of labeled data.

The invention relates to the technical field of voiceprint recognition, and particularly provides a voiceprint recognition and voice synthesis combined intelligent dialogue method, which comprises the following steps: acquiring a user voice input signal in real time; voiceprint feature extraction is carried out on the voice input signal, and a composite voiceprint vector containing user identity features and rhythm features is obtained; user identity matching is carried out based on the composite voiceprint vector, and a dynamic user portrait database is associated and called; generating personalized semantic feedback content according to the weight-updated dynamic parameters in the dynamic user portrait database; generating a target timbre synthesis parameter set based on the timbre representation vector in the composite voiceprint vector in combination with the spatio-temporal characteristic parameters of the interaction scene; and performing voice synthesis on the personalized semantic feedback content by adopting the target timbre synthesis parameter set, adjusting rhythm expression parameters of the synthesized voice based on real-time rhythm characteristics in the composite voiceprint vector, and finally outputting personalized voice, thereby realizing natural interaction of one person, one voice and one strategy.

The invention relates to the technical field of artificial intelligence, and discloses a speech synthesis method based on hierarchical emotional distribution, which comprises the following steps: acquiring audio data and text transcription, segmenting audio into phoneme, word and sentence-level speech units, extracting acoustic characteristics such as pitch, speed and rhythm, generating an emotional distribution vector by using a pre-training model; the method comprises the following steps: processing text transcription to generate text embedded information, combining an emotion distribution vector, the text embedded information and acoustic features, generating rhythm features through a variable converter, generating a Mel-frequency spectrogram by a decoder based on the rhythm features, and converting the Mel-frequency spectrogram into a time domain audio signal by a vocoder and outputting the time domain audio signal. According to the method, the audio data are hierarchically segmented into phonemes, words and sentences, the acoustic features, the emotion distribution vectors and the text embedding information are combined, and the rhythm features are generated through the variable converter, so that fine control over emotion expression is achieved, and the naturalness and emotion expressive force of speech synthesis are remarkably improved.

The invention belongs to the technical field of speech translation, and particularly relates to a video speech automatic translation method based on a pre-training model, and the method comprises the following steps: 1, preprocessing video and audio data; step 2, voice recognition and language detection; step 3, machine translation and text post-processing; step 4, speech synthesis and audio mixing; step 5, synchronizing video processing and subtitles; step 6, quality control and multi-dimensional evaluation; 7, carrying out model iteration and data closed loop; and step 8, system deployment and engineering implementation. Through deep fusion of efficient transfer learning of the pre-training model and the multi-modal technology, a high-precision, low-cost and easy-to-expand video speech translation solution is constructed, the time and labor cost of globalized content production is greatly reduced, the cross-language communication efficiency is improved, immersive multi-language experience is provided, and the method is suitable for popularization and application. And a data-driven continuous optimization mechanism is established, so that the system performance is improved along with the increase of the use scale.

The embodiment of the invention provides a voice conversation method and system, electronic equipment, a storage medium and a program product. One method is implemented as follows: a voice dialogue stream (which can be an inquiry voice stream input by a user for a target commodity) of a user is subjected to streaming response, a plurality of dialogue text segments are generated in a segmented manner in the streaming response process, and the dialogue text segments are output in real time after being generated and stored in a first cache queue; and when it is detected that the reply text fragment is stored in the first cache queue, the stored reply text fragment is converted into a reply voice fragment in real time and played. The streaming response may be performed by an agent. Visibly, according to the response of the voice dialogue stream of the user, the dialogue is played while being generated, the waiting time from text to voice synthesis can be greatly shortened, and therefore the response time can be shortened.

The invention relates to a speech synthesis method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a source language dubbing audio corresponding to a source language subtitle text of a source video and a target language subtitle text translated by the source language subtitle text; extracting audio emotion features from the source language dubbing audio by using an emotion extractor, wherein the audio emotion features represent emotions expressed by the source language dubbing audio; converting the target language subtitle text into a phoneme sequence, and encoding the phoneme sequence by using a text encoder to obtain text encoding features; fusing the audio emotion features with the text coding features to obtain emotion text features; and generating a target language audio based on the emotion text features by using a decoder, the target language audio being used as a dubbing audio of the source video in the target language. Therefore, the high-quality target language audio with the source dubbing audio emotion can be automatically and efficiently generated, the cost is low, and the efficiency is high.

The invention discloses an artificial intelligence speech recognition system, and the system comprises a multi-modal feature extraction module which employs an improved Conformer architecture to synchronously extract the time-frequency features and text embedding vectors of speech signals; the joint training module is used for performing joint optimization on ASR and NMT loss functions through an adversarial training strategy, learning voice recognition and machine translation tasks at the same time through joint training, and completing direct mapping from voice features to a target language; the context perception translation engine is used for integrating an attention mechanism of a pre-training language model, carrying out deep coding on the extracted speech features and generating cross-language semantic representation; the self-adaptive post-processing module is used for dynamically optimizing an output result by adopting a reinforcement learning framework, dynamically adjusting the output result according to a reward function, and optimizing translation quality and a speech synthesis effect; the dynamic language recognition module is a real-time language classifier based on a Wave2Vec 2.0 framework and is used for recognizing the language of the input voice in real time; and the incremental field adaptation module is used for quickly updating a field term library by using a LoRA fine tuning technology.

The invention relates to artificial intelligence, and discloses a speech synthesis method and device with a conditional matching stream, equipment and a medium, and the method comprises the steps: receiving text information, coding the text information into a high-dimensional implicit vector, and obtaining a text information vector; label information is formed according to the alignment relation between the text information and the voice, and the relation between the acoustic features of the text information and the voice is predicted; performing matching in the voice data set according to the text information vector to obtain corresponding text-related voice, generating intermediate-state voice according to Gaussian noise, the text-related voice and the time step, performing noise prediction according to the intermediate-state voice and the time step by adopting a conditional matching flow model to obtain predicted noise, and outputting the predicted noise. Calculating the loss between the input Gaussian noise and the prediction noise, and combining the prediction noise with the intermediate state voice to extract and obtain the acoustic features of the text related voice; and generating a decoded speech waveform according to the obtained acoustic characteristics of the text-related speech by adopting an acoustic decoder. And the speech synthesis quality is improved.

The invention discloses a lecturer video automatic generation method based on AI speech synthesis and animation driving. The method comprises the following steps: performing structured analysis on a PPT or a text script through an improved interior point method and an incremental shortest path algorithm; performing semantic grouping by applying a full-dynamic parallel single-link clustering algorithm and generating an enhanced script with an expressive mark; a CosyVoice technology is combined with a low-rank approximation method to generate a high-quality voice data stream; establishing a mapping relation between contents and action expressions through semantic analysis, and generating a complete action expression instruction set; and driving the digital human model by using the msueTalk technology, and generating a final lecturer teaching video through a parallel rendering algorithm. According to the invention, the method achieves the efficient and automatic generation of the education video, remarkably improves the content production efficiency, reduces the production cost, and guarantees the specialty and expressive force of the teaching video.

The invention provides an immersive concert experience system based on virtual reality, and belongs to the technical field of virtual reality, and the system comprises a dynamic adjustment module which is used for predicting the preference of a user for stage arrangement, light effects and music types through a machine learning model according to the historical preference data of the user, and outputting the preference to the stage arrangement, light effects and music types; dynamically adjusting visual elements and sound field parameters of the virtual scene and a refresh rate and a field angle of the virtual scene in combination with the user physiological indexes and the user head motion trail which are acquired in real time; the action capture and feedback module is used for capturing an interaction instruction sent to the dynamically adjusted virtual scene by the limb action of the user in real time; and the interaction feedback module is used for enabling the virtual performer to respond to the interaction instruction of the user in real time based on the expression recognition and voice synthesis technology, and generating personalized interaction feedback content. A user constructs an immersive music interaction field domain, and experience logic of a virtual music meeting is completely reconstructed.

The invention discloses an intelligent oral English training system and method based on Prompt engineering, and relates to the technical field of oral English training. According to the method, a user is allowed to freely describe training requirements by using a natural language, the system automatically constructs a training Prompt, and personalized customization is realized; content deviating from a training target is accurately recognized through a deviation question detection module, a user is guided to return to a theme through real-time voice or text, and effectiveness and pertinence of training are guaranteed; a clear template definition ensures systematicness and reproducibility of a training task, and clear control and evaluation of a teaching target are facilitated; in combination with voice recognition, voice synthesis and text generation, natural interaction in a real context is realized, and the training experience is improved; on the basis of multi-dimensional automatic evaluation of a keyword hit rate, pronunciation quality and the like, personalized learning suggestions are supported; teachers are supported to generate training tasks in batches, students are supported to customize scenes, and different education environment requirements are met.

The present invention relates to a system for transforming static websites into artificial intelligence (AI)-enabled interactive multi-modal conversational platforms. The system comprises a computing device having a processor for receiving user queries as text or speech input through an input module cooperating with a speech-to-text module. A natural language processing (NLP) module interprets intent, classifies user context, and retrieves grounded information from multiple webpages. A persona adaptation module dynamically modifies vocabulary, tone, and avatar representation across roles such as sales assistant, recruiter, educator, healthcare professional, etc. A response generator module produces structured natural language output, transmitted to a text-to-speech synthesis module and an avatar generation module to render synchronized lifelike video responses. An output rendering module displays multi-modal responses include text, audio, and video, thereby enabling direct navigation and escalation beyond limitations of conventional static websites.

The invention discloses a voice recognition model training method and device, equipment and a storage medium, and relates to the technical field of voice recognition, and the method comprises the steps: processing an initial audio signal through employing a preset audio processing mode, processing the obtained target feature sequence based on an acoustic feature consistency constraint condition, an identification accuracy constraint condition and a teacher-student comparison constraint condition to obtain initial voice training data; processing the initial voice training data by using a voice synthesis center, and processing the obtained to-be-processed voice training data by using a preset acoustic scene simulation system to obtain to-be-enhanced voice training data; and processing the to-be-enhanced speech training data based on the word level by using an attention enhancement mechanism to obtain word enhanced speech training data, processing the word enhanced speech training data based on the sentence level by using a contrast learning framework, and training the initial speech recognition model by using the obtained target speech training data to obtain a target speech recognition model. Therefore, the efficiency of training the speech recognition model can be improved.

The invention relates to the technical field of voice processing, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a voice text bidirectional conversion method, device, equipment and medium, and the method comprises the steps: respectively executing voice recognition or voice synthesis operation according to the type of input information; for the voice information, noise suppression parameters are generated in combination with the lip movement video data, noise reduction processing is executed, and the recognition accuracy is improved; for text information, a pre-generated speaker style vector is obtained, the vector is cited in the speech synthesis process to generate natural personalized speech, and lip movement information and tactile feedback which are synchronous with speech output are generated. According to the method, complex noise is suppressed by fusing lip movement data, personalized voice is generated by using the style vector, and lip movement and touch information is output, so that bidirectional real-time conversion of voice and text in a complex environment is realized, and recognition accuracy, voice naturalness and interaction synchronism are effectively improved.

The invention relates to a method for converting Lao characters into phonemes based on transfer learning, and aims to realize efficient phoneme conversion by utilizing a model pre-trained on other languages and finely adjusting limited Lao data. According to the method, the problem of data scarcity can be relieved, the training process of the model can be accelerated, and the generalization ability of the model on new tasks is improved. A more robust Lao character transphoneme model can be constructed through transfer learning, and a foundation is laid for subsequent speech synthesis. As Thai with high language similarity with Lao is also a low-resource language and cannot be used as a pre-training language, Chinese is selected as the pre-training language. A Chinese open source data set is firstly used for training a Chinese G2P model based on a Transform architecture, decoding end parameters of the model are initialized to a student model, fine tuning training is performed by using a Lao data set with the scale of ten thousand, and the accuracy of a Lao character-to-phoneme model is improved.

The invention belongs to the field of artificial intelligence, medical treatment and finance, and relates to a speech synthesis method, which comprises the following steps: performing phoneme conversion and context embedding on a text data set, obtaining a phoneme sequence and a text context vector, extracting reference speech features of a reference audio sample, inputting a phoneme-level potential diffusion model, and generating phoneme-level prior potential features; inputting the phoneme-level prior potential features into a phoneme-level variational self-encoding model to obtain reconstructed speech features; extracting a target voice feature of the target audio sample; performing alignment and up-sampling on the target voice feature and a phoneme sequence to obtain a target audio frame-level feature; and training based on the reconstructed speech features and the target audio frame-level features to obtain a conditional control speech synthesis model, and synthesizing the to-be-converted text speech. The invention further provides a speech synthesis device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the to-be-converted text can be stored in a block chain. The speaker fidelity and the text understandability are improved.

The invention discloses a 3D digital human real-time dialogue interaction system and method, and the method comprises the steps: obtaining a voice signal inputted by a user, carrying out the recognition processing of the voice signal through a voice recognition model, obtaining text data, inputting the text data into a text generation model, and outputting the voice text data, and inputting the text data into a voice synthesis model, and outputting the voice synthesis data. The method comprises the steps of inputting speech synthesis data into a 3D expression generation model, outputting 3D expression data, constructing a sound clone model according to a speech signal and the 3D expression data, outputting human voice audio data, performing fine tuning on the sound clone model, and storing fine tuning parameter data, so that a user and a digital human are in smooth and natural conversation, timeliness and continuity of communication are ensured, and user experience is improved. The user experience is greatly improved; by accessing the knowledge base, the digital person has massive knowledge reserve, valuable suggestions and guidance are provided for the user, the user can individually set the figure background of the digital person, and the emotion connection between the user and the digital person is enhanced.