1639 results about "Speech synthesis" patented technology

The invention relates to dialect speech synthesis, in particular to a semi-supervised dialect emotion speech synthesis system based on hybrid experts, which comprises a text analysis module for preprocessing an input dialect text and generating a text representation vector through feature extraction and feature fusion; the mixed expert module is used for acquiring dialect acoustic features, rhythm features, emotional features and general acoustic features; the dynamic routing module is used for realizing intelligent cooperation among experts through a task-aware soft routing algorithm; the semi-supervised learning module is used for training supervised learning by using the marked dialect emotion voice data and training self-supervised learning by using the unmarked dialect emotion voice data; the acoustic parameter generation module is used for integrating the output of each expert to generate a complete acoustic parameter set; the neural vocoder is used for converting the acoustic parameter set into final dialect emotion voice; the method can effectively overcome the defect that the dialect emotional voice is difficult to accurately synthesize under the condition of lack of sample resources.

The invention provides a multi-modal intelligent question-answering and recommendation system supporting emotional speech output, and relates to the technical field of visual question-answering, and the system specifically comprises a multi-modal question-answering module, a multi-modal recommendation module, a speech recognition module, and a speech synthesis module. Wherein the multi-mode question answering module converts an image input by a user into a natural language form, and final question answering is completed in combination with a comet knowledge base COMET; the voice recognition module is used for recognizing voice data input into the multi-mode intelligent question answering and recommending system and processing the voice data into a text form; the multi-modal recommendation module identifies whether modal missing exists in input data or not, generates representation of the missing modal, obtains vector representation of the user and the article through a graph convolutional neural network and learning according to an interaction matrix of the user and the article, carries out reliability calculation, and finally sends a multi-modal recommendation result to the user and the article. Sending back to the multi-mode question and answer module; and the voice synthesis module outputs the voice to the user.

An apparatus for end-to-end text-to-speech synthesis is provided. The apparatus comprises input interface circuitry configured to receive first input data indicative of a phoneme and second input data indicative of a first target duration for the phoneme. The apparatus further comprises processing circuitry configured to, using a trained machine-learning model, map the phoneme to a state using an encoder sub-model of the trained machine-learning model, estimate a second target duration for the phoneme based on the state and determine an attention weight based on the first target duration and the second target duration. The processing circuitry is further configured to map the state to audio data based on the attention weight using a decoder sub-model of the trained machine-learning model, wherein the audio data are indicative of an audio waveform representing speech.

The invention relates to the technical field of computer vision and human-computer interaction, and discloses an AI digital human intelligent interaction method, system and equipment based on multiple modalities, and the method comprises the steps: pre-awakening a digital human when a human face is detected, and further thoroughly awakening the digital human based on recognized preset voice information or preset gesture information; voice and video information of a user in the interaction process is obtained, a keyword extraction result, a gesture recognition result and an emotional state tag are generated, a pre-constructed knowledge base is utilized to retrieve related information, a big language generation model module is combined to generate an answer text, and the answer text is input into a preset voice synthesis model to generate emotional voice output. And based on the current emotional state label of the user, driving the digital human animation to be output in an emotional manner. According to the method and the system, the digital human for understanding the emotion of the user, generating personalized answers, providing voices with rich emotions and displaying natural expressions and actions can be created, better interaction with the user can be realized, and more humanized and effective services can be provided.

The invention discloses a construction method of a real-time streaming voice intelligent question and answer service system. The construction method comprises the following steps: receiving input voice data; calling an independent voice recognition service to convert the voice data into an input text; the method comprises the following steps: constructing cue words of a large language model based on an input text, initiating an LLM streaming request to an independent large language model service, and obtaining streaming text answers generated by LLM in real time; segmenting the flow type text answer in real time through a dynamic sentence segmentation device to generate a plurality of clauses; calling an independent speech synthesis service in parallel for each clause, and converting the text into an audio data block; and combining the audio data blocks into streaming audio data according to the generation sequence, and returning the streaming audio data to a client for playing in real time. According to the invention, by decoupling ASR, LLM and TTS services and combining the asynchronous streaming framework and the strategy of "wanting while speaking", the real-time performance and flexibility of the voice question-answering system are significantly improved.

The invention provides a video generation control method and a computer readable storage medium, and is applied to the technical field of computers. According to the method, the structural defect of a split script is solved through a large split model, the consistency of split image features is ensured through image generation based on reference image feature constraint, audio and picture synchronization is realized in combination with emotional speech synthesis and dynamic mouth shape alignment, the video fidelity is improved through a split video cue word guiding multi-subject generation algorithm, and the user experience is improved. Therefore, the technical problems of missing emotion logic, poor consistency of split image features, difficulty in audio synchronization, insufficient controllability of video generation and the like in film and television creation in the existing AIGC technology are effectively solved, and the efficiency and the quality of video content production are improved.

The invention relates to the technical field of data processing, and discloses a multi-modal digital human generation method and system based on AIGC and a storage medium. The method comprises the steps of performing modeling and dynamic weight adjustment processing on multi-modal feature data to obtain dynamic personalized knowledge data; performing intelligent matching processing on the target scene information and the dynamic personalized knowledge data to obtain situational behavior instruction data; performing hierarchical action sequence generation, adaptive resolution optimization and speech synthesis processing on the initial digital human to obtain multi-modal digital human performance data; and behavior deviation identification is carried out on the multi-modal digital human performance data to obtain behavior deviation data and performance feedback data, and adaptive optimization processing is carried out on the multi-modal digital human performance data and the performance feedback data to obtain a target digital human. According to the invention, the efficiency and accuracy of multi-modal digital human generation based on AIGC are improved.

The invention discloses a personalized digital human generation method based on a single video, and relates to the field of virtual digital human modeling and driving, and the method aims at the video and voice data of a target person, through introducing a multi-modal alignment constrained voice driving synchronization mechanism and combining semantic understanding and an emotion label expression generation model, a personalized digital human model is generated. High synchronization, nature and vividness of digital human facial expressions and voice contents are realized. According to the method, a self-supervised style consistency constraint is added in model training to ensure that the style of a generated character image is stable, and a 3D semantic mask is added in an image fusion stage to improve the fusion precision and realistic effect of a synthetic facial expression and a reference face. According to the method, the digital human can be rapidly cloned and driven to synthesize the expression through the voice only through a single video sample, the generation process is efficient, and the obtained digital human video has excellent sense of reality and interactivity.

The invention belongs to the technical field of vehicle state detection and early warning, and particularly relates to a vehicle state monitoring and early warning method and system.The monitoring and early warning method comprises the steps that a sensor obtains real-time data, an anomaly detection algorithm is applied, and an abnormal event is marked; calculating an information priority according to the abnormal severity and the driving scene, and distributing the information priority to a high-priority queue; multi-mode early warning is generated, and high-frequency sound and vibration are used during high-speed driving; extracting a voice prompt, and generating voice waveform data through a voice synthesis module; voice input of a driver is recognized, intention is analyzed, and abnormal information feedback is provided; according to the scene and the abnormal state, interactive output is optimized, and detailed information is displayed during low-speed congestion; dynamically adjusting the interface of the instrument panel, and amplifying the key area in case of abnormal severity; and integrating the image data and the voice data, generating a multi-mode signal, and transmitting the multi-mode signal after rendering processing. The vehicle abnormal information can be timely and accurately transmitted to a driver, and the driving safety is effectively improved.

An automated conversational multi-player gaming platform is described. A machine-learning model is trained to use a natural language processor and a speech synthesizer as an automated dealer communicating with players of a game of chance. The trained machine-learning model receives requests from people to become players of a game of chance, outputs private game content associated with the game of chance to the players' private user interfaces, and outputs public game content associated with the game of chance to the players' public user interface. The trained machine-learning model receives selections of game playing options and wager values associated with the game of chance from the players via their private user interfaces, and outputs information identifying one player as a winner of the game of chance and a credit of the winner's wager value to the players' public user interface and / or the winner's private user interface.

The invention relates to the technical field of voiceprint recognition, and particularly provides a voiceprint recognition and voice synthesis combined intelligent dialogue method, which comprises the following steps: acquiring a user voice input signal in real time; voiceprint feature extraction is carried out on the voice input signal, and a composite voiceprint vector containing user identity features and rhythm features is obtained; user identity matching is carried out based on the composite voiceprint vector, and a dynamic user portrait database is associated and called; generating personalized semantic feedback content according to the weight-updated dynamic parameters in the dynamic user portrait database; generating a target timbre synthesis parameter set based on the timbre representation vector in the composite voiceprint vector in combination with the spatio-temporal characteristic parameters of the interaction scene; and performing voice synthesis on the personalized semantic feedback content by adopting the target timbre synthesis parameter set, adjusting rhythm expression parameters of the synthesized voice based on real-time rhythm characteristics in the composite voiceprint vector, and finally outputting personalized voice, thereby realizing natural interaction of one person, one voice and one strategy.

The invention belongs to the technical field of speech translation, and particularly relates to a video speech automatic translation method based on a pre-training model, and the method comprises the following steps: 1, preprocessing video and audio data; step 2, voice recognition and language detection; step 3, machine translation and text post-processing; step 4, speech synthesis and audio mixing; step 5, synchronizing video processing and subtitles; step 6, quality control and multi-dimensional evaluation; 7, carrying out model iteration and data closed loop; and step 8, system deployment and engineering implementation. Through deep fusion of efficient transfer learning of the pre-training model and the multi-modal technology, a high-precision, low-cost and easy-to-expand video speech translation solution is constructed, the time and labor cost of globalized content production is greatly reduced, the cross-language communication efficiency is improved, immersive multi-language experience is provided, and the method is suitable for popularization and application. And a data-driven continuous optimization mechanism is established, so that the system performance is improved along with the increase of the use scale.

The embodiment of the invention provides a voice conversation method and system, electronic equipment, a storage medium and a program product. One method is implemented as follows: a voice dialogue stream (which can be an inquiry voice stream input by a user for a target commodity) of a user is subjected to streaming response, a plurality of dialogue text segments are generated in a segmented manner in the streaming response process, and the dialogue text segments are output in real time after being generated and stored in a first cache queue; and when it is detected that the reply text fragment is stored in the first cache queue, the stored reply text fragment is converted into a reply voice fragment in real time and played. The streaming response may be performed by an agent. Visibly, according to the response of the voice dialogue stream of the user, the dialogue is played while being generated, the waiting time from text to voice synthesis can be greatly shortened, and therefore the response time can be shortened.

The invention relates to a speech synthesis method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a source language dubbing audio corresponding to a source language subtitle text of a source video and a target language subtitle text translated by the source language subtitle text; extracting audio emotion features from the source language dubbing audio by using an emotion extractor, wherein the audio emotion features represent emotions expressed by the source language dubbing audio; converting the target language subtitle text into a phoneme sequence, and encoding the phoneme sequence by using a text encoder to obtain text encoding features; fusing the audio emotion features with the text coding features to obtain emotion text features; and generating a target language audio based on the emotion text features by using a decoder, the target language audio being used as a dubbing audio of the source video in the target language. Therefore, the high-quality target language audio with the source dubbing audio emotion can be automatically and efficiently generated, the cost is low, and the efficiency is high.

The invention discloses an artificial intelligence speech recognition system, and the system comprises a multi-modal feature extraction module which employs an improved Conformer architecture to synchronously extract the time-frequency features and text embedding vectors of speech signals; the joint training module is used for performing joint optimization on ASR and NMT loss functions through an adversarial training strategy, learning voice recognition and machine translation tasks at the same time through joint training, and completing direct mapping from voice features to a target language; the context perception translation engine is used for integrating an attention mechanism of a pre-training language model, carrying out deep coding on the extracted speech features and generating cross-language semantic representation; the self-adaptive post-processing module is used for dynamically optimizing an output result by adopting a reinforcement learning framework, dynamically adjusting the output result according to a reward function, and optimizing translation quality and a speech synthesis effect; the dynamic language recognition module is a real-time language classifier based on a Wave2Vec 2.0 framework and is used for recognizing the language of the input voice in real time; and the incremental field adaptation module is used for quickly updating a field term library by using a LoRA fine tuning technology.

The invention discloses a lecturer video automatic generation method based on AI speech synthesis and animation driving. The method comprises the following steps: performing structured analysis on a PPT or a text script through an improved interior point method and an incremental shortest path algorithm; performing semantic grouping by applying a full-dynamic parallel single-link clustering algorithm and generating an enhanced script with an expressive mark; a CosyVoice technology is combined with a low-rank approximation method to generate a high-quality voice data stream; establishing a mapping relation between contents and action expressions through semantic analysis, and generating a complete action expression instruction set; and driving the digital human model by using the msueTalk technology, and generating a final lecturer teaching video through a parallel rendering algorithm. According to the invention, the method achieves the efficient and automatic generation of the education video, remarkably improves the content production efficiency, reduces the production cost, and guarantees the specialty and expressive force of the teaching video.

The invention provides an immersive concert experience system based on virtual reality, and belongs to the technical field of virtual reality, and the system comprises a dynamic adjustment module which is used for predicting the preference of a user for stage arrangement, light effects and music types through a machine learning model according to the historical preference data of the user, and outputting the preference to the stage arrangement, light effects and music types; dynamically adjusting visual elements and sound field parameters of the virtual scene and a refresh rate and a field angle of the virtual scene in combination with the user physiological indexes and the user head motion trail which are acquired in real time; the action capture and feedback module is used for capturing an interaction instruction sent to the dynamically adjusted virtual scene by the limb action of the user in real time; and the interaction feedback module is used for enabling the virtual performer to respond to the interaction instruction of the user in real time based on the expression recognition and voice synthesis technology, and generating personalized interaction feedback content. A user constructs an immersive music interaction field domain, and experience logic of a virtual music meeting is completely reconstructed.

The invention discloses an intelligent oral English training system and method based on Prompt engineering, and relates to the technical field of oral English training. According to the method, a user is allowed to freely describe training requirements by using a natural language, the system automatically constructs a training Prompt, and personalized customization is realized; content deviating from a training target is accurately recognized through a deviation question detection module, a user is guided to return to a theme through real-time voice or text, and effectiveness and pertinence of training are guaranteed; a clear template definition ensures systematicness and reproducibility of a training task, and clear control and evaluation of a teaching target are facilitated; in combination with voice recognition, voice synthesis and text generation, natural interaction in a real context is realized, and the training experience is improved; on the basis of multi-dimensional automatic evaluation of a keyword hit rate, pronunciation quality and the like, personalized learning suggestions are supported; teachers are supported to generate training tasks in batches, students are supported to customize scenes, and different education environment requirements are met.

The present invention relates to a system for transforming static websites into artificial intelligence (AI)-enabled interactive multi-modal conversational platforms. The system comprises a computing device having a processor for receiving user queries as text or speech input through an input module cooperating with a speech-to-text module. A natural language processing (NLP) module interprets intent, classifies user context, and retrieves grounded information from multiple webpages. A persona adaptation module dynamically modifies vocabulary, tone, and avatar representation across roles such as sales assistant, recruiter, educator, healthcare professional, etc. A response generator module produces structured natural language output, transmitted to a text-to-speech synthesis module and an avatar generation module to render synchronized lifelike video responses. An output rendering module displays multi-modal responses include text, audio, and video, thereby enabling direct navigation and escalation beyond limitations of conventional static websites.

The invention discloses a voice recognition model training method and device, equipment and a storage medium, and relates to the technical field of voice recognition, and the method comprises the steps: processing an initial audio signal through employing a preset audio processing mode, processing the obtained target feature sequence based on an acoustic feature consistency constraint condition, an identification accuracy constraint condition and a teacher-student comparison constraint condition to obtain initial voice training data; processing the initial voice training data by using a voice synthesis center, and processing the obtained to-be-processed voice training data by using a preset acoustic scene simulation system to obtain to-be-enhanced voice training data; and processing the to-be-enhanced speech training data based on the word level by using an attention enhancement mechanism to obtain word enhanced speech training data, processing the word enhanced speech training data based on the sentence level by using a contrast learning framework, and training the initial speech recognition model by using the obtained target speech training data to obtain a target speech recognition model. Therefore, the efficiency of training the speech recognition model can be improved.

The invention relates to the technical field of voice processing, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a voice text bidirectional conversion method, device, equipment and medium, and the method comprises the steps: respectively executing voice recognition or voice synthesis operation according to the type of input information; for the voice information, noise suppression parameters are generated in combination with the lip movement video data, noise reduction processing is executed, and the recognition accuracy is improved; for text information, a pre-generated speaker style vector is obtained, the vector is cited in the speech synthesis process to generate natural personalized speech, and lip movement information and tactile feedback which are synchronous with speech output are generated. According to the method, complex noise is suppressed by fusing lip movement data, personalized voice is generated by using the style vector, and lip movement and touch information is output, so that bidirectional real-time conversion of voice and text in a complex environment is realized, and recognition accuracy, voice naturalness and interaction synchronism are effectively improved.

The invention discloses a 3D digital human real-time dialogue interaction system and method, and the method comprises the steps: obtaining a voice signal inputted by a user, carrying out the recognition processing of the voice signal through a voice recognition model, obtaining text data, inputting the text data into a text generation model, and outputting the voice text data, and inputting the text data into a voice synthesis model, and outputting the voice synthesis data. The method comprises the steps of inputting speech synthesis data into a 3D expression generation model, outputting 3D expression data, constructing a sound clone model according to a speech signal and the 3D expression data, outputting human voice audio data, performing fine tuning on the sound clone model, and storing fine tuning parameter data, so that a user and a digital human are in smooth and natural conversation, timeliness and continuity of communication are ensured, and user experience is improved. The user experience is greatly improved; by accessing the knowledge base, the digital person has massive knowledge reserve, valuable suggestions and guidance are provided for the user, the user can individually set the figure background of the digital person, and the emotion connection between the user and the digital person is enhanced.

The invention relates to the field of artificial intelligence medical application, and discloses a lightweight intelligent traditional Chinese medicine inquiry system and a construction method thereof, and the system comprises a multi-dialect adaptive speech recognition module, a traditional Chinese medicine intelligent dialogue large language model module, a natural speech synthesis module, and a continuous learning mechanism module. The multi-dialect adaptive speech recognition module is used for converting dialect speech input of a patient into a standard text; the traditional Chinese medicine intelligent dialogue big language model module is the core of the system and is used for carrying out natural language understanding, dialectical reasoning and inquiry dialogue generation, and the natural speech synthesis module is used for converting a text response generated by the system into speech output; and the continuous learning mechanism module realizes continuous optimization of the large language model through incremental learning architecture and clinical feedback integration. According to the method, while the professional traditional Chinese medicine diagnosis capability is maintained, the calculation complexity is remarkably reduced, and the universality and sustainable development capability of system application are improved.

The invention discloses a real-time interaction 3D digital holographic cabin method based on deep learning and sound cloning. The method comprises the following steps that S1, user data are collected and preprocessed; s2, extracting feature vectors of facial expressions and limb actions; s3, generating speech synthesis data by using the improved GE2E network and a preset target speech text; s4, generating a synthesized voice audio based on the voice synthesis data; s5, generating a three-dimensional digital human motion sequence according to the facial expression feature vector and the body motion feature vector; s6, performing timestamp alignment on the three-dimensional digital human action sequence and the synthesized voice audio, and constructing a synchronous output stream; and S7, rendering the synchronous output stream, and performing three-dimensional visual output. According to the method, the improved GE2E network, deep learning and sound cloning methods are fused, three-dimensional virtual human voice action synchronous control is achieved, and the method has the advantages of being high in real-time performance, high in immersion and natural in interaction.

The invention relates to the technical field of artificial intelligence, in particular to a digital human real-time dialogue method and system based on image cloning, a terminal and a medium, and the method comprises the steps: segmenting an original user video into a mouth closing silent video and a non-silent video; intercepting images of a face area in the non-silent video frame by frame and synthesizing the images into a training video; training the image cloning model based on the training video; the answer text generated by the large language model and the voice synthesis model is converted into audio data, a face image sequence matched with the audio data is generated in combination with the trained image clone model, the face image sequence is synthesized into a non-silent video and then pushed out, and if the duration of the non-silent video is not enough, the non-silent video is pushed out. And if not, circularly pushing the mouth closing silence video to the front end in a non-conversation state by adopting a positive-sequence and reverse-sequence loop playing strategy. Through a specific video preprocessing process and a playing mode, the continuity problem of a long dialogue scene is improved, and the naturalness of a digital human in a non-dialogue state is improved.

The invention discloses a fine emotion control TTS method and device based on a large model, equipment and a medium, relates to the technical field of artificial intelligence, and can generate a voice service with fine emotion expression and improve the user experience in the high-sensitivity emotion interaction fields of banks, financial customer service, insurance consultation, medical hospital guide and the like. Acquiring an emotional feature vector of the input text based on a preset large language model; encoding the emotion feature vector to obtain a voice encoding vector; determining an emotion curve vector of the input text based on a pre-trained neural network model; and generating target voice corresponding to the input text based on the voice coding vector and the emotion curve vector. According to the method, emotion feature vectors are extracted through a large language model, a dynamic emotion curve is generated in combination with a neural network, speech synthesis is cooperatively driven, emotion fineness and continuity are remarkably improved, and the problems of traditional TTS emotion expression roughening and fragmentation are solved.

The invention discloses an intelligent voice interaction method and device based on a voice large model, a terminal and a medium, and belongs to the technical field of voice recognition and interaction, and the method comprises the steps: recognizing the voiceprint information of a speaker through a voiceprint recognition model when a wake-up instruction is received; when the identity of the current pronunciator is the specified user registered through voiceprint, acquiring question information of the current pronunciator, calling a prompt word of a character event text of the specified user corresponding to the current pronunciator from a cloud end, and combining with a question proposed by the current pronunciator to obtain the question information of the current pronunciator; generating event text information related to a specified user character corresponding to the current pronunciator; and calling the speech synthesis model, synthesizing the generated event text information into speech according to the tone characteristics of the speaker, and outputting and playing the speech. According to the method, the user is supported to upload the character as the cue word, and the cue word is combined with the large model, so that the dialogue content generation based on the personalized information of the user is realized, and convenience is provided for the use of the user.

The invention discloses a voice cloning system and method fusing rhythm characteristics, belongs to the technical field of voice synthesis and natural language, and is applied to the aspect of fine-grained rhythm control in zero-sample voice synthesis. The implementation method comprises the following steps of: 1, extracting rhythm features and audio features of an audio file, and further respectively acquiring pause features, speed features and tone features in the rhythm features by sequentially adopting transcriptional text inverse coding, syllable-level speed registration quantization and pitch sequence feature splicing modes; 2, fusing the features of the audio files in a manner of discarding feature screening without guidance of a classifier; 3, generating a target Mel spectrogram based on conditional flow matching; generating a target audio file from a to-be-cloned audio file through the trained voice cloning model controlled by the fusion rhythm; compared with the prior art, fine-grained rhythm control of tone and rhythm feature decoupling is realized in zero-sample speech synthesis, so that intonation accuracy based on a context scene is improved.

The invention discloses a user tone cloning and speech synthesis method and device based on deep learning, and relates to the technical field of speech recognition application, and the method comprises the steps: collecting the speech data of a user; extracting timbre features corresponding to the user from the voice data; training the personalized timbre model by adopting a transfer learning technology to obtain a trained timbre model; obtaining picture book content needing to be read, and extracting emotion and rhythm key information contained in the picture book content; according to the extracted picture book content including emotion and rhythm key information, dynamically adjusting each playing parameter of speech synthesis; and inputting the content of the picture book needing to be read into the trained tone model for speech synthesis, adjusting the synthesized speech in combination with the playing parameters of the adjusted speech synthesis, and outputting the adjusted speech. According to the invention, personalized cloning can be carried out according to the timbre characteristics of a specific user, a more natural and vivid speech synthesis effect is provided, and convenience is provided for use of the user.

The invention relates to a simulation digital human real-time intelligent voice interaction system and a simulation digital human real-time intelligent voice interaction method based on vision and a large model, and aims to solve the problems of inaccurate target speaker recognition, high response delay and the like in digital human voice interaction in a complex scene. The system circles an effective recognition range through a camera, triggers audio collection in combination with face detection, locks a target speaker and reduces noise by using lip movement recognition and sound image fusion technologies, converts the target speaker into a text through voice wake-up, generates an answer by means of a large language model (LLM) and knowledge retrieval enhancement (RAG) technologies, generates low-delay voice through a voice synthesis technology accelerated by the vLLM, and performs voice recognition on the target speaker. And driving the preloaded digital human image to synthesize a video stream and pushing the video stream to a front end for rendering in real time. Accurate pickup, low-delay interaction and rapid digital human image switching in a complex environment are realized, the accuracy and real-time performance of intelligent voice question answering are improved, and the method is suitable for government affair halls, exhibition halls and other scenes.