276 results about "Mouth shape" patented technology

The invention provides a digital human interaction method and system based on naked eye 3D visualization, and the method comprises the steps: obtaining query content provided by a user in a multi-modal manner, carrying out the semantic understanding and intention analysis of the query content through a natural language processing technology, and obtaining user intention data; performing matching search in a preset content database according to the user intention data, and if a highly related answer template or video clip exists, calling preset content and feeding back the preset content to the user; according to different answer sources, corresponding digital human video visualization schemes are adopted, a digital human driving engine is used for matching mouth shapes, expressions and actions, and matched digital human videos are synthesized; and displaying the digital human video through a transparent LCD screen.

The invention provides a real-time video translation and audio and picture synchronization method and system based on a multi-modal large model, and relates to the technical field of video translations, and the method comprises the steps: obtaining a source video; extracting the source video based on the multi-modal large model to obtain a multi-modal feature; fusing the multi-modal features through a cross-modal attention mechanism to generate a context semantic vector; translating into a target language text in real time based on the context semantic vector, and processing the translated language text based on the multi-modal features to obtain a translated language sound source; and performing mouth shape adjustment on the source video based on the translation language sound source, and merging the translation language sound source and the mouth shape animation video to obtain a real-time translation video with synchronous sound and picture. According to the method, the limitation of traditional single-modal translation is broken through, and the semantic accuracy of translation is remarkably improved by dynamically aligning the context information through the multi-modal features in combination with a cross-modal attention mechanism.

The invention relates to the technical field of subtitle translation, in particular to a multi-modal time sequence alignment AI video translation method and system, and the method comprises the steps: 1, carrying out the multi-modal analysis of a to-be-translated video, and obtaining audio separation data, voiceprint feature data and visual time sequence data; 2, performing cross-language translation and context optimization on the basis of the voice of the audio separation data to generate a target language text, and synthesizing target language voice retaining the original voice color in combination with the voiceprint feature data and the target language text; generating a mouth shape animation matched with the target language voice based on the lip key point data and the limb action time sequence data; and step 3, performing four-dimensional alignment on the target language voice, the translated text, the mouth shape animation and the limb action sequence through a cross-modal time sequence encoder, and dynamically adjusting the layout of the bilingual subtitles to adapt to a video picture. According to the method and the device, multi-mode synchronization can be taken into consideration during video translation, so that the body actions such as voice, subtitles and mouth shapes are kept aligned.

The invention discloses a video translation method and system based on artificial intelligence. The method relates to the technical field of video translation and comprises the following steps of original sound track extraction, target AI speaker adaptation, AI dubbing generation and mouth shape synchronization and video synthesis. According to the method, independent audio and video streams are obtained by adopting an audio and video separation technology, and multiple original sound tracks are extracted through a voice separation model; matching or generating an adaptive target AI speaker module in a preset tone library; converting the original language voice into a text, translating the text into a target language text, and synthesizing an AI dubbing audio track in combination with a target AI speaker module; and finally, the independent video stream and the multi-AI dubbing audio track are input into the mouth shape synchronization model to output a translated video, so that the timbre fitting degree, the voice quality and the voice consistency of the same speaker of AI dubbing are improved, and meanwhile, the resource utilization rate of video translation and the processing efficiency under batch tasks are improved. The problem that in the prior art, video translation is low in quality and efficiency is solved.

The invention relates to the technical field of artificial intelligence and multimedia processing, and particularly provides an intelligent dialogue method and system based on a digital human, and the method comprises the following steps: S1, an input stage: a user inputs a question or an instruction through a text or voice; s2, the processing stage comprises large model generation reply, TTS model conversion and audio driving mouth shape; and S3, the output stage comprises video generation, stream pushing and display. Compared with the prior art, the method has the advantages that the reply can be generated through the large model, and the TTS model and the audio-driven mouth shape model are combined, so that more natural and anthropomorphic dialogue interaction is realized, and the user experience is improved.

One or more embodiments of the invention provide a digital human rendering method and device, a storage medium and a program product. The digital human rendering method comprises the following steps: analyzing a target voice stream matched with a to-be-rendered digital human to obtain a phoneme sequence and voice rhythm characteristics; a mouth shape control parameter sequence corresponding to the phoneme sequence is determined, all mouth shape control parameters in the mouth shape control parameter sequence are in one-to-one correspondence with all phonemes in the phoneme sequence, and a mouth shape control curve is generated based on the mouth shape control parameter sequence; performing rhythm alignment processing on the mouth shape control curve according to the voice rhythm characteristics, executing rasterization conversion, and generating a mouth shape image block sequence synchronized with the target voice stream; and synthesizing each mouth shape image block in the mouth shape image block sequence with other image contents of the digital human to generate a digital human image frame sequence.

The invention belongs to the technical field of voice processing, and discloses an intelligent cockpit multi-mode voice interaction system and method, and the system comprises a voice triggering unit which collects the environment audio and video information in a cockpit, judges whether to enter a voice interaction mode or not through combining with the environment perception parameters in a vehicle, and sends the voice interaction mode to the vehicle; when the voice interaction triggering condition is satisfied, generating a voice interaction input signal matched with the current environment; the mouth shape analysis unit is used for carrying out acoustic feature extraction on the voice interaction input signal, synchronously analyzing the lip motion trail of the driver in the video information, establishing a corresponding relation between voice phonemes and mouth shape motion, and forming a joint analysis feature; the candidate generation unit is used for carrying out segmented alignment on the joint analysis features and constructing a continuous multi-modal fragment sequence; performing time synchronization on the multi-modal fragment sequence, and projecting the multi-modal fragment sequence to a predefined intention space to obtain a candidate intention set containing different candidate intentions; and the man-machine interaction experience of the intelligent cabin is improved.

The invention discloses a 2D government affair digital human real-time interaction method and system based on an AI painting technology, and the method comprises the steps: making a digital human image based on a modeling mode combining AI painting and a Live2D technology; acquiring an audio input by a user in real time, and identifying the audio input by the user to obtain an audio identification text and a reply text; according to the reply text, synthesizing and outputting a result audio based on a half-stream speech synthesis method of dependency syntactic analysis; and performing mouth shape synchronous real-time driving and audio broadcasting of the digital human image on the manufactured digital human image based on the result audio. According to the invention, the low-cost construction and deployment of the digital human are realized based on the combination of AI drawing and Live2D technology; when system deployment is carried out on the digital people, a semi-streaming speech synthesis scheme based on dependency syntactic analysis is adopted, and the interaction real-time performance and fluency of government affair digital people belonging to service digital people are improved.

The invention relates to the technical field of rehabilitation training, in particular to a multi-logic interaction system for speech disorder rehabilitation training, which comprises a voice accuracy analysis module, a voice mouth shape comparison module, a training item distribution module, a training difficulty adjustment module and a review interval optimization module. By capturing and analyzing the pronunciation information of the user in real time, pronunciation errors can be accurately recognized and corrected, the pertinence and efficiency of voice training are enhanced, a patient can visualize the pronunciation errors by combining voice data with a mouth shape image, and meanwhile, by comparing standard pronunciation images, the pronunciation errors can be accurately recognized and corrected. According to the method, integration of voice and visual information is enhanced, each patient can obtain a rehabilitation scheme meeting target requirements and ability through intelligent distribution of training content and time and dynamic adjustment of training difficulty, training distribution and difficulty adjustment are optimized, a user can train in a proper rhythm, and the training efficiency is improved. And the individuation level and the overall effect of speech disorder rehabilitation training are improved.

A device for creating digital persona, which includes a data collection module responsible for collecting personality data of a target object, a personality training module utilizing a large language model and the personality data to train and generate a virtual personality model with personality characteristics of the target object, thereby generating a virtual personality consistent with the personality characteristics of the target object, an appearance video generation module and voice generation module respectively utilizing face replacement and voice cloning technologies to extract the pictures and sounds from the personality data to generate a virtual personality model with the appearance and voice characteristics of the target object, a lip synchronization module, utilizing lip synchronization technology to ensure that the mouth shape and voice of the digital persona are synchronized, and an interactive module, providing an interactive interface that allows users to interact with the virtual personality and receive responses from it.

The invention relates to a voice-driven high-timeliness digital population type synthesis algorithm, and the algorithm improves the digital population type synthesis effect through introducing lip-reading expert and a pre-trained video automatic encoder as supervision, carefully designing a feature fusion module, training a loss function and other skills. The speed of digital population type synthesis is improved through skills such as channel-by-channel convolution and linear attention. And the device has the functions of controllable mouth opening amplitude and multi-language support.

The invention discloses a bionic robot control method and device, equipment and a storage medium, and relates to the technical field of bionic robots, and the method comprises the steps: obtaining a target character through a speech synthesis model, determining a phoneme sequence based on the target character, coding the phoneme sequence, and carrying out the preset convolution operation of the obtained coded sequence, a corresponding phoneme feature map is obtained; determining voice data based on the phoneme feature map through a voice synthesis model, and determining a voice time sequence based on the voice data; determining a target mouth shape based on the voice time sequence through a voice synthesis model, and determining a mouth shape time sequence based on the voice time sequence and the target mouth shape; and when the voice data is played, a motor of the mouth is controlled to perform corresponding motion based on the mouth shape time sequence, so that the bionic robot generates a corresponding mouth shape. Therefore, the action of the mouth of the bionic robot can be accurately controlled while the bionic robot plays the voice.

The invention discloses a human face mouth shape coordination generation method based on geometric constraints, and relates to the technical field of image generation, and the method comprises the steps: 1, synchronously generating face vertex index tables corresponding to mouth opening and closing spatial form matrixes one by one; 2, based on the facial vertex index table, mapping the lip and gum combined boundary constraint set to a unified coordinate system to form a geometric prior constraint tensor; 3, performing multi-modal topological cascade reversible mixed transformation on the geometric prior constraint tensor to obtain a high-dimensional tooth texture candidate tensor and constraint consistency confidence spectrum; and step 4, using the high-dimensional tooth texture candidate tensor and the constraint consistency confidence spectrum as input, completing anti-aliasing, illumination compensation and gamma correction on a unified rendering pipeline, and finally obtaining a mouth shape coordination synthetic frame sequence. According to the method, the sense of reality, geometric consistency and time sequence coherence of tooth textures in the mouth shape synthesis process are effectively improved, and the problem that lips and teeth are not coordinated in an existing method is solved.

The embodiment of the invention provides a video generation method and system based on AI voice cloning and mouth shape synchronization, and the method comprises the steps: carrying out the fusion of voiceprint features of an input video and an input text through a voice synthesis model after the input video and the input text are obtained, so as to generate a natural voice; analyzing lip key points of the input video by using a lip shape displacement model, and matching lip shape change data according to the lip key points and natural voice; and generating an output video according to the input video and the lip shape change data. According to the method, the phoneme duration prediction of the speech synthesis model and the lip displacement model can be coupled through time sequence convolution, so that the mouth shape of the output video is matched with the speech content, lightweight video restoration is realized by redrawing the lip region, the dynamic response to the input text modified by the user in real time is supported, and the response efficiency is improved.

The invention provides a vehicle-mounted voice recognition method and device, electronic equipment and a storage medium, and belongs to the technical field of voice recognition, and the method comprises the steps: carrying out the fusion of a voice recognition result and a lip language recognition result of a cab, obtaining a multi-modal recognition result, enabling the voice recognition result to be obtained through the recognition of a voice signal collected in real time, and enabling the lip language recognition result to be a lip language recognition result; the lip language recognition result is obtained by recognizing a mouth shape image collected in real time; analyzing the multi-modal recognition result to obtain a voice source of each voice and an intention corresponding to each voice; and based on the voice source and intention of each voice, respectively generating a vehicle control instruction. According to the invention, the voice recognition result and the lip language recognition result are combined to distinguish the voice from different sources, so that the main driver and the co-driver can respond to the voice instructions when sending the voice instructions at the same time, and the requirements of the main driver and the co-driver can be met at the same time.

The embodiment of the invention relates to the field of computer technology application, and discloses a 3D digital human voice and mouth shape action synchronization method and device, equipment and a medium, and the method comprises the steps: obtaining voice data, mouth shape data and limb data, so as to obtain a voice data queue, a mouth shape data queue and a limb data queue; according to the frame information in the mouth shape data queue and the frame information in the limb data queue, adjusting the volume of voice data sent by the voice data queue at a time; playing each frame of data in the mouth shape data queue and each frame of data in the limb data queue frame by frame; when the total amount of the voice data sent by the voice data queue is greater than or equal to a preset voice data volume threshold value, playing voice corresponding to the voice data sent by the voice data queue; according to the 3D digital human voice and mouth shape motion synchronization method disclosed by the invention, the technical problem of low authenticity caused by asynchronous motion and voice of the digital human is solved, so that the motion and voice of the digital human are synchronized, and the authenticity of the digital human is improved.

The present invention relates to the field of computer vision technology, and discloses a virtual digital human lip synchronization optimization method, device, equipment and storage medium. The virtual digital human lip synchronization optimization method includes: obtaining the target audio segment to be output by the virtual digital human at the next moment; judging whether the target audio segment belongs to the audio type to be processed; if the target audio segment belongs to the audio type to be processed, then based on a preset lip synchronization optimization strategy, generating a 3D human face mouth shape parameter frame sequence corresponding to the target audio segment; based on the 3D human face mouth shape parameter frame sequence, generating a corresponding 3D human face mouth shape image frame sequence and rendering it into the virtual digital human. The present invention can adapt to various audio types and improve the fluency and naturalness of the virtual digital human's mouth shape under different audio types.

The invention relates to the field of digital people, and discloses a method and a device for synchronizing voice and mouth shape of a digital person, and electronic equipment. The method comprises the steps of obtaining natural language information, converting the natural language information into character information, determining a video frame and an audio frame based on the character information, and controlling the digital person to speak when the first frame time of the audio frame is the same as the first frame time of the video frame, so that the voice and mouth shape of the digital person can be kept consistent.

The invention discloses an instrument of an intelligent breathing and motion capture system based on six-character table health maintenance, and relates to the technical field of intelligent health maintenance equipment, the instrument comprises a hardware part and a software part; the hardware part comprises a mouth shape visual capture assembly, an action force and angle sensing assembly, an audio acquisition and feedback assembly and a main control and power supply assembly; the software part comprises a data acquisition module, a signal preprocessing module, a parameter analysis and judgment module, a real-time feedback module and a data storage and management module; according to the invention, through integration of the mouth-shaped visual capture assembly, the action force and angle sensing assembly, the audio acquisition and feedback assembly and the master control and power supply assembly, synchronous acquisition and processing of multi-source data are realized; the mouth shape change, the body movement and the audio pronunciation of the user can be accurately captured in real time, and powerful technical support is provided for standardized practice of six-character table health maintenance.

The invention discloses a voice signal processing method for an education robot, relates to the technical field of voice signal processing, and aims to solve the problems of multi-person overlapping language and environmental noise interference in a classroom, multi-channel data is acquired by relying on a microphone array and a camera, and an environmental model is constructed in the step 1 to determine a noise baseline and student distribution; in the second step, overlapped voices are detected, and sound source localization is carried out in combination with the time difference of arrival and mouth shape data; in the third step, directional gain is executed in the target direction, and a deep network is used for separating aliasing voice; and in the step 4, the separated voice is input into a children customized recognition engine to complete high-precision recognition and interaction in combination with confidence evaluation. The recognition accuracy and the interaction efficiency can be remarkably improved under the complex scenes of classroom reverberation and simultaneous speaking of multiple persons, meanwhile, the noise change is tracked through the global environment model so that the education robot can keep stable recognition performance in diversified teaching interaction, and the teaching effect is remarkably enhanced.

The embodiment of the invention discloses a voice action synchronization method and device for a virtual character, equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the voice generation of an answer text, obtaining an answer voice sequence and the attribute information of each phoneme, determining a standard mouth shape identifier of the corresponding phoneme based on the phoneme identifier of each phoneme, and obtaining a voice action synchronization result; generating a mouth shape animation sequence based on the standard mouth shape identifier of each phoneme according to the time range of each phoneme; determining an emotional action sequence and an emotional action starting timestamp corresponding to the emotional label of the answer text, and generating a supplementary animation sequence based on the emotional action sequence and the emotional action starting timestamp; and synchronizing the answer voice sequence, the mouth shape animation sequence and the supplementary animation sequence according to the time sequence to obtain a synchronization relationship, and driving the virtual response character of the user based on the synchronization relationship, the mouth shape animation sequence and the supplementary animation sequence when the answer voice sequence is played, so that the voice action synchronization function of the virtual character is realized.

The invention relates to a voice data extraction method and system adopting artificial intelligence, and relates to the technical field of voice intelligent extraction, and the method comprises the steps: monitoring and collecting video and audio data generated by video communication, and obtaining a corresponding data sequence; carrying out speech speed and volume identification on the audio sequence, obtaining characteristic parameters, and respectively configuring mouth shape and audio weight to obtain a first group of weight and a second group of weight; performing mouth shape image proportion analysis on the video sequence, and obtaining a third group of weights and a fourth group of weights in combination with vocabulary analysis of historical voice data of the user; and fusing the four groups of mouth shapes and audio weights, and performing voice content recognition extraction on the audio and video sequence to obtain final voice data. The problems that in the prior art, dependence on voice data extraction is single, the voice data extraction quality is poor, and personalized adaptation is lacked are solved.

The invention provides a foreign language slang teaching device with a fuzzy recognition function and a use method, and relates to a foreign language teaching device. Comprising a mounting base, a learning machine body mounted on the mounting base and a supporting mechanism for supporting the mounting base, and one end of the learning machine body is hinged to the top of the mounting base through a pin shaft. According to the device, by driving the earphones in the storage box, the headphones can transmit speech during oral training to the speech recognition module in the learning machine body through the microphones on the earphones for recognition, and sliding of the sliding frame on the mounting base is adjusted through the adjusting mechanism; the learning machine body is pushed to turn over under the action of the connecting rod, the observation and use angle of the touch screen display is adjusted, students can observe whether their mouth shapes are correct or not through the arranged screen when doing oral language practice, the students are further assisted in oral language learning, and the teaching effect is improved.

The present application provides a method, system, device and medium for generating audio and video of a digital human, the method comprising: first, obtaining text information and an initial image; determining the emotion category corresponding to the text information based on an emotion analysis model; identifying the face area in the initial image to obtain the gender category corresponding to the digital human in the initial image; generating audio data corresponding to the digital human based on the emotion category and the gender category; generating an initial video of the digital human based on the audio data and the initial image; inputting the initial video of the digital human into a lip alignment model to obtain an output video; aligning the mouth shape of the digital human in the output video with the audio data. Using the emotion analysis model and image recognition technology, the emotion category of the text information and the gender category of the digital human in the initial image can be accurately identified, and a digital human video with emotional expression can be generated.

The invention discloses an automatic video translation system based on FFMPEG, TTS and Wav2Lip, and relates to the technical field of automatic translation processing, the automatic video translation system comprises a video processing unit, a translation synthesis unit and a mobile terminal, the video processing unit comprises a video input module, a video acquisition module, a preprocessing module, an audio extraction module and a video extraction module; the video input module is sequentially connected with the video acquisition module, the preprocessing module and the audio extraction module; the video extraction module is connected with the translation synthesis unit; according to the invention, face key points in a video frame are detected and tracked by using a computer vision technology through a mouth shape synchronization module, related voice features are extracted based on a face tracking result, and accurate synchronization of figure mouth shapes and voice contents in a translated video is ensured through technical means of image-to-image feedback circulation and the like; the translated video looks more natural and smooth, and the watching experience of audiences is improved.

The invention discloses a voice air conditioner control method and device, a voice air conditioner and a medium. The invention relates to the technical field of air conditioners. The method comprises the steps that dialect voice and a mouth shape video corresponding to a dialect instruction are obtained; processing the dialect voice to obtain acoustic features, and processing the mouth shape video to obtain visual features; fusing the acoustic features and the visual features to obtain multi-modal features; acquiring environment data, and determining an activity scene according to the environment data; and according to the multi-mode characteristics, the activity scene and a preset dialect instruction library, whether the voice air conditioner is controlled or not according to the dialect instruction is judged. According to the method, the dialect voice corresponding to the dialect instruction and the feature of the mouth shape video are fused to obtain the multi-mode feature, then the voice air conditioner is controlled according to the multi-mode feature, the activity scene determined by the environment data and the preset dialect instruction library, and the dialect recognition accuracy of the voice air conditioner is improved.

The invention discloses a digital human multi-modal interaction method and related equipment, and the method comprises the steps: responding to the multi-modal input of a user, and obtaining the question information of the user; processing the question information by using a preset response mechanism to obtain answer information; converting the answer information into voice data; and when the voice data is played, driving the digital person to synchronously display the expression and mouth shape corresponding to the voice data by using an expression and mouth shape synchronization technology so as to complete interaction of the digital person. The problem that an existing system is low in flexibility and expansibility is solved.

According to the multi-model-based user emotion monitoring intervention method and device and the computer program product provided by the invention, the communication style of the target object is simulated by training multiple types of models, a virtual image is synchronously generated to match mouth shapes, expressions and actions, and interaction with more emotion support can be provided for the user. Meanwhile, an emotion analysis model is introduced to carry out emotion recognition, early warning and risk level analysis, and when emotion abnormity is recognized, a pacific dialogue is carried out based on a second model obtained by learning a psychological intervention strategy library and a psychological knowledge graph. And sudden emotional abnormal conditions can be detected and responded in real time, and professional pacifying dialogues can be intervened.

The embodiment of the invention relates to a video generation method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the filling processing of each frame of image of an initial video, and obtaining a first image; obtaining a second image from the initial video, and obtaining a target audio; extracting audio features from the target audio, extracting first image features from the first image, and extracting second image features from the second image; performing alignment operation on the first image feature and the second image feature; performing spatial deformation on the second image feature according to the audio feature and the aligned second image feature; generating a mouth shape image according to the deformed second image feature and the first image feature; and generating a target video according to each mouth shape image and the target audio. Therefore, the target video with highly synchronous mouth shape and voice content can be generated while the character identity features are kept, and the naturalness and visual reality of mouth shape and voice matching generation are improved.

The invention provides a digital human question-answering system, and the system comprises a voice receiving and processing module which is used for receiving the voice input of a user in real time, splitting the input voice, extracting the acoustic features of each frame of voice in real time, and caching the acoustic features of continuous frames to form a streaming data queue; the voice feature reasoning module is used for inputting the streaming data queue into a pre-training question and answer model and outputting a question and answer text result and a voice synthesis instruction in real time; and the digital human synchronous driving module is used for generating a synchronous voice signal based on the question and answer text result and the voice synthesis instruction, mapping the synchronous voice signal to a mouth shape mapping library and an action template library, generating a mouth shape sequence and a limb action sequence matched with the voice time sequence, and completing digital human broadcasting. According to the method, user text or voice input is received, the question and answer result is returned after model reasoning, and the digital person is driven to complete voice broadcasting.