437 results about "Mouth shape" patented technology

The invention provides a digital human interaction method and system based on naked eye 3D visualization, and the method comprises the steps: obtaining query content provided by a user in a multi-modal manner, carrying out the semantic understanding and intention analysis of the query content through a natural language processing technology, and obtaining user intention data; performing matching search in a preset content database according to the user intention data, and if a highly related answer template or video clip exists, calling preset content and feeding back the preset content to the user; according to different answer sources, corresponding digital human video visualization schemes are adopted, a digital human driving engine is used for matching mouth shapes, expressions and actions, and matched digital human videos are synthesized; and displaying the digital human video through a transparent LCD screen.

The invention provides a real-time video translation and audio and picture synchronization method and system based on a multi-modal large model, and relates to the technical field of video translations, and the method comprises the steps: obtaining a source video; extracting the source video based on the multi-modal large model to obtain a multi-modal feature; fusing the multi-modal features through a cross-modal attention mechanism to generate a context semantic vector; translating into a target language text in real time based on the context semantic vector, and processing the translated language text based on the multi-modal features to obtain a translated language sound source; and performing mouth shape adjustment on the source video based on the translation language sound source, and merging the translation language sound source and the mouth shape animation video to obtain a real-time translation video with synchronous sound and picture. According to the method, the limitation of traditional single-modal translation is broken through, and the semantic accuracy of translation is remarkably improved by dynamically aligning the context information through the multi-modal features in combination with a cross-modal attention mechanism.

The invention relates to the technical field of subtitle translation, in particular to a multi-modal time sequence alignment AI video translation method and system, and the method comprises the steps: 1, carrying out the multi-modal analysis of a to-be-translated video, and obtaining audio separation data, voiceprint feature data and visual time sequence data; 2, performing cross-language translation and context optimization on the basis of the voice of the audio separation data to generate a target language text, and synthesizing target language voice retaining the original voice color in combination with the voiceprint feature data and the target language text; generating a mouth shape animation matched with the target language voice based on the lip key point data and the limb action time sequence data; and step 3, performing four-dimensional alignment on the target language voice, the translated text, the mouth shape animation and the limb action sequence through a cross-modal time sequence encoder, and dynamically adjusting the layout of the bilingual subtitles to adapt to a video picture. According to the method and the device, multi-mode synchronization can be taken into consideration during video translation, so that the body actions such as voice, subtitles and mouth shapes are kept aligned.

The invention discloses a mouth shape synchronization method and system for a 3D digital human. The method comprises the following steps: screening an audio file; carrying out ASR identification on the audio file to obtain a text expression corresponding to the audio file; performing alignment operation on the audio file and the corresponding text expression by using a multi-modal phoneme alignment tool to generate a phoneme time sequence; further generating a mouth shape animation time sequence according to the phoneme time sequence; classifying the Chinese phoneme mouth shapes, and creating BlendShape for various mouth shapes according to a classification result; carrying out weighted adjustment on the BlendShape weight of the corresponding mouth shape; and based on the mouth shape BlendShape after weight adjustment, using Unity to carry out modeling on a 3D task, and establishing a corresponding image model for each mouth shape. The method has the advantages that the vivid degree of the 3D digital human is remarkably improved, the mouth shape and voice synchronization of the digital human is more accurate and natural, and the immersion and substitution of audiences are enhanced.

The invention discloses a multi-mode speech enhancement system based on audio and video, and the system comprises a data collection module which collects target audio, background noise and interference sound through a microphone; a camera is used for synchronously collecting environment visual information, including facial expressions and mouth shapes of speakers, and noise and interference sounds are normalized to ensure that the noise and the interference sounds are matched with the dynamic range of a target audio; the data analysis module is used for preprocessing the collected audio, including noise suppression and speech enhancement, and improving the speech definition by applying a deep learning model; through the noise reduction and voice enhancement technology, the interference of background noise can be effectively reduced, and the voice is clearer. Thus, audiences can more easily understand speech contents, especially in noisy environments, such as public transport means or busy streets; and the voice recognition accuracy is improved, and the quality of the voice signal can be improved.

The invention discloses a video translation method and system based on artificial intelligence. The method relates to the technical field of video translation and comprises the following steps of original sound track extraction, target AI speaker adaptation, AI dubbing generation and mouth shape synchronization and video synthesis. According to the method, independent audio and video streams are obtained by adopting an audio and video separation technology, and multiple original sound tracks are extracted through a voice separation model; matching or generating an adaptive target AI speaker module in a preset tone library; converting the original language voice into a text, translating the text into a target language text, and synthesizing an AI dubbing audio track in combination with a target AI speaker module; and finally, the independent video stream and the multi-AI dubbing audio track are input into the mouth shape synchronization model to output a translated video, so that the timbre fitting degree, the voice quality and the voice consistency of the same speaker of AI dubbing are improved, and meanwhile, the resource utilization rate of video translation and the processing efficiency under batch tasks are improved. The problem that in the prior art, video translation is low in quality and efficiency is solved.

The invention relates to the technical field of artificial intelligence and multimedia processing, and particularly provides an intelligent dialogue method and system based on a digital human, and the method comprises the following steps: S1, an input stage: a user inputs a question or an instruction through a text or voice; s2, the processing stage comprises large model generation reply, TTS model conversion and audio driving mouth shape; and S3, the output stage comprises video generation, stream pushing and display. Compared with the prior art, the method has the advantages that the reply can be generated through the large model, and the TTS model and the audio-driven mouth shape model are combined, so that more natural and anthropomorphic dialogue interaction is realized, and the user experience is improved.

The invention provides a mouth shape generation method and device for voice driving, and relates to the technical field of natural language processing. The method comprises the following steps: based on an ffmpeg tool, carrying out image processing according to original video data to obtain spliced frame image data and facial feature points; according to the original audio data, performing feature extraction through a deep audio feature extractor to obtain audio features; performing feature fusion through an audio-video sequence feature fusion device to obtain fusion features; performing video generation through a lip action generator according to the facial feature points and the fusion features to obtain synthesized video data; optimizing a lip action generator according to the original video data and the synthesized video data; and performing video generation based on the depth audio feature extractor, the audio-video sequence feature fusion device and the optimized lip action generator to obtain target synthesized video data. The invention relates to a mouth shape generation method aiming at voice driving, which is high in video resolution and sufficient in facial texture detail reservation.

The invention discloses a mouth shape animation generation method and device, a model training method and device, equipment and a medium. The method comprises the following steps: acquiring text information, a virtual object identifier and an emotion identifier; converting the text information into a phoneme sequence, converting the virtual object identifier into a virtual object code, and converting the emotion identifier into an emotion code; inputting the phoneme sequence, the virtual object code and the emotion code into a coding model for feature combination to obtain a comprehensive feature for representing common features of the voice and the mouth shape; and decoding the comprehensive feature to obtain a voice signal corresponding to the text information and a matched mouth shape signal, so as to generate a mouth shape animation by using the voice signal and the mouth shape signal. According to the scheme, the input text information, the virtual object identifier and the emotion identifier are processed through the coding model to obtain the comprehensive characteristics representing the common characteristics of the voice and the mouth shape, and then the voice signal and the mouth shape signal corresponding to the same text information are obtained to generate the mouth shape animation, so that the working efficiency of generating the mouth shape animation through the text information is improved.

One or more embodiments of the invention provide a digital human rendering method and device, a storage medium and a program product. The digital human rendering method comprises the following steps: analyzing a target voice stream matched with a to-be-rendered digital human to obtain a phoneme sequence and voice rhythm characteristics; a mouth shape control parameter sequence corresponding to the phoneme sequence is determined, all mouth shape control parameters in the mouth shape control parameter sequence are in one-to-one correspondence with all phonemes in the phoneme sequence, and a mouth shape control curve is generated based on the mouth shape control parameter sequence; performing rhythm alignment processing on the mouth shape control curve according to the voice rhythm characteristics, executing rasterization conversion, and generating a mouth shape image block sequence synchronized with the target voice stream; and synthesizing each mouth shape image block in the mouth shape image block sequence with other image contents of the digital human to generate a digital human image frame sequence.

The invention belongs to the technical field of voice processing, and discloses an intelligent cockpit multi-mode voice interaction system and method, and the system comprises a voice triggering unit which collects the environment audio and video information in a cockpit, judges whether to enter a voice interaction mode or not through combining with the environment perception parameters in a vehicle, and sends the voice interaction mode to the vehicle; when the voice interaction triggering condition is satisfied, generating a voice interaction input signal matched with the current environment; the mouth shape analysis unit is used for carrying out acoustic feature extraction on the voice interaction input signal, synchronously analyzing the lip motion trail of the driver in the video information, establishing a corresponding relation between voice phonemes and mouth shape motion, and forming a joint analysis feature; the candidate generation unit is used for carrying out segmented alignment on the joint analysis features and constructing a continuous multi-modal fragment sequence; performing time synchronization on the multi-modal fragment sequence, and projecting the multi-modal fragment sequence to a predefined intention space to obtain a candidate intention set containing different candidate intentions; and the man-machine interaction experience of the intelligent cabin is improved.

The invention discloses a 2D government affair digital human real-time interaction method and system based on an AI painting technology, and the method comprises the steps: making a digital human image based on a modeling mode combining AI painting and a Live2D technology; acquiring an audio input by a user in real time, and identifying the audio input by the user to obtain an audio identification text and a reply text; according to the reply text, synthesizing and outputting a result audio based on a half-stream speech synthesis method of dependency syntactic analysis; and performing mouth shape synchronous real-time driving and audio broadcasting of the digital human image on the manufactured digital human image based on the result audio. According to the invention, the low-cost construction and deployment of the digital human are realized based on the combination of AI drawing and Live2D technology; when system deployment is carried out on the digital people, a semi-streaming speech synthesis scheme based on dependency syntactic analysis is adopted, and the interaction real-time performance and fluency of government affair digital people belonging to service digital people are improved.

The invention discloses a skin color and facial form variable expression robot. The eyebrow mechanism and the eye mechanism are fixedly connected, the eyebrow mechanism, the eye mechanism and the mouth mechanism are all connected with the deformation mechanism, the eyebrow mechanism, the mouth mechanism and the deformation mechanism are all installed on the head shell mechanism, the neck mechanism is fixedly installed at the bottom end of the mouth mechanism, and the skin color changing mechanism is attached to the head shell mechanism and used for achieving replacement of the skin color of the robot. The deformation mechanism is mainly driven by a worm and gear structure, and deformation of the eyebrow and the mouth of the robot is achieved through the worm and gear structure. According to the machine, self-adaptive face transformation according to human face characteristics is achieved, the face shape, the skin color and the pupil color can be rapidly changed according to different application scenes, rich mouth shapes can be achieved, and the requirement for real-time dialogue is met.

The invention relates to the technical field of rehabilitation training, in particular to a multi-logic interaction system for speech disorder rehabilitation training, which comprises a voice accuracy analysis module, a voice mouth shape comparison module, a training item distribution module, a training difficulty adjustment module and a review interval optimization module. By capturing and analyzing the pronunciation information of the user in real time, pronunciation errors can be accurately recognized and corrected, the pertinence and efficiency of voice training are enhanced, a patient can visualize the pronunciation errors by combining voice data with a mouth shape image, and meanwhile, by comparing standard pronunciation images, the pronunciation errors can be accurately recognized and corrected. According to the method, integration of voice and visual information is enhanced, each patient can obtain a rehabilitation scheme meeting target requirements and ability through intelligent distribution of training content and time and dynamic adjustment of training difficulty, training distribution and difficulty adjustment are optimized, a user can train in a proper rhythm, and the training efficiency is improved. And the individuation level and the overall effect of speech disorder rehabilitation training are improved.

The embodiment of the invention provides a digital human interaction video generation method and interaction equipment. According to the method, after a generation device obtains a target voice, a basic video and a mouth shape image set can be obtained from a pre-constructed material library, a target mouth shape image sequence matched with the phoneme change condition in the target voice is generated based on the target voice and the mouth shape image set, and then the target mouth shape image sequence is obtained. And fitting the target mouth shape image sequence to the mouth area of the digital human in the basic video, thereby obtaining a target video.

A device for creating digital persona, which includes a data collection module responsible for collecting personality data of a target object, a personality training module utilizing a large language model and the personality data to train and generate a virtual personality model with personality characteristics of the target object, thereby generating a virtual personality consistent with the personality characteristics of the target object, an appearance video generation module and voice generation module respectively utilizing face replacement and voice cloning technologies to extract the pictures and sounds from the personality data to generate a virtual personality model with the appearance and voice characteristics of the target object, a lip synchronization module, utilizing lip synchronization technology to ensure that the mouth shape and voice of the digital persona are synchronized, and an interactive module, providing an interactive interface that allows users to interact with the virtual personality and receive responses from it.

The invention relates to a voice-driven high-timeliness digital population type synthesis algorithm, and the algorithm improves the digital population type synthesis effect through introducing lip-reading expert and a pre-trained video automatic encoder as supervision, carefully designing a feature fusion module, training a loss function and other skills. The speed of digital population type synthesis is improved through skills such as channel-by-channel convolution and linear attention. And the device has the functions of controllable mouth opening amplitude and multi-language support.

The invention discloses a bionic robot control method and device, equipment and a storage medium, and relates to the technical field of bionic robots, and the method comprises the steps: obtaining a target character through a speech synthesis model, determining a phoneme sequence based on the target character, coding the phoneme sequence, and carrying out the preset convolution operation of the obtained coded sequence, a corresponding phoneme feature map is obtained; determining voice data based on the phoneme feature map through a voice synthesis model, and determining a voice time sequence based on the voice data; determining a target mouth shape based on the voice time sequence through a voice synthesis model, and determining a mouth shape time sequence based on the voice time sequence and the target mouth shape; and when the voice data is played, a motor of the mouth is controlled to perform corresponding motion based on the mouth shape time sequence, so that the bionic robot generates a corresponding mouth shape. Therefore, the action of the mouth of the bionic robot can be accurately controlled while the bionic robot plays the voice.

The invention provides a stylized digital human video generation method, electronic equipment and a storage medium. The method belongs to the technical field of image processing, and comprises the following steps: receiving a stylized digital human video generation instruction, the stylized digital human video generation instruction comprising a user photo, a target stylized type and a dubbing file; converting the user photo into a target stylized image according to the target stylized type; the target stylized image and the dubbing file are input into a pre-trained mouth shape driving model, the pre-trained mouth shape driving model extracts identity features of the target stylized image and audio features of the dubbing file, and a stylized digital human video is generated according to the identity features and the audio features; and obtaining a stylized digital human video output by the pre-trained mouth shape driving model, wherein the mouth shape driving of the stylized digital human video is synchronized with the dubbing file. According to the invention, stylized digital human videos which are more personalized, real-time and high in quality can be provided.

The invention discloses a multiple children training sensing system, and relates to the technical field of children education training, and the system comprises the following components: hardware architecture design, a data processing and fusion algorithm, a personalized training feedback mechanism and a cloud service and data analysis platform. The high-definition camera, the high-precision motion sensor and the high-sensitivity microphone array hardware equipment are integrated, the data processing and fusion algorithm is combined, comprehensive and accurate sensing of the training process of children is achieved, the high-definition camera can capture the facial expression and mouth shape changes of the children, and the high-precision motion sensor and the high-sensitivity microphone array hardware equipment are combined. The motion sensor monitors the motion posture, the motion track and the strength distribution of the child in real time, the high-sensitivity microphone array can capture the nuisance of the sound production of the child, and the multi-modal data acquisition mode enables the system to more accurately recognize the key behavior characteristics, the emotional state and the skill mastering condition of the child in the training process, so that the training efficiency is improved. And a solid data basis is provided for personalized training.

The invention discloses a human face mouth shape coordination generation method based on geometric constraints, and relates to the technical field of image generation, and the method comprises the steps: 1, synchronously generating face vertex index tables corresponding to mouth opening and closing spatial form matrixes one by one; 2, based on the facial vertex index table, mapping the lip and gum combined boundary constraint set to a unified coordinate system to form a geometric prior constraint tensor; 3, performing multi-modal topological cascade reversible mixed transformation on the geometric prior constraint tensor to obtain a high-dimensional tooth texture candidate tensor and constraint consistency confidence spectrum; and step 4, using the high-dimensional tooth texture candidate tensor and the constraint consistency confidence spectrum as input, completing anti-aliasing, illumination compensation and gamma correction on a unified rendering pipeline, and finally obtaining a mouth shape coordination synthetic frame sequence. According to the method, the sense of reality, geometric consistency and time sequence coherence of tooth textures in the mouth shape synthesis process are effectively improved, and the problem that lips and teeth are not coordinated in an existing method is solved.

The embodiment of the invention provides a video generation method and system based on AI voice cloning and mouth shape synchronization, and the method comprises the steps: carrying out the fusion of voiceprint features of an input video and an input text through a voice synthesis model after the input video and the input text are obtained, so as to generate a natural voice; analyzing lip key points of the input video by using a lip shape displacement model, and matching lip shape change data according to the lip key points and natural voice; and generating an output video according to the input video and the lip shape change data. According to the method, the phoneme duration prediction of the speech synthesis model and the lip displacement model can be coupled through time sequence convolution, so that the mouth shape of the output video is matched with the speech content, lightweight video restoration is realized by redrawing the lip region, the dynamic response to the input text modified by the user in real time is supported, and the response efficiency is improved.

The invention provides a vehicle-mounted voice recognition method and device, electronic equipment and a storage medium, and belongs to the technical field of voice recognition, and the method comprises the steps: carrying out the fusion of a voice recognition result and a lip language recognition result of a cab, obtaining a multi-modal recognition result, enabling the voice recognition result to be obtained through the recognition of a voice signal collected in real time, and enabling the lip language recognition result to be a lip language recognition result; the lip language recognition result is obtained by recognizing a mouth shape image collected in real time; analyzing the multi-modal recognition result to obtain a voice source of each voice and an intention corresponding to each voice; and based on the voice source and intention of each voice, respectively generating a vehicle control instruction. According to the invention, the voice recognition result and the lip language recognition result are combined to distinguish the voice from different sources, so that the main driver and the co-driver can respond to the voice instructions when sending the voice instructions at the same time, and the requirements of the main driver and the co-driver can be met at the same time.

The embodiment of the invention provides a digital human video generation method and device, a storage medium and a program product. In the embodiment of the invention, the original reasoning environment of the target digital human reasoning model is changed, reasoning operation at the server side is changed into reasoning at the mobile terminal, and the reasoning speed of the target digital human model is improved. Besides, a plurality of target face area images with continuously changed mouth forms are obtained through reasoning based on a plurality of audio feature segments corresponding to the audio data by using a target digital human reasoning model, and a digital human action picture is generated according to the plurality of target digital human images with continuously changed mouth forms. And driving the digital person in the digital person action picture to play the audio data to obtain a target digital person video, thereby not only improving the adaptation degree of the mouth shape and the audio content, but also reducing the delay and lagging problems of playing the digital person video, and improving the playing quality of the target digital person video.

The embodiment of the invention relates to the field of computer technology application, and discloses a 3D digital human voice and mouth shape action synchronization method and device, equipment and a medium, and the method comprises the steps: obtaining voice data, mouth shape data and limb data, so as to obtain a voice data queue, a mouth shape data queue and a limb data queue; according to the frame information in the mouth shape data queue and the frame information in the limb data queue, adjusting the volume of voice data sent by the voice data queue at a time; playing each frame of data in the mouth shape data queue and each frame of data in the limb data queue frame by frame; when the total amount of the voice data sent by the voice data queue is greater than or equal to a preset voice data volume threshold value, playing voice corresponding to the voice data sent by the voice data queue; according to the 3D digital human voice and mouth shape motion synchronization method disclosed by the invention, the technical problem of low authenticity caused by asynchronous motion and voice of the digital human is solved, so that the motion and voice of the digital human are synchronized, and the authenticity of the digital human is improved.

The present invention relates to the field of computer vision technology, and discloses a virtual digital human lip synchronization optimization method, device, equipment and storage medium. The virtual digital human lip synchronization optimization method includes: obtaining the target audio segment to be output by the virtual digital human at the next moment; judging whether the target audio segment belongs to the audio type to be processed; if the target audio segment belongs to the audio type to be processed, then based on a preset lip synchronization optimization strategy, generating a 3D human face mouth shape parameter frame sequence corresponding to the target audio segment; based on the 3D human face mouth shape parameter frame sequence, generating a corresponding 3D human face mouth shape image frame sequence and rendering it into the virtual digital human. The present invention can adapt to various audio types and improve the fluency and naturalness of the virtual digital human's mouth shape under different audio types.

The invention discloses a 2D digital human generation method and device for a multi-element driving source, and the method comprises the steps: obtaining the driving sources of various elements, i.e., text, voice and video, and a to-be-driven virtual image; a lip-shaped driving source and an action driving source are obtained by using algorithms such as text-to-voice, audio feature extraction and video preprocessing in the driving source analysis module; migrating a posture action and an expression action in the action driving source to a to-be-driven virtual image to obtain a driving result synchronous with the action of the action driving source; the mouth shape in the action driving result is replaced with a new mouth shape according to the lip-shaped driving source, and a result synchronous with the lip-shaped driving source is obtained; and synthesizing the fused audio corresponding to the text and the voice signal with the dual-drive result of the action and the lip shape to obtain a 2D digital human generation result with sound and picture matching. According to the invention, a multi-element driving source is supported to carry out controllable 2D digital human drive generation.

The invention relates to the field of digital people, and discloses a method and a device for synchronizing voice and mouth shape of a digital person, and electronic equipment. The method comprises the steps of obtaining natural language information, converting the natural language information into character information, determining a video frame and an audio frame based on the character information, and controlling the digital person to speak when the first frame time of the audio frame is the same as the first frame time of the video frame, so that the voice and mouth shape of the digital person can be kept consistent.

The invention discloses an instrument of an intelligent breathing and motion capture system based on six-character table health maintenance, and relates to the technical field of intelligent health maintenance equipment, the instrument comprises a hardware part and a software part; the hardware part comprises a mouth shape visual capture assembly, an action force and angle sensing assembly, an audio acquisition and feedback assembly and a main control and power supply assembly; the software part comprises a data acquisition module, a signal preprocessing module, a parameter analysis and judgment module, a real-time feedback module and a data storage and management module; according to the invention, through integration of the mouth-shaped visual capture assembly, the action force and angle sensing assembly, the audio acquisition and feedback assembly and the master control and power supply assembly, synchronous acquisition and processing of multi-source data are realized; the mouth shape change, the body movement and the audio pronunciation of the user can be accurately captured in real time, and powerful technical support is provided for standardized practice of six-character table health maintenance.

The invention provides a 3D interactive digital human generation method and device and a customer service project system, and relates to the technical field of computers. The method comprises the following steps: logging in a customer service project system, and creating an initial 3D character model in response to a character role and virtual scene selection operation; according to the virtual scene, responding to editing operation on the initial appearance form and the initial face form, triggering an integrated large language model, and when login succeeds, connecting the virtual scene with the integrated large language model; according to the virtual scene, in response to a selection operation on a voice style of the initial 3D character model, adjusting the initial appearance form according to the voice style to obtain a target appearance form; performing mouth shape matching on the voice style by using the voice model, and combining the matched mouth shape with the initial face shape to obtain a target face shape; project publishing is carried out according to the target appearance form and the target face form, the 3D interactive digital human is displayed, and voice service and mouth shapes can be synchronized into a whole.