177 results about "Mouth shape" patented technology

The invention discloses a video translation method and system based on artificial intelligence. The method relates to the technical field of video translation and comprises the following steps of original sound track extraction, target AI speaker adaptation, AI dubbing generation and mouth shape synchronization and video synthesis. According to the method, independent audio and video streams are obtained by adopting an audio and video separation technology, and multiple original sound tracks are extracted through a voice separation model; matching or generating an adaptive target AI speaker module in a preset tone library; converting the original language voice into a text, translating the text into a target language text, and synthesizing an AI dubbing audio track in combination with a target AI speaker module; and finally, the independent video stream and the multi-AI dubbing audio track are input into the mouth shape synchronization model to output a translated video, so that the timbre fitting degree, the voice quality and the voice consistency of the same speaker of AI dubbing are improved, and meanwhile, the resource utilization rate of video translation and the processing efficiency under batch tasks are improved. The problem that in the prior art, video translation is low in quality and efficiency is solved.

One or more embodiments of the invention provide a digital human rendering method and device, a storage medium and a program product. The digital human rendering method comprises the following steps: analyzing a target voice stream matched with a to-be-rendered digital human to obtain a phoneme sequence and voice rhythm characteristics; a mouth shape control parameter sequence corresponding to the phoneme sequence is determined, all mouth shape control parameters in the mouth shape control parameter sequence are in one-to-one correspondence with all phonemes in the phoneme sequence, and a mouth shape control curve is generated based on the mouth shape control parameter sequence; performing rhythm alignment processing on the mouth shape control curve according to the voice rhythm characteristics, executing rasterization conversion, and generating a mouth shape image block sequence synchronized with the target voice stream; and synthesizing each mouth shape image block in the mouth shape image block sequence with other image contents of the digital human to generate a digital human image frame sequence.

The invention belongs to the technical field of voice processing, and discloses an intelligent cockpit multi-mode voice interaction system and method, and the system comprises a voice triggering unit which collects the environment audio and video information in a cockpit, judges whether to enter a voice interaction mode or not through combining with the environment perception parameters in a vehicle, and sends the voice interaction mode to the vehicle; when the voice interaction triggering condition is satisfied, generating a voice interaction input signal matched with the current environment; the mouth shape analysis unit is used for carrying out acoustic feature extraction on the voice interaction input signal, synchronously analyzing the lip motion trail of the driver in the video information, establishing a corresponding relation between voice phonemes and mouth shape motion, and forming a joint analysis feature; the candidate generation unit is used for carrying out segmented alignment on the joint analysis features and constructing a continuous multi-modal fragment sequence; performing time synchronization on the multi-modal fragment sequence, and projecting the multi-modal fragment sequence to a predefined intention space to obtain a candidate intention set containing different candidate intentions; and the man-machine interaction experience of the intelligent cabin is improved.

The invention discloses a bionic robot control method and device, equipment and a storage medium, and relates to the technical field of bionic robots, and the method comprises the steps: obtaining a target character through a speech synthesis model, determining a phoneme sequence based on the target character, coding the phoneme sequence, and carrying out the preset convolution operation of the obtained coded sequence, a corresponding phoneme feature map is obtained; determining voice data based on the phoneme feature map through a voice synthesis model, and determining a voice time sequence based on the voice data; determining a target mouth shape based on the voice time sequence through a voice synthesis model, and determining a mouth shape time sequence based on the voice time sequence and the target mouth shape; and when the voice data is played, a motor of the mouth is controlled to perform corresponding motion based on the mouth shape time sequence, so that the bionic robot generates a corresponding mouth shape. Therefore, the action of the mouth of the bionic robot can be accurately controlled while the bionic robot plays the voice.

The embodiment of the invention provides a video generation method and system based on AI voice cloning and mouth shape synchronization, and the method comprises the steps: carrying out the fusion of voiceprint features of an input video and an input text through a voice synthesis model after the input video and the input text are obtained, so as to generate a natural voice; analyzing lip key points of the input video by using a lip shape displacement model, and matching lip shape change data according to the lip key points and natural voice; and generating an output video according to the input video and the lip shape change data. According to the method, the phoneme duration prediction of the speech synthesis model and the lip displacement model can be coupled through time sequence convolution, so that the mouth shape of the output video is matched with the speech content, lightweight video restoration is realized by redrawing the lip region, the dynamic response to the input text modified by the user in real time is supported, and the response efficiency is improved.

The invention discloses an instrument of an intelligent breathing and motion capture system based on six-character table health maintenance, and relates to the technical field of intelligent health maintenance equipment, the instrument comprises a hardware part and a software part; the hardware part comprises a mouth shape visual capture assembly, an action force and angle sensing assembly, an audio acquisition and feedback assembly and a main control and power supply assembly; the software part comprises a data acquisition module, a signal preprocessing module, a parameter analysis and judgment module, a real-time feedback module and a data storage and management module; according to the invention, through integration of the mouth-shaped visual capture assembly, the action force and angle sensing assembly, the audio acquisition and feedback assembly and the master control and power supply assembly, synchronous acquisition and processing of multi-source data are realized; the mouth shape change, the body movement and the audio pronunciation of the user can be accurately captured in real time, and powerful technical support is provided for standardized practice of six-character table health maintenance.

The embodiment of the invention discloses a voice action synchronization method and device for a virtual character, equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the voice generation of an answer text, obtaining an answer voice sequence and the attribute information of each phoneme, determining a standard mouth shape identifier of the corresponding phoneme based on the phoneme identifier of each phoneme, and obtaining a voice action synchronization result; generating a mouth shape animation sequence based on the standard mouth shape identifier of each phoneme according to the time range of each phoneme; determining an emotional action sequence and an emotional action starting timestamp corresponding to the emotional label of the answer text, and generating a supplementary animation sequence based on the emotional action sequence and the emotional action starting timestamp; and synchronizing the answer voice sequence, the mouth shape animation sequence and the supplementary animation sequence according to the time sequence to obtain a synchronization relationship, and driving the virtual response character of the user based on the synchronization relationship, the mouth shape animation sequence and the supplementary animation sequence when the answer voice sequence is played, so that the voice action synchronization function of the virtual character is realized.

The invention discloses a voice air conditioner control method and device, a voice air conditioner and a medium. The invention relates to the technical field of air conditioners. The method comprises the steps that dialect voice and a mouth shape video corresponding to a dialect instruction are obtained; processing the dialect voice to obtain acoustic features, and processing the mouth shape video to obtain visual features; fusing the acoustic features and the visual features to obtain multi-modal features; acquiring environment data, and determining an activity scene according to the environment data; and according to the multi-mode characteristics, the activity scene and a preset dialect instruction library, whether the voice air conditioner is controlled or not according to the dialect instruction is judged. According to the method, the dialect voice corresponding to the dialect instruction and the feature of the mouth shape video are fused to obtain the multi-mode feature, then the voice air conditioner is controlled according to the multi-mode feature, the activity scene determined by the environment data and the preset dialect instruction library, and the dialect recognition accuracy of the voice air conditioner is improved.

The invention provides a digital human question-answering system, and the system comprises a voice receiving and processing module which is used for receiving the voice input of a user in real time, splitting the input voice, extracting the acoustic features of each frame of voice in real time, and caching the acoustic features of continuous frames to form a streaming data queue; the voice feature reasoning module is used for inputting the streaming data queue into a pre-training question and answer model and outputting a question and answer text result and a voice synthesis instruction in real time; and the digital human synchronous driving module is used for generating a synchronous voice signal based on the question and answer text result and the voice synthesis instruction, mapping the synchronous voice signal to a mouth shape mapping library and an action template library, generating a mouth shape sequence and a limb action sequence matched with the voice time sequence, and completing digital human broadcasting. According to the method, user text or voice input is received, the question and answer result is returned after model reasoning, and the digital person is driven to complete voice broadcasting.

The invention provides a voice recognition and voice synthesis optimization method and system based on a large model, and the method comprises the steps: extracting lip motion features, lip feature timestamps and audio feature timestamps through obtaining a real-time voice input signal and an image frame sequence of a user face region, so as to generate an initial space-time offset sequence; generating a dynamic offset compensation parameter sequence based on the initial space-time offset sequence in combination with a reference alignment template in the voice and vision synchronization data set; and performing joint processing on the dynamic offset compensation parameter sequence, the lip motion characteristics and the real-time voice input signal by using a large model to reconstruct a target voice segment, generating a corrected phoneme sequence in combination with the lip motion characteristics, retrieving a mouth shape parameter group corresponding to the corrected phoneme sequence from a phoneme mouth shape mapping rule base, and performing mouth shape correction on the mouth shape parameter group. To generate a voice waveform in phase synchronization with the lip motion; according to the invention, the naturalness and immersion of man-machine interaction and the robustness in a voice missing or delay scene are improved.

One or more embodiments of the invention provide a digital human rendering method and device, a storage medium and a program product. The digital human rendering method comprises the following steps: analyzing a target voice stream matched with a to-be-rendered digital human to obtain a phoneme sequence and a voice feature vector of each phoneme in the phoneme sequence; obtaining a pre-stored mouth shape vector library, wherein the mouth shape vector library is used for storing a mapping relationship between the voice feature vectors of different phonemes and mouth shape control parameters; based on the voice feature vector of each phoneme in the phoneme sequence, a mouth shape control parameter sequence is retrieved from a mouth shape vector library, and each mouth shape control parameter in the mouth shape control parameter sequence is in one-to-one correspondence with each phoneme in the phoneme sequence; and executing a digital human rendering task based on the mouth shape control parameter sequence.

The invention relates to the field of voice signal processing, in particular to a multi-language interactive learning system based on voice recognition, which is characterized in that sound waves and mouth shape images are synchronously acquired and discretized by the system, and cross-modal coding is formed after alignment; secondly, the code is injected into a micro-ring photon reserve network through phase modulation to unfold time sequence characteristics, the code is mapped into a quaternion graph to be embedded, and segment boundaries are extracted through a diffusion-pulse coupling method; then, according to a fixed field sequence, encapsulating the quaternion graph embedding and segmentation data into an object mark prompt, inputting the object mark prompt into a low-rank adaptive language model, and generating semantic segmentation data and text transcription; and finally, the adaptive learning module implements sparse gradient updating on the low-rank weight and pulse network by using an integer fractal hash exclusive-or difference mask, and adopts support-query element learning for synchronous iteration after user clarification. According to the system, low-power-consumption, high-precision and second-level accent self-adaptive multi-language voice interaction is realized on the end side.

The present application provides a video parameter adjustment method and apparatus. The method comprises: acquiring a first video segment, the first video segment comprising a digital person, and the digital person being generated by a neural network model; determining a video quality monitoring index on the basis of the first video segment, the video quality monitoring index comprising at least one of the following: the accuracy of the mouth shape of the digital person, the degree of synchronization between the picture and sound of the digital person, and the degree of consistency of the figure of the digital person; and determining a parameter of the neural network model on the basis of the video quality monitoring index, the parameter being used for generating the digital person in a second video segment. In the described method, a quality monitoring index related to a digital person in a livestreaming video can be monitored and corresponding feedback adjustment settings are executed, thereby improving the quality of the livestreaming video of the digital person.

The invention is suitable for the technical field of real-time audio and video communication and digital human interaction, and provides a real-time digital human audio and video synchronization method and device based on TRTC. In the embodiment of the invention, for the voice data needing to be synchronized, the device firstly converts the voice data into the audio stream and uploads the audio stream to the cloud server, and then the cloud server generates the lip-shaped video stream according to the lip-shaped action video clip pre-generated based on the digital human image; the method comprises the following steps: receiving a lip-shaped video stream, generating a second audio stream of which the timestamp is synchronous with that of the lip-shaped video stream, returning the lip-shaped video stream and the second audio stream to equipment, and finally playing by the equipment according to the timestamps in the lip-shaped video stream and the second audio stream, thereby realizing audio and video synchronization. Therefore, the system delay of the real-time dialogue of the digital person is greatly reduced, the phenomenon that the sound and the mouth shape are not synchronous is effectively avoided, and the problem that low delay and high-precision mouth shape synchronization are difficult to consider when the digital person type and the voice are synchronous is solved.

The present disclosure relates to a device and method for generating avatar lip-sync animation based on multimodal biosignals, The device comprises a multimodal data collection unit configured to collect data including biosignal data including brain waves when a user imagines speaking and image data; a preprocessing unit configured to preprocess the multimodal data; a feature extraction unit configured to extract feature vectors including the user's biosignal feature and facial feature from the preprocessed multimodal data; an avatar generation unit configured to generate an avatar; a lip-sync reconstruction unit configured to predict the mouth shape and facial movement when the user imagines speaking by inputting the extracted feature vectors to a pre-prepared lip-sync reconstruction model; and a lip-sync animation implementation unit for implementing an avatar lip-sync animation by applying the mouth shape and facial movement predicted by the lip-sync reconstruction unit to the avatar generated by the avatar generation unit.

1. The name of the design product: the mouth shape of the nail paste. 2. The use of the design product: paste on the surface of the nail with adhesive, correct the ingrown nail with its own rebound tension, and treat paronychia. 3. The design points of the design product: in shape. 4. The picture or photo that best shows the design points: perspective drawing.

The invention discloses a conference privacy protection system and method based on visual sound field fusion. The conference privacy protection system comprises a visual acquisition module, an audio acquisition module, an audio playing module and a processing control module, wherein the visual acquisition module transmits mouth shape area three-dimensional coordinates, ear area three-dimensional coordinates and mouth motion states of participants to the processing control module; the processing control module generates a directional pickup instruction based on the mouth shape area three-dimensional coordinates, and sends the directional pickup instruction to the audio acquisition module to control the audio acquisition module to focus mouth shape area pickup; and meanwhile, a directional playback instruction is generated based on the three-dimensional coordinates of the ear region and is sent to the audio playing module to control the audio playing module to project sound beams to ears. Through visual guidance directional pickup, ultrasonic directional playback and audio and video fusion optimization, the risk of audio diffusion is reduced, so that the conference privacy protection effect is improved.

The utility model discloses a production spare part storage goods shelf which comprises a frame, the frame is arranged in a mouth shape, evenly-distributed installation grooves are formed in the inner wall of the frame, storage boxes are detachably arranged in the installation grooves, and a top plate and a bottom plate are arranged at the top and the bottom of the back face of the frame respectively. The utility model belongs to the technical field of storage goods shelves, and particularly relates to a storage goods shelf for production parts, which solves the problems that the storage goods shelf is inconvenient to take and place the parts at the middle position when the parts are taken and used, and limitation exists when the storage goods shelf is used.

The present disclosure provides a model training method, a speech recognition method and related devices, which are related to the technical field of data processing, and in particular to the technical fields of artificial intelligence, computer vision, speech technology, intelligent search and the like. The specific implementation scheme is as follows: inputting a mouth shape sample sequence into a mouth shape processing model to obtain a first dictionary code prediction result predicted based on the mouth shape sample sequence; determining a loss value based on the first dictionary code prediction result and a second dictionary code prediction result; the second dictionary code prediction result is determined based on a target text corresponding to the mouth shape sample sequence; adjusting model parameters of the mouth shape processing model based on the loss value to obtain a mouth shape interpretation model; wherein the mouth shape interpretation model is used to assist speech recognition. The mouth shape interpretation model trained by the method can adapt to any speech recognition network and can realize plug and play.

The invention provides a mouth shape synchronization method and device, a computing device cluster and a storage medium. The method comprises the following steps: associating an audio clip and a target image corresponding to the same object in audio data and media data, and according to the association relationship, performing mouth shape synchronization on the target image in the media data having the association relationship with the audio clip to obtain target media data. The picture of the mouth region in the target image of the same object in the target media data is the mouth shape image formed by driving based on the audio clip associated with the target image, even if the mouth regions of multiple objects, such as a multi-person dialogue scene, exist in the picture, the mouth shape synchronization accuracy can be improved, and the user experience is improved. And the problems of wrong die synchronization and disordered die synchronization are avoided.

The application discloses a kind of composite vision, hearing multimodal lip detection method and device and equipment.The method comprises: synchronously collecting voice data and face data;Multi-modal neural network model based on the voice data and face data is constructed;The multi-modal neural network model is trained using the way of double decoder joint optimization;According to the trained multi-modal neural network model, the speech of human body and face key point are predicted according to the face lip corresponding to the speech.This application uses key point detection algorithm to extract the face key point of face image and aggregate face features.On the other hand, considering the change of facial features and voice features, the change of mouth shape is represented by extracting more rich features by fusing the two, so that the vivid image is more lively.

This wearable device may comprise: a memory for storing instructions; one or more first cameras arranged in relation to the face of a user; one or more second cameras arranged in relation to the eyes of the user; one or more microphones; a display assembly including at least one display; and at least one processor. The wearable device may be caused to: display an avatar on the basis of generating first data for expressing a facial expression of the avatar using the one or more first cameras, generating second data for expressing a gaze of the avatar using the one or more second cameras, and generating third data for expressing a shape of a mouth of the avatar using the one or more microphones; identify an event that causes a change in a scheme for displaying the avatar while the avatar is being displayed on the basis of generating the first data, generating the second data, and generating the third data; and stop at least one of generating the first data, generating the second data, or generating the third data in response to the event.

The utility model belongs to the technical field of plastic bottle processing equipment, and discloses a bottle opening molding device for plastic bottle processing, which comprises an operation table, supporting legs are respectively and fixedly mounted at four corners of the bottom end of the operation table, and a fixed box is movably sleeved at the top end of the middle part of the front end of the operation table; the left side of the top end of the fixed box is movably sleeved with a circular shaft. Through the arrangement of the first clamping rod, the first gear, the first motor, the second clamping rod and the second gear, when the first motor is started, the second clamping rod and the second gear are driven by the rotating shaft to rotate, and the outer surface of the second gear is meshed with the outer surface of the first gear, so that the clamping effect is improved. Due to the fact that the first clamping rod and the second clamping rod rotate reversely under the interaction of the second gear and the first gear, the plastic bottle can be clamped and positioned through the cooperation of the first clamping rod and the second clamping rod, and the situation that the position of a bottle opening of the plastic bottle deviates due to the influence of inertia and the like is prevented.

The invention provides a deep learning-based human mouth shape and voice matching recognition method, which comprises the following steps of: recognizing respective speaking mouth shape characteristics of all people in a surrounding environment through an image, and recognizing a sound source position and an audio characteristic of the surrounding environment through a pickup array; according to the speaking mouth shape features, correcting the sound source position so as to separate the audio features to obtain the subordinate audio features of each person; performing deep learning on the audio features and the speaking mouth shape features of each person to obtain voice information sent by each person; carrying out background noise processing on the voice information to obtain the speaking voice and the text content of each person; personnel around the hearing aid in a noisy environment are identified in a visual and voice recognition mode, voice of the personnel is recorded and converted, the cocktail problem of the hearing aid in far-field recognition is effectively solved, voice interference is effectively restrained, and voice recognition accuracy and definition are improved.

The present application relates to a kind of breathing training systems using six-word formula, including user end hardware and cloud server, using the VR head-mounted integrated machine of user end hardware and elastic fabric waistband, guide and detect the breathing training condition of user, realize the accurate classification of abdominal breathing, chest breathing, mixed breathing by means of EMG, displacement, triple fusion calculation mode of flow, utilize mouth shape identification auxiliary phoneme confidence correction, accurately identify different exhale mode of six-word formula, introduce mouth shape identification, phoneme model, physiological sensing fusion multi-modal scoring mechanism, compared with traditional single chest and abdominal displacement monitoring scheme or single audio identification scheme, accurately evaluate the training quality of user from many aspects and comprehensive score, it is convenient for user to understand the improvement direction and progress space of self breathing training.

The utility model belongs to the field of medical instruments, and particularly relates to a buccal device suitable for nasopharyngeal carcinoma radiotherapy, which comprises a Y-shaped supporting rod, the Y-shaped supporting rod is of a hollow structure, the bottom of the Y-shaped supporting rod is in threaded connection with a balloon, a one-way valve is fixedly mounted in an inner cavity of the balloon, and the one-way valve is connected with the Y-shaped supporting rod. Two alveolar cavity bite pads are fixedly mounted on the inner side of the Y-shaped supporting rod; according to the tongue muscle training device, the Y-shaped supporting rod is matched with the tooth groove biting pad, the tongue muscle training device can be suitable for mouth shapes of different sizes, in the using process, the tooth groove biting pad is bitten by teeth and is not prone to sliding, the tongue of a patient can be efficiently positioned through matching of the shaking block and the tongue muscle training block, and tongue muscle training of the patient is facilitated; the problems that an existing oral device for radiotherapy is prone to sliding in the mouth of a patient, a tongue depressor wound with adhesive tape or a small medicine bottle needs to be stuffed into the mouth of the patient to achieve the effects of opening the mouth and fixing the tongue, the mode is not standard, and the effects of opening the mouth and fixing the tongue are not ideal are solved.

The utility model relates to the technical field of anti-falling wheelchairs, in particular to a wheelchair with an anti-falling function. The two sides of the seat plate are fixedly connected with a mouth-shaped frame, an anti-falling assembly is arranged at the position, close to the middle, of the bottom side of the seat plate, the anti-falling assembly comprises a fixing plate, the fixing plate is fixedly connected to the position, close to the middle, of the bottom side of the seat plate, sliding holes are formed in the positions, close to the two corners of the bottom side, of the fixing plate, and sliding rods are installed in the two sliding holes in a sliding mode. By means of the anti-falling assembly, when the gravity center of a patient faces backwards, the wheelchair can be supported by the fixing plate through the two sliding rods and the anti-falling wheels on the two connecting blocks, and the wheelchair can be prevented from falling backwards; and meanwhile, the two connecting blocks, the two anti-falling wheels and the two sliding rods can be driven to slide on the fixing plate by rotating the threaded rod, so that the two connecting blocks, the two anti-falling wheels and the two sliding rods can be adjusted, and the situation that pushing of the wheelchair is affected due to too much extension is avoided.

The application relates to the technical field of video data processing, and relates to an audio-video synchronous noise reduction method and system based on AI visual perception, which comprises the following steps: pre-processing audio data to obtain pre-processed audio data; performing time-frequency analysis on the pre-processed audio data to obtain a speech feature set and a background sound feature set; performing noise reduction on video data to obtain primary noise reduction video data, performing visual perception on the primary noise reduction video data to obtain a video feature set; performing time axis correction operation on the speech feature set based on a mouth shape feature according to the video feature set to obtain an updated time axis; performing active noise reduction operation on the progress correction audio data according to the updated time axis and a pre-constructed background sound adaptation degree sequence to obtain noise correction audio data; and performing merging operation on the noise correction audio data and the primary noise reduction video data to obtain synchronous noise reduction audio-video. The application can improve the clarity of images and sounds in a video.

The application discloses a speech-driven digital human mouth shape generation method based on latent space feature fusion, and belongs to the technical field of artificial intelligence and image synthesis; mainly improves the quality and time sequence continuity of a speech-driven digital human mouth shape generation image; the scheme of the application is that after modal encoding of speech audio and video images is carried out respectively, an image reconstruction process is guided by speech features in a latent space constructed by an image encoder, and a mouth shape change image frame sequence consistent with the speech features is generated; a complete process from user speech input to digital human response is realized, intelligent expression capability of the digital human in a human-computer interaction process is enhanced, and therefore more natural and intelligent digital human speech expression is realized.

This utility model discloses an integrated multifunctional intubation device with flushing, drainage, and drug administration functions. The device includes an intubation tube, a three-way drug administration flushing and drainage device, a connecting tube, and an anti-backflow negative pressure drainage ball. The inner surface of the intubation tube has supporting teeth to prevent compression closure. One end is a connecting section, and the working section's tail end is fish-mouth shaped with staggered flushing holes on its smooth surface. The outer circumference of the intubation tube has an "I"-shaped fixing ring. The three-way valve is connected to the anti-backflow negative pressure drainage ball through the connecting tube, allowing liquid in the intubation tube to flow into the drainage ball, preventing backflow and allowing liquid to flow out through the drain hole. The connections between the connecting tube, the intubation tube, and the three-way valve are all threaded and tightened. This device integrates multiple functions, has a reasonable structural design, and can effectively realize flushing, drainage, and drug administration operations, while also preventing blockage of the intubation tube due to compression and dislodgement due to external force.