59 results about "Human video" patented technology

Disclosed in the present invention is a dynamic human modeling method based on three-dimensional Gaussians. In the present method, a multi-view human video is given firstly, and in order to correct and transform the dynamic human appearance, a latent code and a set of blend weights are added to each Gaussian; and the latent code and a target pose are inputted to an MLP to correct the Gaussian in a canonical space, so as to capture an appearance change under the target pose. The corrected Gaussian is transformed into the target pose by means of the blend weights thereof in a linear blend skinning (LBS) manner. Finally, a realistic human image at a new viewing angle or a new pose can be rendered in real time by means of Gaussian splatting. Compared with an existing method based on an implicit neural radiance field (NeRF), the three-dimensional Gaussian representation of the present invention can better capture high-frequency details while achieving optimal rendering performance.

The invention relates to a multi-modal virtual human real-time generation method based on an expression control embedding space, and belongs to the field of artificial intelligence. According to the method, an expression control embedding space is constructed and used for fusing voice semantics, a rhythm structure and multi-dimensional emotion information, and continuous and controllable multi-modal driving vectors are generated. The whole system has an end-to-end linkage mechanism from audio input to expression and action output. Semantic features, rhythm structures and emotional states jointly act on generation paths of lip and upper body postures and expression modalities, and all modal features are fused and expressed in a unified control space through a collaborative coding and time sequence alignment mechanism. And finally, a high-consistency and high-fidelity virtual human video is generated in real time through an output scheduling mechanism. The method has remarkable advantages in the aspects of modal fusion consistency, generation expression naturalness and emotion control flexibility, and can be widely applied to key scenes such as virtual human broadcasting, voice interaction agency and meta-universe digital identity construction.

The embodiment of the invention discloses a virtual human video generation method and device, equipment and a storage medium. The method includes: acquiring input content; wherein the input content comprises a line text and a video script and is used for generating dubbing and pictures of a video; determining at least one moving time point based on the line text and the video script; at least one lower body moving animation is added to each moving time point, and an animation set corresponding to each moving time point is obtained; wherein the moving time point is a starting time point for playing the lower body moving animation; determining a target lower body moving animation sequence based on the animation set corresponding to each moving time point; and generating a virtual human whole body movement video according to the target lower body movement animation sequence and the input content. According to the generation method of the virtual human video provided by the embodiment of the invention, the problems of uncontrollable walking, sliding and the like of the virtual human in the generated video can be prevented, so that the generation quality and the display effect of the virtual human video are improved.

The invention relates to the field of artificial intelligence, and discloses a method and device for generating a video based on a copywriting, equipment and a medium, and the method comprises the steps: obtaining a to-be-generated video copywriting, extracting a corresponding key semantic feature through a preset semantic understanding model, converting the to-be-generated video copywriting into corresponding audio data through a preset voice conversion model, and storing the audio data in a database; the method comprises the following steps: acquiring key semantic features of audio data, marking timestamps on the audio data by adopting a text-to-speech algorithm, matching the key semantic features with a preset material library to obtain picture materials, generating an image-text video with the timestamps, performing preset mouth shape synchronous rendering on the audio data by adopting a preset dynamic conversation face video model, and generating a virtual human video with the timestamps. And according to the timestamp of the image-text video and the timestamp of the virtual human video, the image-text video and the virtual human video are merged, and a final video is generated and uploaded, so that the technical problem that the customer experience is affected due to the fact that too templated video generation cannot meet the requirements of different customers at present is solved.

The invention discloses a virtual human video generation method and device, electronic equipment and a medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: determining broadcast audio data and broadcast emotion data of a virtual human corresponding to a target user based on a to-be-broadcast text and an original video of the target user, the original video is acquired in advance according to a preset time period; based on a stable video clip in the original video, broadcast face data of the virtual human is determined, and the video duration of the stable video clip is greater than or equal to a preset effective duration; and based on the broadcast audio data, the broadcast emotion data and the broadcast face data, generating a face video of the virtual human, so as to generate a virtual human video corresponding to the target user according to the face video of the virtual human. Through collaborative fusion of broadcast audio, emotion and face data, precise matching between mouth shapes and audio and between expressions and emotions is realized, personalized generation can be completed only through a relatively short original video, and the recording cost and the technical threshold are reduced.

The invention provides a 3D human body posture estimation method and system based on dual-channel fusion driving in the technical field of computer vision. The method comprises the following steps: S1, constructing a 3D human body posture estimation model; s2, a large number of historical human body videos are acquired to construct a data set; s3, dividing the data set into a training set, a verification set and a test set, training the 3D human body posture estimation model through the training set, verifying the trained 3D human body posture estimation model through the verification set, and testing the verified 3D human body posture estimation model through the test set; s4, deploying the 3D human body posture estimation model passing the test; and S5, acquiring a real-time human body video, preprocessing the real-time human body video, and inputting the preprocessed real-time human body video into the deployed 3D human body posture estimation model to obtain a human body posture estimation result. The method has the advantages that the precision and robustness of 3D human body posture estimation are greatly improved.

The embodiment of the invention provides a virtual human video generation method and device, a computer program product and electronic equipment. The method comprises the following steps: identifying corresponding text content according to original audio, and determining skeleton posture graphs of multiple groups of human body actions; processing the text content and the skeleton posture graph through a large language model, and outputting split mirror planning information corresponding to the text content; the sub-mirror planning information comprises an overall style, scene description, figure description and sub-mirror attribute information of each sub-mirror; generating a scene dynamic graph based on the overall style and the scene description, and generating a character static graph based on the overall style and the character description; and according to the character static graph bound to each sub-mirror and the sub-mirror attribute information, generating a character motion graph of each sub-mirror, and combining the scene motion graph and the character motion graphs of the plurality of sub-mirrors to generate a virtual human video in the same scene. According to the technical scheme of the embodiment of the invention, the efficiency of generating the virtual human video can be improved.

The embodiment of the invention provides a three-dimensional virtual human video synthesis method, system and device and a storage medium, and the method comprises the steps: employing a unified camera position to separately shoot a site video and a field video, carrying out the detection of a person in each frame of image of the field video, and generating a pixel-level mask of each human body; on the basis of a parameterized SMPL human body model, registering human body contours formed by masks in each frame of mask grey map to obtain SMPL parameters of each registered human body; and taking the registered SMPL parameters of each human body as a new view angle of the implicit neural radiation field, synthesizing a video picture under the new view angle under volume rendering, and fusing the video picture with a three-dimensional virtual human to generate a video. The problem that in an existing method, registration optimization of a parameterized three-dimensional human body and the contour of human body segmentation is difficult to achieve is avoided, and the visual defect that in an existing image synthesis method, the posture of the virtual human in a video picture is adjusted only based on the human body posture, and consequently the virtual human foot suspension or foot sliding animation is likely to be generated can be avoided.

The invention discloses a digital human voice lip synchronization method based on an adversarial texture enhancement network. The method comprises the following steps: acquiring input audio and video data, extracting audio features by adopting a Wav2Lip model, and realizing preliminary audio and video alignment; performing dynamic modeling on the extracted lip key point sequence by adopting a graph neural network (GCN) so as to capture the time consistency of lip movement; performing texture detail enhancement on the lip region image by using a generative adversarial network (GAN), and introducing an edge perception loss function to optimize the contour definition of the lip region; through a multi-modal fusion strategy, audio and visual features are subjected to joint alignment, and the accuracy of sound and lip synchronization is enhanced; and constructing an audio and video feature consistency evaluation module, and judging and optimizing the generated lip image region. According to the method, the authenticity of the lip movement of the digital human is improved, so that the lip movement better conforms to a real voice expression mode, and the method can be widely applied to the fields of virtual human video generation, film and television post-processing, game role animation, intelligent customer service and the like.

The invention relates to the technical field of intelligent equipment. A trajectory diffusion imitation learning method based on single human video demonstration comprises the steps that a demonstration video is analyzed to obtain an operation body posture motion trajectory and an object pose motion trajectory, and the demonstration video is a video of a human or machine operation object; respectively carrying out interpolation calculation on the positions and postures of the operation body posture motion trail and the object pose motion trail to obtain a training data set; using the training data set to train a diffusion strategy model to obtain a target diffusion strategy model; and during deployment, the real-time pose of an end effector of the robot serves as observation input of the target diffusion type strategy model, an operation body pose movement track and an object pose movement track in the deployment stage are formed, and a joint control instruction of a target operation body is obtained. And the task success rate is kept, and the practicability and the expandability of the imitation learning method are improved at the same time.

The invention discloses a low-cost robot imitation learning method and system based on a human video. The method comprises the following steps: S1, data acquisition; s2, data extraction and physical alignment are carried out to eliminate man-machine physical differences; the step is divided into two parallel processing modules of action space alignment and visual space alignment; s3, data set construction: mixing the aligned human data with real robot teleoperation data, carrying out balanced sampling, and constructing a mixed data set Dmix; and S4, cooperative training: constructing a strategy network based on diffusion Transform for training. According to the method, data can be acquired only through the monocular RGB camera, expensive robot teleoperation data are replaced with cheap and easily available human videos, and the data acquisition threshold is greatly reduced. Through a visual alignment strategy of random color grid rendering, a network can learn neglect skin color textures and pay attention to geometric structures without a complex generative model, so that the robot can be seamlessly migrated to robots in different forms.

The invention relates to the field of artificial intelligence, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a video generation method and device based on audio driving, equipment and a medium. Performing voice recognition and voice feature extraction on the driving audio, and generating a composite feature in an audio mode according to the decoupled semantic feature and the speaker embedded feature; performing multi-layer coding processing on the reference video to obtain multi-layer face image features; performing style feature extraction according to the multi-layer face image features to obtain style features; and inputting the composite features, the multi-layer face image features and the style features in the audio mode into a video generator which completes generative adversarial training in advance, performing multi-mode feature fusion on the input features, and fusing the fused features into spatial semantic information for decoding to generate a virtual human video corresponding to the driving audio. Through audio decoupling and multi-modal feature fusion, the naturalness and authenticity of virtual human video generation are improved.

The invention discloses a data processing method and system for digital human model training, and the method comprises the steps: screening a broadcast type high-definition real human video in a digital human material library, and storing the video as a video file; recording the starting time and the ending time of the segment where only one face is located in the video file; independently extracting audio content in the video file; recording the starting time and the ending time of the segment where the voice is located in the audio file; and based on the starting time and the ending time of the segment where the human face is located in the video file and the starting time and the ending time of an intersection of the starting time and the ending time of the segment where the human voice is located in the audio file, cutting the video file to obtain an effective video segment. The method has the advantages that high-quality data is provided for model training, synchronization and matching of human faces and human voices in the data are ensured, simulation of the digital human in the aspects of facial expressions, language expressions and the like is more accurate and harmonious, and the continuity of overall expression is improved.

The invention discloses a real-time human video extinction model training method, and the method comprises the steps: collecting application data, and generating a training data set based on the application data; inputting the training data set into a preset initial model to obtain an output set corresponding to the training data set under the current model parameters; a semantic and detail prediction high-low resolution reversible branch is set in the preset initial model, the semantic and detail prediction high-low resolution reversible branch is connected with an encoder, and high-resolution detail prediction output and low-resolution semantic estimation output are output; based on the output set, determining a corresponding loss function value under the current model parameter; and adjusting model parameters in the preset initial model until the loss function value is lower than a preset threshold value. Reversible branches only participating in the training process of the model are introduced in the RVM coding stage, and under the condition that the model reasoning scale is not increased, full extraction of shallow detail information and depth semantic information in a video airspace by an encoder is achieved.

The invention discloses an artificial intelligence robot system based on human video learning, which comprises a human operation video data set and a small-scale robot teaching data set, and is characterized in that human operation videos of various daily tasks are stored in the human operation video data set; a small amount of robot teaching data is stored in the small-scale robot teaching data set, the output end of the human operation video data set is in signal connection with a general operation prediction model, and the general operation prediction model is used for learning physical interaction rules according to human operation videos in the human operation video data set. And the output end of the general operation prediction model is in signal connection with a physical interaction rule generation module. According to the method, the error-prone steps of attitude estimation, redirection and image restoration are completely abandoned, knowledge migration is directly carried out on advanced semantics and representation levels, the learning precision is ensured, and the conversion link of error accumulation is avoided.

This application discloses a method, device, and storage medium for generating multi-action human body videos, comprising: inputting reference images and multi-action prompt text into a multimodal large language model; performing inference on the multi-action prompt text through the multimodal large language model to output a fine-grained action description sequence; generating intermediate image frames based on the reference images and the fine-grained action description sequence; extracting human skeleton key point data from the intermediate image frames to obtain a visual skeleton sequence; inputting the visual skeleton sequence, multi-action text prompts, and reference images into a video diffusion model; and integrating the visual skeleton sequence into the denoising generation process of the video diffusion model through a pose guidance adapter to output a coherent human body video matching the multi-action text prompts. This application achieves accurate, coherent, and efficient generation of complex multi-action videos by decoupling and fusing the chained inference of the multimodal language model, the intermediate representation of the skeleton sequence, and the pose guidance of the video diffusion model.

The embodiment of the invention provides an information display control method, a computer storage medium and electronic equipment, and relates to the technical field of computers. The virtual human video generation method comprises the following steps: displaying a first game interface; in response to an acquisition operation for the target resource in the first game interface, recording demand information of the target resource, and switching to a second game interface corresponding to the target resource; and providing a suspension control in the second game interface, wherein the suspension control displays the demand information of the target resource. According to the technical scheme, the interface development and maintenance cost can be reduced, and the user operation process is simplified.

The application discloses a virtual human video generation method and apparatus. The method includes: obtaining a driving text; obtaining, based on the driving text and an action annotation of a first video, an action type corresponding to the driving text, where the action annotation includes a plurality of action types of a character in the first video; extracting, from the first video based on the action type, an action representation corresponding to the driving text; and generating a virtual human video based on the action representation. According to this application, the virtual human video in which the action of the character is accurate, controllable, and compliant with a preset action specification can be automatically generated, and personalized customization of an action of a virtual human can be implemented by adjusting the action specification.

The invention provides a man-vehicle interaction method and a man-vehicle interaction system, and belongs to the field of vehicle man-machine interaction technology and artificial intelligence technology.The man-vehicle interaction method comprises the steps that in response to a man-scene group photo instruction, a landscape video shot by a camera outside a vehicle and a figure video shot by a camera inside the vehicle are obtained; extracting a scenery picture from the scenery video; extracting a figure image from the figure video; fusing the scenery image and the figure image to obtain a figure and scenery group photo image; and displaying the people-scene group photo on a display module of the vehicle. According to the embodiment of the invention, when encountering beautiful scenery and being inconvenient to park and get off to take a group photo, people and scenery group photo can be carried out in scenery inside and outside the vehicle.

The invention discloses an enhanced human video extinction model design method, equipment and a medium, belongs to the technical field of artificial intelligence, and is used for solving the problems that the existing human video extinction technology has limitation when processing a dynamic background or camera visual angle change, needs certain artificial participation cost, and is inconvenient to use. And the time domain information cannot be fully utilized. The method comprises the following steps: carrying out data acquisition processing on multi-category video data containing character features to obtain a multi-dimensional data set; performing feedforward network structure coding calculation processing based on a feature coding module on an input image in the model training data set to obtain a convolutional deep feature map; carrying out a collocation operation on the convolutional deep feature maps, carrying out input and output control on a feature decoding module, and carrying out a channel splitting and collocation operation on other deep feature maps to obtain a final deep feature map; and completing content information extraction of the to-be-processed character feature video data through the human video extinction model.

The application discloses a virtual human video generation method and apparatus. The method includes: obtaining a driving text; obtaining, based on the driving text and an action annotation of a first video, an action type corresponding to the driving text, where the action annotation includes a plurality of action types of a character in the first video; extracting, from the first video based on the action type, an action representation corresponding to the driving text; and generating a virtual human video based on the action representation. According to this application, the virtual human video in which the action of the character is accurate, controllable, and compliant with a preset action specification can be automatically generated, and personalized customization of an action of a virtual human can be implemented by adjusting the action specification.

The application discloses a method and device for virtual human video fusion, which comprises the following steps: collecting a scene video based on a binocular camera, processing the scene video to obtain first depth information of at least one object in the scene, importing multiple images of virtual humans into the scene video, associating each image with second depth information of the virtual human, fusing the images of the virtual human and the scene video based on the first depth information and the second depth information to obtain a virtual human fusion video and transmitting the virtual human fusion video to a display device for displaying the virtual human fusion video on an interactive interface. The application improves the accuracy of the position display of the virtual human in the video scene.

The application belongs to the technical field of embodied intelligence technology and robot technology, and particularly relates to a method and device for learning robot operation strategy from human video. The method comprises the following steps: acquiring an RGB-D video of a human operation from an egocentric perspective, and converting the RGB-D video into a three-dimensional scene point cloud sequence with time sequence; segmenting a hand point cloud and an object point cloud from the three-dimensional scene point cloud sequence, estimating a hand pose based on the hand point cloud to form a motion trajectory, and after simplification by a key frame, mapping the hand trajectory to a seed action trajectory of a robot end effector through a coordinate system; performing spatial transformation augmentation on the seed action trajectory to generate a plurality of enhanced action trajectories, combining the object point cloud to synthesize a synchronous observation point cloud sequence; and constructing a training data set by using the enhanced action trajectory and the corresponding synthesized observation point cloud sequence, training a 3D diffusion strategy model based on the data set, and obtaining the robot operation strategy.

The method, device, storage medium, and computer equipment for driving human expressions provided by this application, when performing expression driving, first determine consecutive video frame images and an expression driving target in the human video to achieve expression driving on the video frame images and ensure the naturalness of the expression driving effect; then perform face alignment based on the facial key points in the video frame images to obtain a standard face image, and use the target expression driving model to perform expression driving on the standard face image based on the expression driving target to generate an expression-driven video, thereby improving the quality of the expression driving effect. Since the target expression driving model is composed of a three-dimensional convolutional neural network and is used to map two-dimensional images into a three-dimensional latent space for feature encoding and expression driving, this model can accurately track faces at various angles in the video frame images and perform expression editing and driving, achieving natural expression connection between frames of the video frame images without affecting the quality of the human video.

The invention provides a virtual human video generation method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. In the method, a face image, a covering image and a voice feature in each video frame are input into a target image generation model together for processing; compared with the video to be processed, only the mouth shape of the virtual human changes and is consistent with the driving voice, and the posture is still kept consistent, so that the virtual human image is more vivid, and the watching experience of the user is improved. Moreover, according to the method, the face image and the covering image are respectively coded, so that the information of the face image can be fully applied, and image features and voice features are fused by combining a self-attention mechanism and a cross attention mechanism, so that the detail retention degree of the face image is higher, the difference between a virtual person and a real person is reduced, and the user experience is improved. And the vivid effect of the virtual human video is improved.

The present invention discloses a method, apparatus, computer device and storage medium for generating a human video, including: decoding an original human video to obtain a plurality of image frames, and mapping each image frame onto a two-dimensional image to obtain an initial human pose sequence map; obtaining time position information determined for each image frame in the initial human pose sequence map to obtain a time position encoding vector, and splicing and fusing the initial human pose sequence map and the time position encoding vector to obtain a target human pose sequence map; determining noise features based on each image frame in the target human pose sequence map; inputting the noise features into a fast denoising diffusion model to generate new human image features corresponding to the image frames as target image features; and decoding each obtained target image feature to obtain a target human video. The use of the present invention improves the clarity and smoothness of the generated human video.

The application discloses a world action model enhancement method and device fusing structured prior and continuous dynamics, wherein the method takes human videos in a first view and a third view as input, constructs scene-event graphs and structured priors such as causality, counterfactual constraints and execution constraints; under weak labeling conditions, performs spectrum analysis on local time sequence features of actions, and jointly encodes the time sequence features and space-time features into latent actions; further, continuous time latent dynamics modeling is introduced into a unified world action model through a neural control differential equation, and joint reasoning is performed on video state changes and latent action recovery in combination with task semantics, structured priors and counterfactual conditions. Finally, long-time rolling error accumulation is inhibited through real observation rewriting, action block generation and asynchronous closed-loop execution, so that the feasibility of weakly labeled video action learning, the rationality of action generation and the stability of closed-loop control are improved.

The application discloses a kind of explicit denoising driven spatio-temporal fusion 3D human posture estimation method and medium, the method first constructs the network model containing explicit denoising attention module, spatio-temporal fusion module and regression layer;Explicit denoising attention module is modeled in parallel by signal, noise double branch, realizes feature denoising by feature alignment, dynamic gating and explicit difference, spatio-temporal fusion module adopts "space-time" and "time-space" double path decoupling modeling and adaptive fusion feature.When training, human posture dataset containing different characters, posture and random occlusion is used to train model end to end;When testing, the human video to be estimated is input into the trained model after preprocessing, and the 3D human posture result is output by regression layer after explicit denoising, spatio-temporal fusion processing.The application effectively suppresses long sequence noise and occlusion interference, relieves single path modeling error accumulation, and improves the accuracy and robustness of posture estimation in complex scenes.

A video call control method, a communication device and a storage medium are disclosed in the present application. The method may include: acquiring a video call connection request of a first terminal device sent by the call session control function unit; sending the video call connection request to the virtual human service unit, and acquiring a virtual human image corresponding to the first terminal device sent by the virtual human service unit; performing a replacement processing on a target object in a video based on the virtual human image to obtain virtual human video data; and sending the virtual human video data to the session access unit, so as to send the virtual human video data to a second terminal device via the session access unit.

The invention discloses an explicit denoising-driven space-time fusion 3D human body posture estimation method and a medium. The method comprises the following steps: firstly, constructing a network model containing an explicit denoising attention module, a space-time fusion module and a regression layer; the explicit denoising attention module performs parallel modeling through signal and noise double branches, feature denoising is realized through feature alignment, dynamic gating and explicit difference, and the space-time fusion module performs decoupling modeling through'space-time 'and'time-space' double paths and performs adaptive feature fusion. During training, performing end-to-end training on the model by using a human body posture data set containing different figures, postures and random shielding; in the testing process, a to-be-estimated human body video is preprocessed and then input into the trained model, explicit denoising and space-time fusion processing are conducted, and a 3D human body posture result is output through the regression layer. According to the method, long-sequence noise and shielding interference are effectively suppressed, single-path modeling error accumulation is relieved, and the precision and robustness of attitude estimation in a complex scene are improved.