Animation generation method and apparatus, and device, medium and program product

Abstract

Provided in the embodiments of the present application are an animation generation method and apparatus, and a device, a medium and a program product, which are used for improving the capture precision of facial dynamic details, thereby improving the facial control precision of a figure in an animation, and can be applied to the fields of computers, image processing, artificial intelligence, etc. The method comprises: acquiring a reference video and a reference image; performing feature extraction on pixel points in the reference video and the reference image, so as to obtain first feature representations; performing action feature extraction on the reference video, so as to obtain a first action feature representation and a second action feature representation of the reference video, wherein the first action feature representation is used for indicating a global action feature corresponding to the reference video, and the second action feature representation is used for indicating a facial action feature corresponding to the reference video; and on the basis of the first feature representations, the first action feature representation and the second action feature representation, performing decoding processing to obtain a simulated animation.

Description

An animation generation method, apparatus, device, medium, and program product

[0001] This application claims priority to Chinese Patent Application No. 202510107807.3, filed on January 22, 2025, entitled "An Animation Generation Method, Apparatus, Device, Medium and Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of computer science, and more specifically to animation generation. Background Technology

[0003] With the popularization of short videos and the rapid development of the animation industry, systems for intelligently generating animations based on videos and images have been widely used. In this process, users acquire images or videos containing human figures through a camera or other image capture tools, and then select an animation template (i.e., an animated video) to synthesize the human figures from the images or videos into virtual animations. For example, in one scenario, when a user is chatting with a friend through an instant messaging application, they can send emoticons to express their meaning; these emoticons can be generated based on the above method.

[0004] Current solutions typically control facial expressions by using explicit facial landmarks, but this not only leads to a loss of detail but also heavily relies on the accuracy of landmark extraction technology.

[0005] Therefore, how to improve the facial dynamic details and control precision of human figures in animation is an urgent problem that needs to be solved. Summary of the Invention

[0006] This application provides an animation generation method, apparatus, device, medium, and program product to improve the precision of capturing facial dynamic details, thereby improving the facial control accuracy of human figures in animation.

[0007] In view of this, this application provides an animation generation method, including:

[0008] Acquire reference videos and reference images, wherein the reference videos are used to indicate the behavioral actions corresponding to the target object in the simulated animation, and the reference images are used to indicate the portrait information corresponding to the target object in the simulated animation;

[0009] Feature extraction is performed on the reference video and the reference image to obtain a first feature representation, which is a latent feature of the reference video and the reference image;

[0010] Feature extraction is performed on the reference video to obtain a first action feature representation and a second action feature representation of the reference video. The first action feature representation is used to indicate the global action features corresponding to the reference video, and the second action feature representation is used to indicate the facial action features corresponding to the reference video.

[0011] The simulation animation is obtained by decoding based on the first feature representation, the first action feature representation, and the second action feature representation.

[0012] This application also provides an animation generation apparatus, comprising:

[0013] The acquisition module is used to acquire reference videos and reference images. The reference videos are used to indicate the behavioral actions corresponding to the target object in the simulated animation, and the reference images are used to indicate the portrait information corresponding to the target object in the simulated animation.

[0014] The processing module is configured to perform feature extraction on the reference video and the reference image to obtain a first feature representation, wherein the first feature representation is a latent feature of the reference video and the reference image; perform feature extraction on the reference video to obtain a first motion feature representation and a second motion feature representation, wherein the first motion feature representation is used to indicate the global motion features corresponding to the reference video, and the second motion feature representation is used to indicate the facial motion features corresponding to the reference video; and perform decoding processing based on the first feature representation, the first motion feature representation, and the second motion feature representation to obtain the simulated animation.

[0015] This application also provides a computer device, including: a memory, a processor, and a bus system;

[0016] The memory is used to store programs;

[0017] The processor is used to execute programs in memory, and the processor is used to execute the methods mentioned above according to the instructions in the program code;

[0018] Bus systems are used to connect memory and processor to enable communication between them.

[0019] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0020] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0021] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: Implicit feature vectors of images are extracted from reference images and reference videos. These feature vectors include high-dimensional information such as facial expressions and poses. Furthermore, these feature vectors are continuous value vectors, which can represent richer facial information, thereby improving the precision of capturing facial dynamic details. On the other hand, learning facial features and action features separately enables good decoupling of facial features and action features under different image styles, improving the generalization ability of the solution. Attached Figure Description

[0022] Figure 1 is a schematic diagram of an application scenario of animation generation in an embodiment of this application;

[0023] Figure 2 is a schematic diagram of an interface for generating emoticons in an instant messaging software according to an embodiment of this application;

[0024] Figure 3 is a schematic diagram of an expression template in an embodiment of this application;

[0025] Figure 4 is a schematic diagram of an embodiment of animation generation in this application;

[0026] Figure 5 is a schematic diagram of a network structure of the intensity-sensing action extraction network in an embodiment of this application;

[0027] Figure 6 is a schematic diagram of a network structure of a fine-grained appearance extraction network in an embodiment of this application;

[0028] Figure 7 is a schematic diagram of a network structure of the identity-aware adaptation network in an embodiment of this application;

[0029] Figure 8 is a schematic diagram of the processing flow of the animation generation model in the embodiments of this application;

[0030] Figure 9 is a schematic diagram of an embodiment of the animation generation device in this application;

[0031] Figure 10 is a schematic diagram of one embodiment of the server in this application;

[0032] Figure 11 is a schematic diagram of one embodiment of the terminal in this application. Detailed Implementation

[0033] This application provides an animation generation method, apparatus, device, medium, and program product to improve the precision of capturing facial dynamic details, thereby improving the facial control accuracy of human figures in animation.

[0034] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functions of that module or unit.

[0036] With the popularization of short videos and the rapid development of the animation industry, systems for intelligently generating animations based on videos and images have been widely used. In this process, users acquire images or videos containing human figures through cameras or other image acquisition tools, and then select an animation template (i.e., animated video) to synthesize virtual animations from the images or videos. For example, in one scenario, when a user is chatting with a friend through an instant messaging application, they can send emojis to express their meaning; these emojis can be generated based on the above method. Current solutions typically control facial expressions through explicit facial key points, but this not only leads to a loss of detail but also heavily relies on the accuracy of landmark extraction technology. Therefore, how to improve the facial dynamic details and control precision of human figures in animations is a problem that urgently needs to be solved.

[0037] To address this technical problem, this application provides the following technical solution: Acquiring a reference video and a reference image, wherein the reference video indicates the behavioral actions of a target object in a simulated animation, and the reference image indicates the facial information of the target object in the simulated animation; extracting features from pixels in the reference video and the reference image to obtain a first feature representation, which is a latent feature of the reference video and the reference image; adding random noise feature representation to the first feature representation to obtain another first feature representation; extracting motion features from the reference video to obtain a first motion feature representation and a second motion feature representation, wherein the first motion feature representation indicates the global motion features corresponding to the reference video, and the second motion feature representation indicates the facial motion features corresponding to the reference video; and performing decoding processing based on the first feature representation, the first motion feature representation, and the second motion feature representation to obtain the simulated animation. This extracts latent feature vectors from the reference image and reference video. These feature vectors include high-dimensional information such as facial expressions and postures. Furthermore, these feature vectors are continuous value vectors, which can represent richer facial information, thereby improving the precision of capturing facial dynamic details. On the other hand, learning facial features and motion features separately allows for good decoupling of facial features and motion features under different image styles, thereby improving the generalization ability of the solution.

[0038] Given that this application involves some technical terms, these terms will be introduced below.

[0039] Stable video diffusion model: This refers to a deep learning-based video generation model that achieves efficient and stable video generation through diffusion and denoising processes in the latent space. It can be understood as the video constructed using the variational network and denoising network in this application.

[0040] Implicit video-driven animation refers to the process of using implicitly represented motion information as control signals to drive the generation of human portrait animation, without directly using explicit facial key points or facial markers.

[0041] Human portrait animation generation: refers to the technology of converting static single-person images into video content with dynamic expressions and head postures through algorithms.

[0042] Identity information decoupling: The process of separating the identity information of a person from their motion information in a video, so as to allow for independent control and adjustment during animation generation.

[0043] Implicit representation: refers to a method that does not directly describe data or features, but indirectly defines data or features through a series of deep learning network operators. In this embodiment, it is used to encode human motion information and serve as a control signal.

[0044] Control signal injection: refers to integrating implicitly represented motion information into a denoising network through an attention mechanism to achieve precise control over the animation generation process.

[0045] Denoising network: A type of deep learning network used to learn the inverse denoising process in a latent space, including self-attention and cross-attention mechanisms.

[0046] Spatiotemporal consistency refers to the preservation of spatial details and temporal continuity in the generated video, ensuring that the movements and facial expressions of the characters in the video are natural and smooth.

[0047] Identity consistency: In the process of generating portrait animation, maintain the consistency of the identity features (such as facial features) of the person in the source image.

[0048] Action memory bank: a technique for enhancing implicit motion feature representation by storing and retrieving contextual information between different frames to improve the continuity and context-awareness of motion features.

[0049] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0050] The animation generation method provided in this application can be applied to the system shown in Figure 1, which includes a terminal 100, a server 200, a database 300, and a network 400. The terminal 100, server 200, and database 300 are connected via the network 400. The database 300 is used to provide the system with reference videos, reference images, and generation models, among other data. The number of terminals, servers, and databases in the system shown in Figure 1 is merely an example; for instance, there can be multiple terminals, servers, and databases. This application does not limit the number of terminals, servers, and databases.

[0051] The aforementioned system is used in the fields of image processing, animation simulation, emoji generation, and computer technology, and this application embodiment does not limit these applications. In this solution, server 200 can receive reference video and reference image sent by terminal 100, and then perform feature extraction on the pixels in the reference video and reference image to obtain a first feature representation, which is a latent feature of the reference video and reference image; add random noise feature representation to the first feature representation to obtain a second feature representation; extract motion features from the reference video to obtain a first motion feature representation and a second motion feature representation, where the first motion feature representation indicates the global motion features corresponding to the reference video, and the second motion feature representation indicates the facial motion features corresponding to the reference video; perform decoding processing based on the first feature representation, the first motion feature representation, and the second motion feature representation to obtain the simulated animation; and finally send the simulated animation to terminal 100. It should be noted that the reference video, reference image, and generation model are authorized by the user or fully authorized by all parties.

[0052] In this process, terminal 100 communicates with server 200 via a network. Database 300 can be integrated onto server 200, or it can be located in the cloud or on another server. During the animation generation process, interaction can occur between terminal 100 and server 200. For example, a user generates an animation generation request through terminal 100, including a reference video and reference image. Terminal 100 then sends the animation generation request, along with the reference video and reference image, to server 200. Server 200 generates a corresponding simulated animation based on the reference video and reference image and sends it back to terminal 100.

[0053] Terminal 100 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, or in-vehicle terminal, but is not limited to these. Server 200 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), big data, and artificial intelligence platforms.

[0054] In short, a database can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, shared by multiple users, with minimal redundancy, and independent of application programs. A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or Extensible Markup Language (XML); or according to the type of computer they support, such as server clusters or mobile phones; or according to the query language used, such as Structured Query Language (SQL) or XQuery (a free-form language); or according to performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, simultaneously supporting multiple query languages.

[0055] It is understood that in the specific embodiments of this application, data related to the reference video, reference image, generation model and motion memory library are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0056] Based on the above description, the animation application scheme of this application will be explained below using a specific application scenario. For example, when a user chats with a friend through an instant messaging application, they can send emoticons to express their meaning, and the emoticons can be generated based on the above scheme. As shown in Figure 2, in the chat interface of an instant messaging application, after the user selects the emoticon option, the AI ​​emoticon creation interface shown in Figure 2 will appear. After the user clicks to upload a photo, they can be redirected to the photo upload interface, where they can upload a reference image by selecting the corresponding upload control. For example, as shown in Figure 2, when "upload photo" is selected, the user's portrait image can be directly captured through the camera. When "upload from album" is selected, the album application can be directly invoked, and a portrait image can be selected for upload. After uploading the portrait image, a corresponding reference video can be selected (i.e., the emoticon template shown in Figure 3). As shown in Figure 3, emoticon template 1 is an animated video of "saluting"; emoticon template 2 is an animated video of "happy"; emoticon template 3 is an animated video of "sending a heart"; and emoticon template 4 is an animated video of "rejection". Then, after the user selects the corresponding emoji template, the emoji template is used as a reference video, and the portrait image is used as a reference image to generate a simulated animation. This will generate a representation of the user's portrait information, but the actions will be those indicated in the emoji template.

[0057] Based on the above description, taking the server as the execution entity, the animation generation method in this application will be described. Please refer to Figure 4. One embodiment of the animation generation method in this application includes:

[0058] 401. Obtain a reference video and a reference image. The reference video is used to indicate the behavior and actions of the target object in the simulated animation, and the reference image is used to indicate the human image information of the target object in the simulated animation.

[0059] As one possible implementation, when applied to simulated animation generation scenarios, the server can receive simulated animation generation requests from the terminal and obtain corresponding reference videos and reference images based on these requests.

[0060] The simulated animation is generated based on reference videos and reference images. The target objects in the simulated animation are also generated based on reference videos and reference images. Specifically, the target objects are generated based on human image information in the reference images and behavioral actions in the reference videos. The reference videos and reference images are explained below.

[0061] Reference videos are used to indicate the behavior of a target object. They can take the form of dynamic video sequences that include behavioral actions, providing motion trajectories and trend information for the target object in the simulated animation. In other words, the main function of reference videos is to provide behavioral patterns of the target object in the simulated animation, including key features such as limb movements, posture changes, speed, and rhythm. These key features are directly used to simulate the behavior of the target object in the animation.

[0062] Reference videos are a key input for generating dynamic simulation animations, and their sources typically include: 1. Actual video footage: Capturing the actual movements of the first reference object using camera equipment (such as mobile phones, camcorders, etc.) to capture its natural behavior. 2. Third-party video resources: Selecting a corresponding reference video from a stored video library. This reference video contains the movements of the first reference object and serves as a behavioral template for generating the simulation animation. 3. Motion capture systems: Generating motion data of the first reference object using motion capture technology (such as optical capture, inertial capture, etc.). This motion data can usually be converted into a reference video format. 4. Virtually generated video: Virtual videos generated or fitted by a computer, such as synthetic videos generated using existing motion data.

[0063] Reference images are static images used to indicate the appearance characteristics (human image information) of a target object. They primarily provide visual attribute information about the target object in the simulated animation, such as facial features, expressions, and appearance style. The purpose of reference images is to provide clear guidance for the appearance information of the target object, including hairstyle, clothing, facial features, and expressions. These features will be used to construct the image of the target object in the simulated animation, giving it unique appearance characteristics.

[0064] The sources of reference images can include the following: 1. Actual photographs of the second reference object: Still photographs of the second reference object acquired through a camera or other capture device, which can realistically reflect its appearance features. The first reference object is a reference object providing behavioral actions in the reference video (if any), and the second reference object is a reference object providing facial information in the reference image (if any). The second reference object and the first reference object can be the same reference object or different reference objects; this application does not specifically limit this. 2. Still video frame extraction: Extracting a frame from other videos as a reference image, for example, selecting the keyframe with the most obvious features of the second reference object from a video including the second reference object. 3. Design drawings or illustrations: Sketches or design drawings of the second reference object drawn by an artist, used to provide visual reference for animation. 4. Virtually generated images: Target human figures generated through artificial intelligence, or virtual reference images selected from an external material library.

[0065] In summary, the reference video provides motion information in the temporal dimension, ensuring that the simulated behavior of the target object conforms to the real or expected motion trajectory. The reference image provides appearance features in the spatial dimension, ensuring that the image of the target object is accurately reproduced in the animation.

[0066] Depending on the application scenario, the reference video and reference image can have different sources. For example, in an application scenario based on generating emoticons from images, the reference video can be an emoticon template, and the reference image can be any portrait image. In an animation capture scenario, the reference video can be a video generated by capturing the actual actions of the reference object through a camera device (such as a mobile phone, camcorder, etc.), and the reference image can be a portrait image of another object. This application does not impose any limitations on this.

[0067] 402. Perform feature extraction on the reference video and the reference image to obtain a first feature representation, which is the latent feature of the reference video and the reference image.

[0068] This application does not specifically limit the method of feature extraction. For example, feature extraction can be performed on the reference image and the reference video as a whole. Alternatively, feature extraction can be performed on the reference image and the reference video separately. As a possible implementation, feature extraction can also be performed on individual pixels within the reference image, or even on individual pixels within video frames in image form within the reference video, to achieve feature extraction of the reference video.

[0069] As one possible implementation, after receiving the reference video and the reference image, the server can extract features from the pixels in the reference video and the reference image respectively to generate pixel signals for the reference video and the reference image, where the pixel signal is a signal form representing image information with pixels as the basic unit. Then, the server performs latent space modeling on the pixel signals of the reference video and the reference image respectively to obtain corresponding feature representations, namely, the feature representation corresponding to the reference video and the feature identifier corresponding to the reference image. Finally, the feature representations corresponding to the reference video and the reference image are concatenated to obtain the first feature representation. The first feature representation is a type of latent feature, obtained based on the feature representations of the reference video and the reference image.

[0070] As one possible implementation, when extracting features from the reference video and reference image, to ensure that the feature representations of the reference image and reference video can be efficiently learned and generated in a low-dimensional, continuous latent space, the server can invoke a variational network to extract features from the reference video and reference image. The process can be as follows: the server invokes a first variational network to extract features from the pixels of each video frame in the reference video to obtain a first pixel signal; it invokes a second variational network to extract features from the pixels in the reference image to obtain a second pixel signal; the first pixel signal and the second pixel signal are then used for latent space modeling and feature concatenation to obtain the first feature representation. It should be understood that the first pixel signal includes dynamic behavioral features in the reference video, such as motion trajectory, posture changes, and movement speed; the second pixel signal includes static appearance features of the human figure in the reference image, such as facial texture, expression information, and clothing features.

[0071] Variational networks, which combine deep learning with variational inference, are widely used to solve inverse problems, such as medical image reconstruction (MRI, CT), image denoising, and image deblurring. Their goal is to leverage the expressive power of neural networks to embed prior knowledge into the optimization problem, thereby improving the quality and efficiency of the solution. The first and second variational networks can have the same network structure or differ in design.

[0072] The extraction process of the first pixel signal by the first variational network is described below: Each video frame in the reference video is used as the input image of the first variational network; then, the convolutional neural network in the first variational network is called to extract the local features (such as edge information, texture, shape, etc.) of the pixels of each video frame; then, the extracted features are processed by a non-linear activation function (such as ReLU) and mapped to a higher-level feature space; finally, a latent space representation corresponding to each frame is generated, which is a low-dimensional, continuous latent feature.

[0073] Similarly, the second variational network uses the same process when processing the reference image, which will not be elaborated here.

[0074] Therefore, feature extraction is performed on the reference image and reference video based on pixels, and the latent space is modeled and concatenated to obtain the first feature identifier. This allows for the learning of effective feature representations of the reference image and reference video. These feature representations are continuous value vectors, which can represent richer facial information, thereby improving the precision of capturing facial dynamic details. Moreover, the use of variational networks enables the efficient learning and generation of image and video feature representations in a low-dimensional, continuous latent space.

[0075] 403. Perform feature extraction on the reference video to obtain a first action feature representation and a second action feature representation of the reference video. The first action feature representation is used to indicate the global action features corresponding to the reference video, and the second action feature representation is used to indicate the facial action features corresponding to the reference video.

[0076] It should be noted that feature extraction from the reference video refers to extracting features from the actions within the reference video, thereby obtaining a first action representation and a second action representation. The first action feature representation indicates the global action features corresponding to the reference video, while the second action feature representation indicates the facial action features corresponding to the reference video.

[0077] As one possible implementation, in order to capture more accurate global motion features and facial motion features, the server can use different feature extraction networks to extract features from the reference video.

[0078] In one exemplary scheme, the server invokes a pose extraction network to extract global action features from a reference video to obtain a first action feature representation; then it invokes an intensity-aware action extraction network to extract facial features from the reference video to obtain a second action feature representation.

[0079] Therefore, by using different feature extraction networks to extract features from the reference video, more accurate global motion features and facial motion features can be captured. This allows for subsequent learning of the global motion features and facial motion features, resulting in richer motion feature representations and thus improving the motion accuracy in the simulated animation.

[0080] In one exemplary scheme, the pose extraction network may include a spatial feature extraction module and a temporal series feature extraction module. The feature extraction process of the spatial feature extraction module can be as follows: each frame of the reference video is processed by a convolutional neural network in the spatial feature extraction module to extract spatial features of various body parts (such as the position and angle of the head, arms, and legs). The extraction process of the convolutional neural network includes multiple convolutional layers and pooling layers, progressively capturing low-level features (such as edges and contours) to high-level features (such as the overall shape of the limbs). The processing flow of the temporal series feature extraction module can be as follows: the spatial feature sequence extracted by the spatial feature extraction module is further processed by a temporal modeling network such as a Long Short-Term Memory (LSTM) network, a Gated Recurrent Unit (GRU), or a Transformer (a deep learning model based on self-attention) to capture the dynamic relationships between frames. In this case, the temporal series feature extraction module is responsible for modeling the movement patterns in the time dimension, such as the temporal relationships of the start, duration, and end of the movement. Finally, the frame-level features are fused into a high-dimensional global feature representation to obtain the first action feature representation. This first action feature representation reflects the global action information of the person in the reference video, such as walking trajectory, overall body movement direction, and posture change pattern.

[0081] In one exemplary scheme, the intensity-aware action extraction network can extract fine-grained features of the facial region of a person in a reference video, particularly action information related to facial expressions and muscle movements. The feature extraction process for facial action features can be illustrated in Figure 5: Facial images are cropped from each video frame in the reference video to obtain multiple facial images; a convolutional network (i.e., the action encoder shown in Figure 5) is invoked to extract features from these multiple facial images to obtain facial action feature representations; these facial action feature representations are added to intensity numbers to obtain a second input feature representation, where the intensity number indicates the intensity feature representation corresponding to a facial key point. This second input feature representation is the input to the intensity-aware action extraction network; the intensity-aware action extraction network is then invoked to extract features from the second input feature representation to obtain a second action feature representation.

[0082] The multiple facial images include faces appearing in various video frames of the reference video. Facial motion feature representations are used to describe the characteristics of facial motions in the facial images. Intensity numbers are used to indicate the intensity feature representations corresponding to facial key points. Facial key points are used to locate facial contours, expression changes, facial structures, etc., and can include eyebrows, eyes, nose, mouth, etc., which are not specifically limited in this application.

[0083] In the above scheme, when performing facial pre-cropping on each video frame of the reference video, the server can use a face detection model (such as a detector based on Haar features, Dlib, or deep learning) to locate the facial region of the person in the video; then, the facial region in each frame is cropped out to reduce background interference and focus on the feature extraction of facial movements.

[0084] Simultaneously, when the server adds the intensity number to the facial action feature representation, it also needs to calculate the intensity number. In this embodiment, the intensity number is an encoding mechanism that characterizes the intensity of facial action changes by calculating the positional offset of facial key points in consecutive video frames. Each intensity number can describe the amplitude of movement in a specific facial region (such as the tip of the nose, corner of the mouth, eyebrows, etc.). The process can be as follows: In the initial frame of the video, the initial position of the facial key point is recorded as a reference, and then the offset distance of the facial key point in subsequent frames is calculated; then, based on the offset distance of the facial key point, each frame is assigned an action intensity number. A larger offset distance corresponds to a higher action intensity, and vice versa.

[0085] In this scheme, the server can use the following common models for detection: 1. Dlib model, which is based on HOG features and regression trees and can detect 68 facial keypoints. 2. MediaPipe Face Mesh model, a deep learning-based facial keypoint detection model that can detect 468 high-precision keypoints. 3. OpenPose model, which supports multimodal keypoint detection for the human body and face. In an exemplary scheme, the server can calculate the Euclidean distance of each facial keypoint relative to its initial position on a two-dimensional plane as the offset distance. Then, the calculated offset distance is quantized to obtain the intensity number. For example, several intensity intervals can be defined, dividing the offset distance into multiple discrete intervals. For example, when the offset distance is [0, 1], the intensity number is 0 (indicating no obvious movement); when the offset distance is (1, 3], the intensity number is 1 (indicating small movement); when the offset distance is (3, 5], the intensity number is 2 (indicating medium movement); and when the offset distance is greater than 5, the intensity number is 3 (indicating large movement).

[0086] Therefore, by first extracting features from each face in the reference video frame to obtain facial action feature representation, and then adding the facial action feature identifier with the intensity number feature to obtain the second input feature representation, each intensity number can describe the range of motion of a specific area of ​​the face (such as the tip of the nose, the corner of the mouth, the eyebrows, etc.), thereby making the second action feature representation more accurate and improving the naturalness and realism of the simulated animation.

[0087] As one possible implementation, a specific approach to using an intensity-aware action extraction network to extract features from the second input feature representation to obtain a second action feature representation is as follows: Referring again to Figure 5, the first normalization layer in the intensity-aware action extraction network processes the second input feature to obtain a first intermediate feature representation; the multi-head attention network in the intensity-aware action extraction network processes the first intermediate feature representation to obtain a second intermediate feature representation; the second normalization layer in the intensity-aware action extraction network processes the second intermediate feature representation to obtain a third intermediate feature representation; and the multilayer perception network in the intensity-aware action extraction network processes the third intermediate feature representation to obtain the second action feature representation. Thus, the intensity-aware action extraction network is refined into a first normalization layer, a multi-head attention network, a second normalization layer, and a multilayer perception network. By analyzing facial expressions and head postures in the reference video through this refined intensity-aware action extraction network, implicit representations that can control animation details are generated, thereby learning richer action feature representations and improving the accuracy of actions in the simulated animation. Taking into account both the intensity and complexity of the movement, it is possible to more accurately capture and reproduce the dynamic characteristics of the characters, thereby improving the naturalness and realism of the animation.

[0088] To further learn the temporal information of the context in video frames, an action memory can be added to the intensity-aware action extraction network during action extraction. As shown in Figure 5, when the server calls the multi-head attention network in the intensity-aware action extraction network to process the first intermediate feature representation to obtain the second intermediate feature representation, the following process can be adopted: Obtain the learnable action feature representation, which is the feature representation already stored in the action memory; call the multi-head attention network in the intensity-aware action extraction network to process the first intermediate feature representation and the learnable action feature representation to obtain the second intermediate feature representation. It should be understood that the action memory is a knowledge base or storage system specifically used to store and manage action data, typically used for action-related generation, recognition, analysis, or simulation tasks. It can be viewed as a database containing various action information, enabling models or systems to retrieve or utilize this action information when needed.

[0089] A typical action memory bank usually includes the following components:

[0090] (1) Action feature data

[0091] Key point data: Stores the trajectory and positional changes of key points of a target object (such as the human body, face, or hand). Key points may include: two-dimensional or three-dimensional coordinate sequences of body joints (such as knees, elbows, etc.); motion trajectories of facial key points (such as corners of the eyes, corners of the mouth, tip of the nose, etc.); and dynamic changes of finger joints.

[0092] Action feature vector: A high-dimensional feature representation used to describe an action, which may be generated by neural networks, PCA (principal component analysis) or other feature extraction methods.

[0093] (2) Action Tags and Classification

[0094] Action tags: Add semantic tags to each action data, such as "walking", "running", "waving", "nodding", etc.

[0095] Action classification: Action data is classified according to the type, speed, intensity or other characteristics of the action to facilitate efficient retrieval.

[0096] (3) Time series data

[0097] Action memory banks typically store time-series information to describe the dynamic changes in actions. For example:

[0098] The movement trajectory of the target object over a period of time (such as the process from standing to walking).

[0099] The time segments of an action (such as the start, duration, and end time of the action).

[0100] (4) Action video or sample data

[0101] The action memory bank may also include video clips, images, or animations related to the action as reference material.

[0102] (5) Action-related information

[0103] The relationships between actions, for example:

[0104] The continuity of certain actions (such as "jumping" followed by "landing").

[0105] The intensity of actions (such as the difference in intensity between "smiling" and "laughing").

[0106] Therefore, in the process of motion feature learning, the motion features of the implicit representation are enhanced by the learnable motion feature representation, thereby supplementing the temporal information of the action based on the context information. That is, by storing and retrieving the context information between different video frames, the continuity and complexity of the corresponding motion in the video frame are captured and understood, thereby realizing smoother and more accurate dynamic changes in the simulated animation.

[0107] 404. Decode the simulation animation based on the first feature representation, the first action feature representation, and the second action feature representation.

[0108] Decoding refers to converting the feature representation into an image that the user can understand. For example, by performing decoding based on the first feature representation, the first action feature representation, and the second action feature representation, a simulated animation that the user can understand can be obtained.

[0109] As one possible implementation, after obtaining the first feature representation, the first action feature representation, and the second action feature representation, the server inputs the first feature representation, the first action feature representation, and the second action feature representation into a denoising network for learning, i.e., performing decoding processing, to reconstruct the simulated animation.

[0110] Therefore, implicit feature vectors are extracted from reference images and videos. These feature vectors include high-dimensional information such as facial expressions and poses. Furthermore, these feature vectors are continuous value vectors, which can represent richer facial information, thereby improving the precision of capturing dynamic facial details. On the other hand, facial features and motion features are learned separately. This allows for good decoupling of facial features and motion features across different image styles, improving the generalization ability of the solution.

[0111] To extract and process fine-grained features from the reference image, ensuring that the generated simulated animation accurately reproduces background details while preserving character identity features, thereby improving the overall visual realism of the simulated animation, the server can also add a fine-grained appearance extraction network to extract identity features from the reference image to obtain a character identity feature representation. The process of extracting the character identity feature representation by the server can be as follows: the server extracts identity features from the reference image to obtain an identity code identifier; simultaneously, it extracts global features from the reference image to obtain a global feature representation; based on the identity code representation and the global feature representation, it determines the character identity feature representation of the parameter image, which is used to generate the simulated animation.

[0112] Among them, identity feature extraction refers to extracting features of the person's identity in the reference image, while global feature extraction refers to extracting features of the entire reference image.

[0113] Therefore, by extracting and processing fine-grained features from the reference image, the generated simulated animation can accurately reproduce background details while maintaining the character's identity, thereby improving the overall visual realism of the simulated animation.

[0114] As one possible implementation, the network structure of the aforementioned fine-grained appearance extraction network can be as shown in Figure 6. Based on the network structure shown in Figure 6, the process of the server performing identity encoding and identification can be as follows: The server performs facial cropping on the reference image to obtain a first facial image; the convolutional network (identity encoder as shown in Figure 6) is called to encode the identity of the first facial image to obtain the identity encoding representation of the corresponding human image in the reference image.

[0115] Therefore, facial cropping can remove information that is not relevant to the face from the reference image, thereby ensuring that the generated simulated animation maintains the identity characteristics of the person while improving the accuracy of the identity encoding representation, thus improving the overall visual realism of the simulated animation.

[0116] As one possible implementation, the process of the server performing global feature extraction on the reference image to obtain a global feature representation can be as follows: As shown in Figure 6, the server calls a multi-head attention layer to perform global feature extraction on the reference image to obtain a global feature representation.

[0117] Furthermore, in the development of fine-grained appearance extraction networks, to achieve more refined appearance control, a multi-scale adapter for identity perception (also referred to as the identity-aware adapter in this application, as shown in Figure 6) is proposed to enhance the model's ability to preserve portrait identity information and background details. In the network structure shown in Figure 6, when determining the person's identity feature representation of the parameter image based on the identity encoding representation and the global feature representation, the identity-aware adapter can be used to adapt the global feature representation to the identity encoding representation to obtain the fourth intermediate feature representation; simultaneously, the global feature representation will also be input into the multilayer perceptual network for feature learning to obtain the tenth intermediate feature representation; finally, the two intermediate feature representations (i.e., the fourth intermediate feature and the tenth intermediate feature) are added together to obtain the person's identity feature representation.

[0118] Therefore, by using a multi-scale adapter and a multi-layer perceptual network to extract and fuse multi-scale features from reference images, the consistency of character identity can be maintained during the generation of simulated animations, and the ability to capture facial details can be improved, so as to achieve precise control over character identity and background in simulated animations.

[0119] In an exemplary scheme, the network structure of the identity-aware adapter can be as shown in Figure 7. Based on the network structure shown in Figure 7, the processing flow of the identity-aware adapter can be as follows: The global feature representation is downsampled through a downsampling layer to obtain the corresponding latent features; then, a multi-size convolutional layer is used to extract features from the latent features to obtain multi-size feature representations, and these multi-size feature representations are concatenated to obtain an intermediate feature representation; this intermediate feature representation and the identity feature (i.e., the identity encoding representation) are used as input to the multi-head attention layer for feature learning to obtain another intermediate feature representation; this intermediate feature representation is upsampled to obtain a multi-size feature fusion intermediate feature representation. This maintains the consistency of character identity during the generation of simulated animations and improves the ability to capture facial details, thereby achieving precise control over character identity and background in simulated animations.

[0120] Based on the above scheme, a simulated animation can be obtained through a denoising network. The process of the denoising network is as follows: The server calls multiple convolutional modules of the denoising network to learn features from the character identity feature representation, the first feature representation, the first action feature representation, and the second action feature representation to obtain a denoised feature representation. The denoising network is generated by stacking multiple convolutional modules. The first convolutional module includes a spatial convolutional layer, a spatiotemporal convolutional layer, a spatiotemporal attention layer, an appearance attention layer, and an action attention layer, while the other convolutional modules (i.e., convolutional modules other than the first convolutional module) include a spatiotemporal convolutional layer, a spatiotemporal attention layer, an appearance attention layer, and an action attention layer. The decoder of the variational network is called to decode the denoised feature representation to obtain the simulated animation.

[0121] Therefore, a denoising network is used to progressively remove noise from the input conditions in the latent space, thereby generating a simulated animation that matches the input conditions. This improves generation efficiency and enhances the quality and temporal consistency of the simulated animation. Furthermore, through an attention mechanism, motion and appearance features are precisely integrated into the generation process of the simulated animation, ensuring that the generated simulated animation not only visually matches the reference image but also dynamically matches the reference video.

[0122] The following section illustrates the feature learning process of this denoising network using the execution process of a specific convolutional module:

[0123] The server calls the spatial convolutional layer in the denoising network to learn features from the first feature representation and the first action feature representation to obtain a fifth intermediate feature representation; it calls the spatiotemporal convolutional layer in the denoising network to learn features from the fifth intermediate feature representation to obtain a sixth intermediate feature representation; it calls the spatiotemporal attention layer in the denoising network to learn features from the sixth intermediate feature representation to obtain a seventh intermediate feature representation; it calls the appearance attention layer in the denoising network to learn features from the seventh intermediate feature representation and the person's identity feature representation to obtain an eighth intermediate feature representation; it calls the action attention layer in the denoising network to learn features from the eighth intermediate feature representation and the second action feature representation to obtain a ninth intermediate feature representation; and so on, passing the remaining convolutional modules sequentially to obtain the denoised feature representation.

[0124] Specifically, the process of learning features from the first convolutional module in the denoising network involves calling the spatial convolutional layer in the denoising network to learn features from the first feature representation and the first action feature representation to obtain a fifth intermediate feature representation; calling the spatiotemporal convolutional layer in the denoising network to learn features from the fifth intermediate feature representation to obtain a sixth intermediate feature representation; calling the spatiotemporal attention layer in the denoising network to learn features from the sixth intermediate feature representation to obtain a seventh intermediate feature representation; calling the appearance attention layer in the denoising network to learn features from the seventh intermediate feature representation and the person's identity feature representation to obtain an eighth intermediate feature representation; and calling the action attention layer in the denoising network to learn features from the eighth intermediate feature representation and the second action feature representation to obtain a ninth intermediate feature representation.

[0125] The feature learning process, sequentially passing through the remaining convolutional modules, refers to the process of feature learning through other convolutional modules in the denoising network. The second convolutional module is used as an example. This second convolutional module includes a spatiotemporal convolutional layer, a spatiotemporal attention layer, an appearance attention layer, and an action attention layer. The spatiotemporal attention layer of the second convolutional module is used to learn features from the ninth intermediate feature representation to obtain the eleventh intermediate feature representation. The appearance attention layer of the second convolutional module is used to learn features from the eleventh intermediate feature representation and the character identity feature representation to obtain the twelfth intermediate feature representation. The action attention layer of the second convolutional module is used to learn features from the twelfth intermediate feature representation and the second action feature representation to obtain the thirteenth intermediate feature representation. Subsequent convolutional modules follow the same pattern as the second convolutional module until all convolutional modules in the denoising network have performed feature learning, resulting in the simulated animation.

[0126] Therefore, a denoising network is used to progressively remove noise from the input conditions in the latent space, thereby generating a simulated animation that matches the input conditions. This improves generation efficiency and enhances the quality and temporal consistency of the simulated animation. Furthermore, through an attention mechanism, motion and appearance features are precisely integrated into the generation process of the simulated animation, ensuring that the generated simulated animation not only visually matches the reference image but also dynamically matches the reference video.

[0127] As described above, this embodiment, based on a stable video diffusion model, integrates identity and motion information through appearance and motion attention mechanisms, eliminating the need for fine-tuning the image diffusion model and separately training the motion module. Simultaneously, it reduces background jitter and portrait blurring when the identities of the individuals in the source image (i.e., the reference image) and the driving video (reference video) differ, thus improving video quality.

[0128] Based on the above scheme, the following describes an animation generation method in this application using the network structure (i.e., the generative model) shown in Figure 8: The generative model shown in Figure 8 includes a variational network, a denoising network, a fine-grained appearance extraction network, a pose extraction network, and an intensity-aware action extraction network. Among these, an action memory (not shown in Figure 8) assists the intensity-aware action extraction network, enhancing the implicit representation of motion features and supplementing temporal information based on contextual information.

[0129] The variational network is used to extract pixel signals from the raw video and convert them into modeling information in the latent space. An intensity-aware motion extraction network encodes motion information and serves as a control signal for the animation stage. The denoising network models features from the appearance and motion extraction networks in the latent space. The fine-grained appearance extraction network manages the identities of characters and the background in the generated video. The identity-aware adapter in the fine-grained appearance extraction network enhances the visual appearance representation. (The text then repeats the description of the intensity-aware motion extraction network.)

[0130] The network parameters of the aforementioned variational network, denoising network, fine-grained appearance extraction network, pose extraction network, and intensity-aware action extraction network can all be trained. The training process can be as follows:

[0131] Construct training samples. In this embodiment, the training samples include training videos and training images, wherein the training image should be a video frame extracted from the training video. Furthermore, to improve the generalization ability of the generative model, the training samples should be diversified.

[0132] In this embodiment, a series of data augmentation and training strategies are also set up to adapt to different animation styles, and spatial control condition mapping is used to improve the stability of areas other than the face during the inference stage.

[0133] In one exemplary approach, to maximize the diversity of training samples within a limited pool of samples, data color enhancement and processing based on different style transfer networks can be employed to increase the number of training samples.

[0134] Data color enhancement is a method that generates new samples by altering image color attributes (such as brightness, contrast, and saturation). It manipulates the pixel values ​​of an image to generate samples with different color distributions while preserving the semantic information of the image. This method can simulate various scenarios, including different lighting conditions and camera settings, thereby improving the model's robustness to color changes. For example, suppose the original training sample is a face image. The original image is a frontal photograph of a face under normal lighting.

[0135] The color-enhanced samples can be categorized as follows:

[0136] Adjust brightness: Generate a brighter image and a darker image.

[0137] Adjust contrast: Generate a high-contrast image (with sharper facial contours) and a low-contrast image (with a softer overall look).

[0138] Change the saturation: Generate a brightly colored image and a near-grayscale image.

[0139] Add noise: Generate an image with random noise.

[0140] Style transfer is a deep learning-based technique that generates new images with different visual styles by applying a target style (such as artistic style or specific texture) to the original image. Style transfer networks not only increase the number of training samples but also introduce diverse image styles, thereby enhancing the model's adaptability to style changes. For example, suppose the original training sample is a face image. The original image is a photograph of a face taken under normal conditions.

[0141] The style-transferred samples can include the following categories:

[0142] Oil painting style: Transform facial images into the style of Van Gogh or Monet oil paintings.

[0143] Sketch Style: Transforms facial images into black and white sketches.

[0144] Cartoon style: Uses an animation style transfer network to generate face images with a cartoon effect.

[0145] Retro Style: Generates images with retro filter effects, simulating the style of old photos.

[0146] Cyberpunk Style: Generates images with neon lighting effects and high contrast, simulating the visual effects of cyberpunk scenes.

[0147] Then, a variational network is used to extract features from the video and image data in the training samples to obtain latent features. Random noise is then added to the latent features as input to the denoising network. At the same time, other networks are called to perform corresponding feature extraction to obtain corresponding feature representations. The corresponding feature representations are then input into the corresponding attention layers for denoising learning. Finally, the variational network is used for decoding to obtain the predicted animation.

[0148] The loss value is calculated based on the predicted animation and the training video of the training sample; finally, the generative model is trained based on the loss value to obtain a generative model that can be applied online.

[0149] It should be understood that when generative models are applied to online inference, the stability of regions other than the face can be improved through spatial control conditional mapping. For example, to generate better simulated animations with skeletal differences in different body shapes and spatial positions, certain strategies can be used to adjust positions and proportions. That is, in animation generation or motion transfer tasks, characters in reference images and reference videos may have different body shapes (such as height, build, limb proportions, etc.). Differences in skeletal structure can lead to distortions and unnatural effects when directly transferring the driven action. For example, transferring the action of a long arm to a short-armed character may cause the action to exceed the natural range. On the other hand, characters in reference videos and reference images may be located in different spatial positions (e.g., the driving character may be standing, while the source character may be sitting or leaning). If the differences in spatial position are not considered, motion transfer may lead to misalignment or distortion in the generated results.

[0150] To address the aforementioned issues, appropriate reasoning strategies can be set during the online generation process.

[0151] In one exemplary approach, the inference strategy involves adjusting using standardized skeletal proportions. This means normalizing the skeletons in the reference image and video, mapping different bone lengths to a standardized proportional space. For example, the bone proportions can be calculated using the coordinates of keypoints (such as shoulders, elbows, and wrists); then, the motion in the reference video is scaled according to the standardized proportions to fit the skeletal structure of the reference image.

[0152] In another exemplary scheme, the inference strategy is to perform non-linear scaling adjustments. That is, for certain specific body shapes (such as babies or cartoon characters), a non-linear scaling strategy needs to be introduced to adjust different parts (such as the head, limbs, and torso) independently.

[0153] In another exemplary approach, the inference strategy is body shape matching. That is, the skeleton in the reference video is deformed using a shape matching algorithm (such as Thin-Plate Spline transform) to make it consistent with the skeleton shape of the reference image.

[0154] In another exemplary scheme, the inference strategy is keypoint alignment. That is, by translating and rotating the keypoint coordinates of the reference video and reference image, the skeletal center points (such as the midpoint of the pelvis or spine) of the two are made to coincide.

[0155] In another exemplary approach, the inference strategy is pose normalization. This involves mapping the pose of the skeleton in the reference video to the initial pose of the reference image, for example, by calculating a rotation matrix or affine transformation, to adjust the overall position of the skeleton in the reference image to a coordinate system consistent with that of the skeleton in the reference image.

[0156] In another exemplary approach, the inference strategy is flexible alignment. That is, if the initial postures of the two are significantly different (e.g., the driving character is standing while the source character is sitting), the action needs to be flexibly transformed, for example, by gradually transitioning the standing action to the sitting action.

[0157] The beneficial effects of this application are illustrated below with specific experimental results:

[0158] Experimental results (as shown in Table 1) demonstrate that the solution provided in this application outperforms existing technologies in both Frechet sensing distance and Frechet video distance, proving its superiority in generating video quality.

[0159] The results shown in Table 1 are objective comparison results on the VFHQ dataset. Indicators marked with ↓ are better the lower they are, and indicators marked with ↑ are better the higher they are.

[0160] Table 1

[0161] FADM stands for Face animation with an attribute-guided diffusion model, a diffusion model framework for generating high-fidelity photorealistic talking head videos. FollowYourEmoji stands for Fine-controllable and expressive freestyle portrait animation.

[0162] Secondly, in terms of enhancing identity preservation and facial dynamic generation capabilities, the technical solution provided in this application effectively preserves the identity features of the person in the source image while more accurately capturing facial dynamics through the application of an identity-aware multi-scale adapter and an action memory bank. As shown in Tables 1 and 2, the technical solution provided in this application shows significant improvements over existing methods in terms of identity similarity, facial motion quality, image visual quality, and image smoothness, demonstrating its superior performance in processing complex portraits.

[0163] Secondly, regarding the improvement of video detail capture capabilities, the technical solution provided in this application, through a fine-grained appearance extraction network and an intensity-aware motion extraction network, is able to capture richer facial details and dynamics. As shown in Table 2, the technical solution provided in this application performs better in capturing facial movements such as eye movement direction, eye rotation, and lip synchronization. These details are often difficult to capture in previous technologies, therefore, these methods show a significant gap compared to the technical solution provided in this application in terms of facial movement quality.

[0164] The results shown in Table 1 are subjective comparison results on the VFHQ dataset. Indicators marked with ↓ are better the lower they are, and indicators marked with ↑ are better the higher they are.

[0165] Table 2

[0166] The animation generation apparatus of this application is described in detail below. Please refer to Figure 9, which is a schematic diagram of one embodiment of the animation generation apparatus in this application. The animation generation apparatus 20 includes:

[0167] The acquisition module 201 is used to acquire a reference video and a reference image. The reference video is used to indicate the behavior and actions of the target object in the simulated animation, and the reference image is used to indicate the portrait information of the target object in the simulated animation.

[0168] The processing module 202 is configured to perform feature extraction on the reference video and the reference image to obtain a first feature representation, which is a latent feature of the reference video and the reference image; perform feature extraction on the reference video to obtain a first motion feature representation and a second motion feature representation, wherein the first motion feature representation is used to indicate the global motion features corresponding to the reference video, and the second motion feature representation is used to indicate the facial motion features corresponding to the reference video; and perform decoding processing based on the first feature representation, the first motion feature representation, and the second motion feature representation to obtain the simulated animation.

[0169] This application provides an animation generation apparatus. Using this apparatus, implicit feature vectors of images are extracted from reference images and reference videos. These feature vectors include high-dimensional information such as facial expressions and poses. Furthermore, these feature vectors are continuous value vectors, which can represent richer facial information, thereby improving the precision of capturing facial dynamic details. On the other hand, facial features and motion features are learned separately, which enables good decoupling of facial features and motion features under different image styles, improving the generalization ability of the solution.

[0170] Optionally, based on the embodiment corresponding to FIG9 above, in another embodiment of the animation generation apparatus 20 provided in this application,

[0171] The processing module 202 is used to call the first variational network to extract features from the pixels of each video frame in the reference video to obtain the first pixel signal;

[0172] The second variational network is invoked to extract features from the pixels in the reference image to obtain the second pixel signal;

[0173] The first pixel signal and the second pixel signal are modeled in the latent space and their features are concatenated to obtain the first feature representation.

[0174] This application provides an animation generation apparatus. Using this apparatus, feature extraction is performed on reference images and reference videos based on pixel points, and latent space modeling is then performed and concatenated to learn effective feature representations of the reference images and videos. These feature vectors are continuous value vectors, which can represent richer facial information, thereby improving the precision of capturing facial dynamic details. The use of variational networks enables the efficient learning and generation of image and video feature representations in a low-dimensional, continuous latent space.

[0175] Optionally, based on the embodiment corresponding to FIG9 above, in another embodiment of the animation generation apparatus 20 provided in this application, the processing module 202 is used to call the pose extraction network to perform global motion feature extraction on the reference video to obtain the first motion feature representation;

[0176] The intensity-aware action extraction network is invoked to extract facial features from the reference video to obtain the second action feature representation.

[0177] This application provides an animation generation apparatus. Using this apparatus, global motion features and facial motion features are extracted using different network layers. By learning these global pose and facial motion features, richer motion feature representations can be learned, thereby improving the motion accuracy in simulated animations.

[0178] Optionally, based on the embodiment corresponding to FIG9 above, in another embodiment of the animation generation apparatus 20 provided in this application,

[0179] The processing module 202 is used to perform facial cropping on each video frame in the reference video to obtain multiple facial images;

[0180] A convolutional network is invoked to extract features from the multiple facial images to obtain facial action feature representations;

[0181] The facial motion feature representation is added to the intensity number to obtain the second input feature representation, which is used to indicate the intensity feature representation corresponding to the facial key point;

[0182] The intensity-aware action extraction network is invoked to extract features from the second input feature representation to obtain the second action feature representation.

[0183] In this embodiment, an animation generation apparatus is provided. Using this apparatus, features are first extracted from each face in a reference video frame to obtain facial motion feature representations. The facial motion feature identifiers are then added to intensity number features to obtain a second input feature representation. Each intensity number can describe the range of motion of a specific facial region (such as the tip of the nose, corners of the mouth, eyebrows, etc.), thereby making the second motion feature representation more accurate and improving the naturalness and realism of the simulated animation.

[0184] Optionally, based on the embodiment corresponding to FIG9 above, in another embodiment of the animation generation apparatus 20 provided in this application,

[0185] The processing module 202 is used to call the first normalization layer in the intensity-aware action extraction network to perform feature processing on the second input feature to obtain a first intermediate feature representation;

[0186] The multi-head attention network in the intensity-aware action extraction network is invoked to process the first intermediate feature representation to obtain the second intermediate feature representation;

[0187] The second normalization layer in the intensity-aware action extraction network is invoked to process the second intermediate feature representation to obtain the third intermediate feature representation.

[0188] The multilayer perceptual network in the intensity-aware action extraction network is invoked to process the third intermediate feature representation to obtain the second action feature representation.

[0189] This application provides an animation generation apparatus. Using this apparatus, the intensity-aware motion extraction network is refined into a first normalization layer, a multi-head attention network, a second normalization layer, and a multi-layer perceptual network. The refined intensity-aware motion extraction network analyzes facial expressions and head postures in a reference video to generate implicit representations that control animation details. This allows for the learning of richer motion feature representations, thereby improving the accuracy of motion in simulated animation. Simultaneously considering the intensity and complexity of motion, it can more accurately capture and reproduce the dynamic features of characters, thus improving the naturalness and realism of the animation.

[0190] Optionally, based on the embodiment corresponding to FIG9 above, in another embodiment of the animation generation apparatus 20 provided in this application, the processing module 202 is used to obtain a learnable action feature representation, which is a feature representation already stored in the action memory bank;

[0191] The multi-head attention network in the intensity-aware action extraction network is invoked to perform feature processing on the first intermediate feature representation and the learnable action feature representation to obtain the second intermediate feature representation.

[0192] This application provides an animation generation apparatus. Using this apparatus, during the motion feature learning process, the implicitly represented motion features are enhanced, thereby supplementing the temporal information of the motion based on contextual information. Specifically, by storing and retrieving contextual information between different video frames, the continuity and complexity of corresponding motions within video frames are captured and understood, thereby achieving smoother and more accurate dynamic changes in simulated animation.

[0193] Optionally, based on the embodiment corresponding to FIG9 above, in another embodiment of the animation generation apparatus 20 provided in this application, the processing module 202 is used to extract identity features from the reference image to obtain an identity code representation;

[0194] Global feature extraction is performed on the reference image to obtain a global feature representation;

[0195] Based on the identity encoding representation and the global feature representation, the character identity feature representation of the parameter image is determined, and the character identity feature representation is used to generate the simulated animation.

[0196] In this embodiment of the application, an animation generation apparatus is provided. Using this apparatus, by extracting and processing fine-grained features from a reference image, it ensures that the generated simulated animation accurately reproduces background details while maintaining the character's identity features, thereby improving the overall visual realism of the simulated animation.

[0197] Optionally, based on the embodiment corresponding to FIG9 above, in another embodiment of the animation generation apparatus 20 provided in this application,

[0198] The processing module 202 is used to crop the face of the reference image to obtain a first face image;

[0199] The convolutional network is invoked to encode the identity of the first facial image, so as to obtain the identity encoding representation of the corresponding human image in the reference image.

[0200] In this embodiment, an animation generation apparatus is provided. Using this apparatus, facial cropping can remove information useless to the face from the reference image, thereby ensuring that the generated simulated animation maintains the character's identity features while improving the accuracy of the identity encoding representation, thus enhancing the overall visual realism of the simulated animation.

[0201] Optionally, based on the embodiment corresponding to FIG9 above, in another embodiment of the animation generation apparatus 20 provided in this application,

[0202] The processing module 202 is used to call the identity-aware adapter to adapt the global feature representation to the identity encoding representation in order to obtain a fourth intermediate feature representation;

[0203] A multilayer perceptron is invoked to learn the global feature representation in order to obtain the tenth intermediate feature representation;

[0204] The person's identity feature representation is obtained based on the fourth intermediate feature representation and the tenth intermediate feature representation.

[0205] This application provides an animation generation apparatus. Using this apparatus, multi-scale features of reference images are extracted and fused using a multi-scale adapter and a multi-layer perceptual network, thereby maintaining consistency in character identity during the generation of simulated animation and improving the ability to capture facial details, thus achieving precise control over character identity and background in the simulated animation.

[0206] Optionally, based on the embodiment corresponding to FIG9 above, in another embodiment of the animation generation apparatus 20 provided in this application,

[0207] The processing module 202 is used to call multiple convolutional modules of the denoising network to learn features from the person's identity feature representation, the first feature representation, the first action feature representation, and the second action feature representation to obtain a denoised feature representation. The denoising network is generated by stacking multiple convolutional modules. The first convolutional module includes a spatial convolutional layer, a spatiotemporal convolutional layer, a spatiotemporal attention layer, an appearance attention layer, and an action attention layer, while the other convolutional modules include a spatiotemporal convolutional layer, a spatiotemporal attention layer, an appearance attention layer, and an action attention layer.

[0208] The decoder of the third variational network is invoked to decode the denoised feature representation to obtain the simulated animation.

[0209] This application provides an animation generation apparatus. Using this apparatus, a denoising network is employed to progressively remove noise from the input conditions in the latent space, thereby generating a simulated animation that matches the input conditions. This improves generation efficiency and enhances the quality and temporal consistency of the simulated animation. Furthermore, through an attention mechanism, motion and appearance features are precisely integrated into the simulated animation generation process, ensuring that the generated simulated animation not only visually matches the reference image but also dynamically matches the reference video.

[0210] Optionally, based on the embodiment corresponding to FIG9 above, in another embodiment of the animation generation apparatus 20 provided in this application,

[0211] The processing module 202 is used to call the spatial convolutional layer in the denoising network to perform feature learning on the first feature representation and the first action feature representation to obtain the fifth intermediate feature representation;

[0212] The spatiotemporal convolutional layer in the denoising network is invoked to learn features from the fifth intermediate feature representation in order to obtain the sixth intermediate feature representation.

[0213] The spatiotemporal attention layer in the denoising network is invoked to learn features from the sixth intermediate feature representation in order to obtain the seventh intermediate feature representation.

[0214] The appearance attention layer in the denoising network is invoked to learn features from the seventh intermediate feature representation and the character identity feature representation to obtain the eighth intermediate feature representation;

[0215] The action attention layer in the denoising network is invoked to learn features from the eighth intermediate feature representation and the second action feature representation to obtain the ninth intermediate feature representation;

[0216] The features are then learned sequentially through the remaining convolutional modules to obtain the denoised feature representation.

[0217] This application provides an animation generation apparatus. Using this apparatus, a denoising network is employed to progressively remove noise from the input conditions in the latent space, thereby generating a simulated animation that matches the input conditions. This improves generation efficiency and enhances the quality and temporal consistency of the simulated animation. Furthermore, through an attention mechanism, motion and appearance features are precisely integrated into the simulated animation generation process, ensuring that the generated simulated animation not only visually matches the reference image but also dynamically matches the reference video.

[0218] The animation generation apparatus provided in this application can be used on a server. Please refer to Figure 10, which is a schematic diagram of a server structure provided in an embodiment of this application. The server 300 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and a memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. The memory 332 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the server. Furthermore, the CPU 322 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the server 300.

[0219] Server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0220] The steps performed by the server in the above embodiments can be based on the server structure shown in Figure 10.

[0221] The animation generation apparatus provided in this application can be used in terminal devices. Please refer to Figure 11. For ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. In the embodiments of this application, a smartphone is used as an example for illustration:

[0222] Figure 11 shows a block diagram of a partial structure of a smartphone related to the terminal device provided in the embodiments of this application. Referring to Figure 11, the smartphone includes components such as a radio frequency (RF) circuit 410, a memory 420, an input unit 430, a display unit 440, a sensor 450, an audio circuit 460, a wireless fidelity (WiFi) module 470, a processor 480, and a power supply 490. Those skilled in the art will understand that the smartphone structure shown in Figure 11 does not constitute a limitation on the smartphone, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0223] The following section, with reference to Figure 11, provides a detailed introduction to the various components of a smartphone:

[0224] RF circuit 410 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 480; additionally, it transmits uplink data to the base station. Typically, RF circuit 410 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 410 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Message Service (SMS).

[0225] The memory 420 can be used to store software programs and modules. The processor 480 executes various functions and data processing of the smartphone by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, phonebook, etc.). In addition, the memory 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0226] The input unit 430 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the smartphone. Specifically, the input unit 430 may include a touch panel 431 and other input devices 432. The touch panel 431, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 431), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 431 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 480, and can also receive and execute commands sent by the processor 480. In addition, the touch panel 431 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 431, the input unit 430 may also include other input devices 432. Specifically, other input devices 432 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0227] Display unit 440 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. Display unit 440 may include display panel 441, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar device. Further, touch panel 431 may cover display panel 441. When touch panel 431 detects a touch operation on or near it, it transmits the information to processor 480 to determine the type of touch event. Subsequently, processor 480 provides corresponding visual output on display panel 441 based on the type of touch event. Although in Figure 11, touch panel 431 and display panel 441 are shown as two separate components for implementing smartphone input and output functions, in some embodiments, touch panel 431 and display panel 441 can be integrated to achieve smartphone input and output functions.

[0228] The smartphone may also include at least one sensor 450, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 441 according to the ambient light level, and the proximity sensor can turn off the display panel 441 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the smartphone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the smartphone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0229] Audio circuit 460, speaker 461, and microphone 462 provide an audio interface between the user and the smartphone. Audio circuit 460 converts received audio data into electrical signals and transmits them to speaker 461, where speaker 461 converts them into sound signals for output. On the other hand, microphone 462 converts collected sound signals into electrical signals, which are received by audio circuit 460, converted into audio data, and then processed by processor 480 before being transmitted via RF circuit 410 to, for example, another smartphone, or the audio data can be output to memory 420 for further processing.

[0230] WiFi is a short-range wireless transmission technology. Smartphones, through the WiFi module 470, can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 11 shows the WiFi module 470, it is understood that it is not an essential component of a smartphone and can be omitted as needed without altering the essence of the invention.

[0231] The processor 480 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 420, and by calling data stored in the memory 420, thereby providing overall monitoring of the smartphone. Optionally, the processor 480 may include one or more processing units; optionally, the processor 480 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 480.

[0232] The smartphone also includes a power supply 490 (such as a battery) that supplies power to various components. Optionally, the power supply can be logically connected to the processor 480 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0233] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0234] The steps performed by the terminal device in the above embodiments can be based on the terminal device structure shown in Figure 11.

[0235] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0236] This application also provides a computer program product including a program, which, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0237] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0238] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0239] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0240] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0241] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0242] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Images