Digital-human control method, and server, storage medium and program product

Abstract

Provided in the present disclosure are a digital-human control method, and a server, a storage medium and a program product. The method relates to artificial intelligence technology, and comprises: when controlling state switching of a digital human, determining a currently played frame and a target frame to be switched to; on the basis of whichever of the current frame and the target frame is a motion frame, selecting, from an original video sequence of the digital human, at least one switching frame corresponding to the motion frame; using video frame interpolation technology to generate a state switching sequence, which starts from the current frame, transitions through the at least one switching frame, and ends at the target frame; and playing the state switching sequence, so as to implement smooth switching of the digital human from an idle state to a non-idle state or from a non-idle state to an idle state. By means of adding, between a current frame and a target frame, at least one switching frame from an original video sequence as an intermediate transition frame, and using video frame interpolation technology to achieve smooth transition between non-consecutive frames, the smooth transition from the current frame to the target frame is achieved, thereby making the state switching of a digital human relatively natural and smooth.

Description

Digital human control methods, servers, storage media and software products

[0001] This disclosure claims priority to Chinese Patent Application No. 202411732339.0, filed on November 27, 2024, entitled "Digital Human Control Method, Server, Storage Medium and Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to artificial intelligence technology, and more particularly to a digital human control method, server, storage medium, and program product. Background Technology

[0003] With the continuous development of artificial intelligence technology, the exchange between digital people through computer programs and artificial intelligence technology has significant application value and is widely used in fields such as intelligent customer service, robots, games, and virtual reality.

[0004] Traditional real-time interactive digital humans typically achieve state switching through carousel video templates, or by switching between two states based on several fixed frames, such as switching a digital human from a silent standby state to a non-silent working state. The state switching of digital humans is unnatural and not smooth. Summary of the Invention

[0005] This disclosure provides a digital human control method, server, storage medium, and program product to solve the problem of unnatural and unsmooth state switching in digital humans.

[0006] In a first aspect, this disclosure provides a digital human control method, comprising:

[0007] When controlling the switching state of the digital human, the current frame being played and the target frame to be switched to are determined, wherein one of the current frame and the target frame is a silent frame and the other is an action frame;

[0008] Based on the action frames in the current frame and the target frame, at least one switching frame corresponding to the action frame is selected from the original video sequence of the digital human;

[0009] Using video frame interpolation technology, a state switching sequence is generated starting from the current frame, passing through at least one switching frame, and ending at the target frame;

[0010] Play the state switching sequence.

[0011] In a second aspect, this disclosure provides a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the server to perform the method provided in any of the foregoing aspects.

[0012] Thirdly, this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method provided in any of the foregoing aspects.

[0013] Fourthly, this disclosure provides a computer program product including a computer program that, when executed by a processor, implements the methods provided in any of the foregoing aspects.

[0014] The digital human control method, server, storage medium, and program product disclosed herein describe a method for controlling a digital human's state switching. The method determines the currently playing frame and the target frame to which the digital human will switch, where one frame is a silent frame and the other is an action frame. Based on the action frame in the current and target frames, at least one switching frame corresponding to the action frame is selected from the original video sequence of the digital human. Using video frame interpolation technology, a state switching sequence is generated, starting from the current frame, transitioning through at least one switching frame, and ending at the target frame. Playing this state switching sequence enables the digital human to switch from a silent state to a non-silent state, and vice versa. This disclosed solution, by selecting at least one switching frame from the original video sequence of the digital human, adding at least one switching frame as an intermediate transition frame between the current and target frames, and utilizing video frame interpolation technology to achieve a smooth transition between discontinuous frames, allows the generated state switching sequence to achieve a smooth transition from the current frame to the target frame, making the digital human's state switching more natural and fluid. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0016] Figure 1 is a flowchart of a digital human control method provided in an exemplary embodiment of the present disclosure;

[0017] Figure 2 is a flowchart of a preprocessing procedure provided in an exemplary embodiment of this disclosure;

[0018] Figure 3 is a flowchart of a digital human control method provided in another exemplary embodiment of this disclosure;

[0019] Figure 4 is a flowchart of a digital human control method provided in another exemplary embodiment of this disclosure;

[0020] Figure 5 is a flowchart of real-time digital human interaction provided in another exemplary embodiment of this disclosure;

[0021] Figure 6 is a schematic diagram of the structure of a server provided in an embodiment of this disclosure.

[0022] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0024] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0025] First, let's explain the terms used in this disclosure:

[0026] Human Pose Estimation (HPE) is a method for identifying and classifying human joints. Essentially, it is a method for capturing the coordinates of various joints in the human body (such as arms, head, torso, etc.), which are called key points that can describe a person's posture.

[0027] Video frame interpolation: This refers to adding one or more frames to the original two frames of a video to shorten the display time between the two frames, correct the illusion caused by the persistence of vision in the human eye, and thus improve the smoothness of the video.

[0028] Large models refer to deep learning models with a massive number of parameters, typically containing hundreds of millions, tens of billions, or even trillions of parameters. Large models are also known as foundation models (FM), which are pre-trained on large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. These models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and Multi-modal Pre-training Models.

[0029] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0030] To address the issue of unnatural and choppy state transitions in digital humans, this disclosure provides a digital human control method to achieve smooth transitions between digital human states. Specifically, when controlling the digital human's state transition (e.g., switching from a silent state to a non-silent state, or from a non-silent state to a silent state), the current frame being played and the target frame to be switched to are determined, where one of the current frame and the target frame is a silent frame and the other is an action frame. Based on the action frames in the current frame and the target frame, at least one switching frame corresponding to the action frame is selected from the digital human's original video sequence. Using video frame interpolation technology, a state transition sequence is generated starting from the current frame, transitioning through at least one switching frame, and ending at the target frame. The state transition sequence is then played to achieve the switching of the digital human from a silent state to a non-silent state, and from a non-silent state to a silent state.

[0031] Among them, a silent frame refers to a video frame in which the digital human is in a silent state, while an action frame refers to a video frame in which the digital human is in a non-silent state.

[0032] The scheme disclosed herein selects at least one switching frame from the original video sequence of the digital human, adds at least one switching frame as an intermediate transition frame between the current frame and the target frame, and uses video frame interpolation technology to achieve a smooth transition between discontinuous frames, so that the generated state switching sequence can achieve a smooth transition from the current frame to the target frame, making the state switching of the digital human more natural and smoother.

[0033] It should be noted that the solution disclosed herein can realize the switching of a digital human from a silent state to a non-silent state, and from a non-silent state back to a silent state. Based on this, the switching from the first non-silent state to the second non-silent state of the digital human is achieved by first switching the digital human from the first non-silent state to a silent state, and then switching the digital human from a silent state to the second non-silent state. The switching from the first silent state to the second silent state of the digital human can be directly achieved through video frame interpolation technology. Therefore, based on the solution disclosed herein, switching between any state of the digital human can be realized, achieving a smooth transition between any state of the digital human, thereby enabling a smooth transition of the entire real-time interactive digital human process.

[0034] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0035] Figure 1 is a flowchart of a digital human control method provided in an exemplary embodiment of this disclosure. The executing entity in this embodiment is a server that implements real-time interactive digital human functions, such as a server for a digital human interaction system or an intelligent customer service system. As shown in Figure 1, the specific steps of the method are as follows:

[0036] Step S101: When controlling the digital human to switch states, determine the current frame being played and the target frame to be switched to. One of the current frame and the target frame is a silent frame and the other is an action frame.

[0037] A silent frame refers to a video frame in which the digital human is in a silent state; that is, the digital human is in a silent state within a silent frame. An action frame refers to a video frame in which the digital human is in a non-silent state; that is, the digital human is in a non-silent state within an action frame.

[0038] In practical applications, by using pre-shot videos of models, the Human Pose Estimation (HPE) algorithm is employed to detect human pose in the videos, obtaining pose information for each frame. This includes the coordinates of various joints (key points describing the human posture). Based on the pose information from each frame of the model video, the digital human model is driven to assume the corresponding pose, generating the corresponding video frames of the digital human and obtaining the original video sequence of the digital human.

[0039] In the model video, the model is initially in a silent state, and the first few consecutive frames in the model video are silent frames. Therefore, the first few consecutive frames in the original video sequence of the digital human are also silent frames, forming a silent frame sequence.

[0040] After the real-time interactive digital human is activated, the server keeps the digital human in a silent state by playing a sequence of silent frames in a loop. Upon receiving user input, the server obtains the response information as output by calling a response interface. At this point, the server needs to control the digital human to switch from a silent state to a non-silent state, so that the silent digital human can begin broadcasting the output information. After the output information broadcast ends, the server needs to control the digital human to switch from a non-silent state back to a silent state, so that the digital human returns to a silent state after the broadcast has finished.

[0041] This embodiment can achieve a smooth transition from a silent state to a non-silent state for the digital human, as well as a smooth transition from a non-silent state to a silent state. When transitioning from a silent state to a non-silent state, the current frame is the currently playing silent frame, and the target frame is the action frame to which the digital human will switch. When transitioning from a non-silent state to a silent state, the current frame is the currently playing action frame, and the target frame is the silent frame to which the digital human will switch.

[0042] Furthermore, during the process of the digital human broadcasting output information, the corresponding emotion category can be identified. Different content segments within the output information may correspond to different emotion categories. For example, the emotion categories that the digital human can express include, but are not limited to: calmness, confusion, anger, joy, sadness, excitement, and surprise. While broadcasting content segments of different emotion categories, the digital human is in a non-silent state, but needs to make gestures that reflect different emotions, requiring the digital human to switch from one non-silent state to another. Through the scheme disclosed herein, the digital human can be switched from one non-silent state to a silent state first, and then from a silent state to another non-silent state, thus achieving the switching between different non-silent states.

[0043] Step S102: Based on the action frames in the current frame and the target frame, select at least one switching frame corresponding to the action frame from the original video sequence of the digital human.

[0044] In this embodiment, by performing human pose detection on pre-shot model videos, the pose information of each frame in the model video can be obtained, specifically including the coordinates of each joint point of the human body (i.e., the key points describing the human posture). Based on the pose information of each frame in the model video, the digital human model is driven to make the corresponding poses to generate the corresponding video frames of the digital human, thus obtaining the original video sequence of the digital human.

[0045] In the model video, the model is initially in a silent state, and the first few consecutive frames in the model video are silent frames. Therefore, the first few consecutive frames in the original video sequence of the digital human are also silent frames, forming a silent frame sequence.

[0046] After acquiring the current frame and the target frame to be switched, based on the action frames in the current and target frames, at least one similar video frame is found from the original video sequence of the digital human, serving as at least one switching frame corresponding to the action frame. The similarity between different video frames refers to the similarity of the digital human's poses in different video frames, which can be measured by the pose distance between video frames.

[0047] Step S103: Using video frame interpolation technology, generate a state switching sequence that starts from the current frame, transitions through at least one switching frame, and ends at the target frame.

[0048] After obtaining at least one switching frame corresponding to the action frame, video frame interpolation technology is used to generate a smooth sequence starting from the current frame, transitioning through at least one switching frame, and ending at the target frame, thus obtaining the state switching sequence.

[0049] At least one switching frame may include an action switching frame and a corresponding silent switching frame. When generating the state transition sequence from the current frame to the target frame, a subsequence from the silent switching frame to the action switching frame is extracted from the original video sequence of the digital human. This subsequence, or its inverse, is used as the subsequence between the current frame and the target frame in the state transition sequence. Using video frame interpolation technology, at least one transition frame is inserted between the current frame and the subsequence, and between the target frame and the subsequence, respectively, to generate a smooth sequence starting from the current frame, transitioning through at least one switching frame, and ending at the target frame. This results in a smooth state transition sequence from the current frame to the target frame.

[0050] In an optional embodiment, when switching the digital human from a silent state to a non-silent state, the current frame is the currently playing silent frame, and the target frame is the action frame to be switched to. Through step S102, the server can select the video frame with the smallest pose distance between the target frame and each video frame in the original video sequence of the digital human, as the action switching frame. From the video frames in the original video sequence preceding this action switching frame, the video frame with the smallest pose distance to the starting silent frame less than or equal to a distance threshold and the shortest interval between it and the action switching frame is selected as the silent switching frame corresponding to the action switching frame.

[0051] Further, in step S103, the server uses video frame interpolation technology to generate a smooth sequence from the current frame (silent frame) to the silent transition frame, denoted as sequence S1; and uses video frame interpolation technology to generate a smooth sequence from the action transition frame to the target frame (action frame), denoted as sequence S3. For the sequence from silent transition frame to action transition frame, a sequence from silent transition frame to action transition frame can be extracted from the original video sequence of the digital human, denoted as sequence S2. By concatenating sequences S1, S2, and S3 in sequence, a state transition sequence can be obtained, starting from the current frame, transitioning through at least one transition frame, and ending at the target frame.

[0052] In another optional embodiment, when switching the digital human from a non-silent state to a silent state, the current frame is the currently playing action frame, and the target frame is the silent frame to be switched to. Through step S102, the server can obtain n video frames after the current frame in the original video sequence of the digital human, as n first candidate frames; where n is a positive integer. For any first candidate frame, based on the pose distance between the first candidate frame and each video frame in the original video sequence of the digital human, the video frame with the smallest pose distance is selected as the second candidate frame corresponding to the first candidate frame, and the first and second candidate frames form a candidate frame pair. Based on the pose distance between the two candidate frames in the candidate frame pair, the candidate frame pair with the smallest pose distance between the two candidate frames is selected, and the first and second candidate frames in this candidate frame pair are respectively used as the first action switching frame and the second action switching frame. From the video frames before the second action switching frame in the original video sequence, the video frame with the pose distance to the starting silent frame less than or equal to a distance threshold and the shortest interval duration with the second action switching frame is found, and is used as the silent switching frame corresponding to the second action switching frame.

[0053] Further, in step S103, the server extracts a sequence from the current frame (action) to the first action frame from the original video sequence of the digital human, denoted as sequence S1'; using video frame interpolation technology, a smooth sequence from the first action frame to the second action frame is generated, denoted as sequence S2'; a sequence from the silent transition frame to the second action frame is extracted from the original video sequence, denoted as sequence S3', and the reverse sequence of sequence S3' is generated to obtain a smooth sequence from the second action frame to the silent transition frame; using video frame interpolation technology, a smooth sequence from the silent transition frame to the starting silent frame is generated, denoted as sequence S4'. By sequentially concatenating the reverse sequences of sequences S1', S2', and S3' with sequence S4', a state switching sequence is obtained, starting from the current frame, transitioning through the first action frame, the second action frame, and the silent transition frame, to the end of the target frame.

[0054] In another optional embodiment, when switching the digital human from a non-silent state to a silent state, the current frame is the currently playing action frame, and the target frame is the silent frame to which it is to be switched. Through step S102, the server can select the video frame with the smallest pose distance between the current frame (action frame) and each video frame in the original video sequence of the digital human, as the action switching frame. From the video frames in the original video sequence before the action switching frame, find the video frame with the pose distance to the starting silent frame less than or equal to a distance threshold and the shortest interval duration between it and the action switching frame, as the silent switching frame corresponding to the action switching frame.

[0055] Further, in step S103, the server uses video frame interpolation technology to generate a smooth sequence from the current frame (action frame) to the action switching frame, denoted as sequence S1”; it extracts a sequence from the silent switching frame to the action switching frame from the original video sequence of the digital human, denoted as sequence S2”, and generates the reverse sequence of sequence S2”; and uses video frame interpolation technology to generate a smooth sequence from the silent switching frame to the target frame (silent frame), denoted as sequence S3”. By sequentially concatenating sequence S1”, the reverse sequence of sequence S2”, and sequence S3”, a state switching sequence is obtained, starting from the current frame, transitioning through the action switching frame and the silent switching frame, and ending at the target frame.

[0056] Step S104: Playback state switching sequence.

[0057] After obtaining a smooth state transition sequence from the current frame to the target frame, the state transition sequence is played to achieve the state transition of the digital human.

[0058] In this embodiment, when controlling the digital human's state switching (e.g., switching from a silent state to a non-silent state, or from a non-silent state to a silent state), the current frame being played and the target frame to be switched to are determined. One of the current frame and the target frame is a silent frame, and the other is an action frame. Based on the action frame in the current frame and the target frame, at least one switching frame corresponding to the action frame is selected from the digital human's original video sequence. Using video frame interpolation technology, a state switching sequence is generated, starting from the current frame, transitioning through at least one switching frame, and ending at the target frame. Playing the state switching sequence realizes the switching of the digital human from a silent state to a non-silent state, and from a non-silent state to a silent state. By selecting at least one switching frame from the digital human's original video sequence, adding at least one switching frame as an intermediate transition frame between the current frame and the target frame, and using video frame interpolation technology to achieve a smooth transition between discontinuous frames, the generated state switching sequence can achieve a smooth transition from the current frame to the target frame, making the digital human's state switching more natural and smoother.

[0059] In one optional embodiment, the server can obtain and store the basic data required for the digital human's state switching through a preprocessing process. During the real-time interaction between the digital human and the user, the required basic data can be directly obtained, reducing the time consumption for real-time calculation of basic data and the resource consumption of repeated calculations, thereby improving the efficiency of the digital human's state switching.

[0060] Figure 2 is a flowchart of a preprocessing procedure provided in an exemplary embodiment of this disclosure. As shown in Figure 2, the specific steps of preprocessing to obtain the basic data required for digital human state switching are as follows:

[0061] Step S201: Obtain the original video sequence of the digital human. The initial state of the digital human in the original video sequence is a silent state. Take the first frame in the original video sequence as the starting silent frame.

[0062] In this step, a video of the model is obtained by shooting the model. The model is initially in a silent state in the video.

[0063] In one example scenario, the model's image in the model video serves as the digital human's image, and the filmed model video is used as the original video sequence for the digital human. Optionally, after obtaining the model video, preprocessing such as noise reduction can be performed on the model video before using it as the original video sequence for the digital human.

[0064] In another example scenario, the model's image in the model video differs from the digital human's image. By performing human pose detection on each frame of the model video, pose information for each frame can be obtained. The pose information for any frame includes the coordinates of each joint point of the human body (i.e., the key points describing the human posture). Furthermore, based on the digital human's pose information in each frame of the model video, the digital human model is driven to perform the corresponding poses to generate the corresponding video frames of the digital human, thus obtaining the original video sequence of the digital human.

[0065] In the model video, the model's initial state is silent, and the first few consecutive frames in the model video are silent frames. Therefore, the initial state of the digital human in the original video sequence is also silent, and the first few consecutive frames in the original video sequence are also silent frames, forming a silent frame sequence.

[0066] In this embodiment, for ease of description, the first frame (silent frame) in the original video sequence of the digital human is taken as the starting silent frame.

[0067] Step S202: Obtain the pose information of the digital human in each video frame of the original video sequence.

[0068] After obtaining the original video sequence of the digital human, the pose information of the digital human in each video frame of the original video sequence is obtained, specifically including the coordinates of each joint of the human body.

[0069] In one example scenario, the original video sequence of the digital human is a video of a model. In this step, a human pose estimation algorithm can be used to detect the human pose in each video frame of the original video sequence of the digital human, thereby obtaining the pose information of the digital human in each video frame.

[0070] In another example scenario, the original video sequence of the digital human is obtained by driving the digital human model to perform corresponding poses based on the pose information of the digital human in each frame of the model's video. Therefore, the pose information of the digital human in each video frame is obtained before the original video sequence of the digital human is generated.

[0071] After obtaining the pose information of the digital human in each video frame of the original video sequence, the server stores the pose information of the digital human in each video frame of the original video sequence for use in subsequent preprocessing and digital human state switching.

[0072] Step S203: Calculate the distance between the pose information of the digital human in any two video frames to obtain the pose distance between any two video frames.

[0073] In this embodiment, the distance between the pose information of the digital human in any two video frames is calculated based on the pose information of the digital human in each video frame of the original video sequence, thereby obtaining the pose distance between any two video frames.

[0074] Optionally, when calculating the pose distance between any two video frames, the mean or median of the distance between corresponding joints in the two video frames can be calculated as the pose distance between the two video frames.

[0075] Optionally, when calculating the pose distance between any two video frames, the pose information of the digital human in each video frame can be stored as a vector (here referred to as the pose vector of the video frame), and the distance between the pose vectors of the two video frames can be calculated as the pose distance between the two video frames. The calculation of the pose vector distance between the two video frames can be achieved using methods for calculating the distance between arbitrary vectors, such as calculating the Euclidean distance between the two pose vectors, 1 minus the cosine similarity between the two pose vectors, the Manhattan distance, etc., without specific limitations here.

[0076] In this embodiment, the server stores the pose distance between any two video frames for use in subsequent preprocessing and digital human state switching.

[0077] Step S204: Based on the pose distance between any two video frames, select video frames from the original video sequence whose pose distance to the starting silent frame is less than or equal to the distance threshold, obtain silent switching frames, and construct a silent switching frame pool.

[0078] In this step, based on the pose distance between the starting silent frame and other video frames in the original video sequence, video frames whose pose distance to the starting silent frame is less than or equal to a distance threshold are selected from the original video sequence as silent switching frames, and a silent switching frame pool containing these silent switching frames is constructed.

[0079] The distance threshold can be determined based on the pose distance between the initial silent frame and other video frames, ensuring that the proportion of the selected silent switching frames in the original video sequence does not exceed a preset percentage, such as 10% or 15%. The specific percentage can be configured and adjusted according to the actual application scenario, and is not specifically limited here. The distance threshold can also be configured as a fixed value, which can also be configured and adjusted according to the actual application scenario, and is not specifically limited here.

[0080] It should be noted that the silent transition frames selected from the original video sequence of the digital human in this step are video frames whose posture is similar to that of the digital human in the initial silent frame. The state of the digital human in the silent transition frame is close to the initial silent state. The silent transition frame may include silent frames, but generally it is not entirely silent frames. During the digital human's state transition process, the silent transition frame can serve as a transition frame between action frames and silent frames, making the digital human's state transition smoother and more natural.

[0081] Step S205: If there are consecutive silent switching frames in the silent switching frame pool, then one of the consecutive silent switching frames is taken as the reserved frame, and the silent switching frames other than the reserved frame are deleted from the consecutive silent switching frames.

[0082] In this step, for the existing silent switching frame pool, it is checked whether there are consecutive silent switching frames in the pool. If there are consecutive silent switching frames in the pool, one of the silent switching frames is retained and the rest are deleted.

[0083] For example, for a pool of consecutive silent switching frames, the last silent switching frame can be retained, and the remaining frames in the pool can be deleted. Alternatively, for any pool of consecutive silent switching frames, the last frame, an intermediate frame, or any random frame can be retained; no specific limitation is made here.

[0084] Step S206: In the original video sequence, obtain the video frame sequence from any silent switching frame to the frame with the largest attitude distance from the silent switching frame, and use it as the action frame sequence corresponding to the silent switching frame to construct the action frame sequence pool.

[0085] The action frame sequence pool includes multiple action frame sequences, with the starting frame of each action frame sequence being a silent switching frame. In this step, for any silent switching frame retained in the silent switching frame pool, the corresponding action frame sequence is extracted from the original video sequence of the digital human.

[0086] Specifically, in the original video sequence of the digital human, the frame with the largest pose distance relative to the silent switching frame is found starting from the silent switching frame. This frame represents the frame with the largest motion amplitude relative to the silent switching frame. The video frame sequence from the silent switching frame to the frame with the largest pose distance relative to the silent switching frame is extracted from the original video sequence and used as the action frame sequence corresponding to the silent switching frame.

[0087] After obtaining the action frame sequences corresponding to each silent switching frame in the silent switching frame pool, an action frame sequence pool including these action frame sequences is constructed. During the digital human state transition, based on the action frame sequence pool, similar video frames to the action frames in the current frame or target frame can be found as action switching frames, and the corresponding silent switching frames can be obtained as transition frames between the current frame and the target frame, making the digital human state transition smoother and more natural.

[0088] In one optional embodiment, after constructing the action frame sequence pool, corresponding action frames can be bound to each emotion category according to at least one emotion category that the digital human can express, in which the digital human presents the emotion of the corresponding emotion category well.

[0089] The solution in this embodiment obtains the original video sequence of the digital human, calculates the pose distance between any two video frames in the original video sequence, and selects silent switching frames from the original video sequence whose pose distance to the starting silent frame is less than or equal to a distance threshold based on the pose distance between any two video frames, thus constructing a silent switching frame pool. One frame from a series of consecutive silent switching frames in the silent switching frame pool is selected as the retained frame, and the silent switching frames other than the retained frame are deleted from the series of consecutive silent switching frames. The video frame sequence from the start of the silent switching frame to the frame with the largest pose distance to the silent switching frame is obtained from the original video sequence and used as the action frame sequence corresponding to the silent switching frame, thus constructing an action frame sequence pool. This provides a data foundation for finding the switching frame between the current frame and the target frame during the state switching process of the digital human, which can reduce the time consumption of real-time calculation of basic data and the resource consumption of repeated calculation of basic data, thereby improving the efficiency of digital human state switching.

[0090] Figure 3 is a flowchart of a digital human control method provided in another exemplary embodiment of this disclosure. Based on any of the foregoing embodiments, this embodiment provides a detailed implementation of controlling a digital human to switch from a silent state to a non-silent state. As shown in Figure 3, the process of a digital human switching from a silent state to a non-silent state includes the following steps:

[0091] Step S30: Control the digital human to be in a silent state.

[0092] After the real-time interactive digital human is activated, the server keeps the digital human in a silent state by playing a sequence of silent frames of the digital human in a forward and reverse loop.

[0093] Step S31: Determine whether input information has been received.

[0094] In this step, it is determined whether user input information has been received. The input information refers to the information entered by the user during the digital human interaction process, which can be natural language text, voice, or other formats. The specific configuration and adjustment depend on the application requirements of the digital human interaction system and are not limited here. If no user input information is received, proceed to step S30, keeping the digital human in a silent state.

[0095] After receiving the user's input information, the server executes step S32 and subsequent steps to obtain the response information of the input information, uses the response information of the input information as the output information to be broadcast, controls the digital human to switch from a silent state to a non-silent state, and broadcasts the output information.

[0096] Step S32: Obtain the response information of the input information as the output information.

[0097] After receiving the user's input information, the server obtains the response information by calling the response interface. This response information is the output information that the digital human will broadcast.

[0098] The response interface can be a system-provided interface for generating response information to the input information, a call interface for the human-computer interaction model, or other program interfaces. The human-computer interaction model refers to an artificial intelligence model used to implement human-computer interaction, such as a Large Language Model (LLM).

[0099] For example, the user's input can be a question, such as "What is the capital of country XX?". The server generates a response to this question by calling the LLM (Local Management Module), and uses this response as the output to be broadcast by the digital human.

[0100] Step S33: Determine the currently playing silent frame as the current frame; and select the action frame corresponding to the output information from the original video sequence of the digital human as the target frame.

[0101] After obtaining the output information, when controlling the digital human in a silent state to broadcast the output information, in this step, the currently playing silent frame is taken as the current frame, and the target frame to be switched to is determined according to the output information.

[0102] When determining the target frame to switch to based on the output information, the action frame corresponding to the output information is selected from the original video sequence of the digital human as the target frame.

[0103] For example, from the action frames contained in each action frame sequence in the action frame sequence pool, the action frame corresponding to the output information is selected as the target frame.

[0104] When selecting the action frame corresponding to the output information, emotion recognition is performed on the output information to determine the corresponding emotion category. It should be noted that different content segments in the output information may correspond to different emotion categories. If the output information corresponds to one emotion category, then the action frame corresponding to that emotion category is used as the action frame for the output information. If the output information corresponds to multiple emotion categories, that is, the output information is divided into multiple content segments, and different content segments correspond to different emotion categories, then the emotion category corresponding to the first content segment of the output information is obtained, and the action frame corresponding to the emotion category of the first content segment is used as the action frame for the output information.

[0105] The emotion recognition of the output information can be achieved using any emotion classification model for text or speech, such as LLM for emotion classification or other emotion classification models, without specific limitations here.

[0106] In this embodiment, the server can pre-bind corresponding action frames for each emotion category that the digital human can express. In these action frames, the digital human effectively presents the emotion of the corresponding emotion category. The server stores the action frames corresponding to each emotion category.

[0107] After obtaining the current frame and the target frame to be switched to, the server selects at least one switching frame corresponding to the target frame from the original video sequence of the digital human by executing steps S34-S35, based on the target frame (action frame).

[0108] Step S34: Select similar action frames from the original video sequence of the digital human as the action switching frames corresponding to the target frame.

[0109] In this embodiment, the target frame to be switched to is an action frame. In this step, a similar action frame to the target frame is selected from the original video sequence of the digital human as a switching frame between the current frame and the target frame, which is called the action switching frame corresponding to the target frame.

[0110] In this step, the server can select the action frame with the smallest pose distance to the target frame from the action frame sequence contained in the action frame sequence pool, based on the pose distance between the target frame and each video frame in the original video sequence, and use it as the action switching frame corresponding to the target frame.

[0111] Optionally, the server can also select the video frame with the smallest pose distance from the target frame to each video frame in the original video sequence, based on the pose distance between the target frame and each video frame in the original video sequence, and use this selected frame as the action switching frame corresponding to the target frame. Since the target frame is an action frame, the video frame with the smallest pose distance to the target frame is usually also an action frame and is included in the action frame sequence pool.

[0112] Step S35: Obtain the silent switching frame corresponding to the action switching frame in the original video sequence.

[0113] In this embodiment, the silent switching frame is a video frame whose pose distance from the starting silent frame in the original video sequence is less than or equal to a distance threshold.

[0114] After acquiring the action switching frame, this step involves acquiring the first frame in the action frame sequence containing the action switching frame, thus obtaining the silent switching frame corresponding to the action switching frame. The silent switching frame and the action switching frame can smoothly transition between each other through several frames in the action frame sequence.

[0115] Optionally, the server can also find the silent transition frame with the shortest interval before the action transition frame in the original video sequence, and use it as the silent transition frame corresponding to the action transition frame. The silent transition frame and the action transition frame can be smoothly transitioned between the video frames in the original video sequence between the silent transition frame and the action transition frame.

[0116] After obtaining at least one switching frame (including an action switching frame and a silent switching frame corresponding to the action switching frame) corresponding to the target frame, through steps S36-S37, video frame interpolation technology is used to generate a state switching sequence that starts from the current frame, transitions through at least one switching frame, and ends at the target frame.

[0117] Step S36: Using video frame interpolation technology, generate a first sequence from the current frame to the silent switching frame; obtain a second sequence from the silent switching frame to the action switching frame from the original video sequence; using video frame interpolation technology, generate a third sequence from the action switching frame to the target frame; connect the first sequence, the second sequence, and the third sequence in sequence to obtain the state switching sequence.

[0118] In this step, video frame interpolation technology is used to insert at least one frame between the current frame and the silent transition frame, generating a sequence that transitions from the current frame to the silent transition frame through at least one inserted frame, denoted as the first sequence. This first sequence can achieve a smooth transition from the current frame to the silent transition frame.

[0119] From the original video sequence, the sequence of silent transition frames to action transition frames is obtained, denoted as the second sequence. This second sequence achieves a smooth transition from silent transition frames to action transition frames.

[0120] By employing video frame interpolation technology, at least one frame is inserted between the action switching frame and the target frame, generating a sequence that transitions from the action switching frame to the target frame via at least one inserted frame, denoted as the third sequence. This third sequence enables a smooth transition from the action switching frame to the target frame.

[0121] Furthermore, by connecting the first sequence, the second sequence, and the third sequence in sequence, a state switching sequence can be obtained that starts from the current frame, smoothly transitions through at least one switching frame, and ends at the target frame, thus achieving a smooth transition from the current frame to the target frame.

[0122] It should be noted that when connecting the first, second, and third sequences sequentially, the ending frame of the first sequence and the starting frame of the second sequence are the same frame (silent transition frame). One of the silent transition frames can be removed before connecting. Similarly, the ending frame of the second sequence and the starting frame of the third sequence are the same frame (action transition frame). One of the action transition frames can be removed before connecting. This ensures that the resulting state transition sequence does not contain duplicate silent transition frames or duplicate action transition frames.

[0123] Alternatively, the complete first sequence, second sequence, and third sequence can be connected sequentially, and then the consecutive silent switching frames and consecutive action switching frames in the connection result can be deduplicated, retaining only one silent switching frame and one action switching frame to obtain the final state switching sequence.

[0124] In this embodiment, for the current frame, silent transition frame, action transition frame and target frame, a smooth transition is achieved between discontinuous but similarly oriented frames (such as the current frame and silent transition frame, action transition frame and target frame) using video frame interpolation technology; and a smooth transition is achieved between continuous but dissimilarly oriented frames (silent transition frame and action transition frame) using a subsequence from the original video sequence, thereby achieving a smooth transition of the overall state switching.

[0125] Step S37, Playback state switching sequence.

[0126] After obtaining the state switching sequence, the server plays the state switching sequence to switch the digital human from a silent state to a non-silent state that broadcasts output information.

[0127] Step S38: Start playing the original video sequence from the target frame in the original video sequence and broadcast the output information.

[0128] After switching the digital human from a silent state to a non-silent state that broadcasts output information, the original video sequence is played from the target frame in the original video sequence, and the output information is broadcast, so that the digital human presents a state corresponding to the broadcast output information.

[0129] In this embodiment, when controlling the digital human to switch from a silent state to a non-silent state, the current frame being played and the target frame to be switched to are determined. The target frame is an action frame. The action switching frame corresponding to the target frame and the silent switching frame corresponding to the action switching frame are selected from the original video sequence of the digital human. Video frame interpolation technology is used to generate a state switching sequence starting from the current frame, transitioning through the silent switching frame and the action switching frame, and ending at the target frame. By adding the silent switching frame and the action switching frame as intermediate transition frames between the current frame and the target frame, and by using video frame interpolation technology and the video sequence from the silent switching frame to the action switching frame in the original video sequence, a smooth transition between non-continuous frames is achieved, thereby realizing a smooth switch of the digital human from a silent state to a non-silent state, making the state switch of the digital human more natural and smoother.

[0130] Figure 4 is a flowchart of a digital human control method provided in another exemplary embodiment of this disclosure. Based on any of the foregoing embodiments, this embodiment provides a detailed implementation of controlling a digital human to switch from a non-silent state to a silent state. As shown in Figure 4, the process of a digital human switching from a non-silent state to a silent state includes the following steps:

[0131] Step S40: Control the digital human to broadcast output information.

[0132] After the real-time interactive digital human is activated, the server keeps the digital human in a silent state by playing a sequence of silent frames in a loop. Upon receiving user input, the server obtains the response information as output by calling a reply interface. At this point, the server needs to control the digital human to switch from a silent state to a non-silent state and control the digital human to broadcast the output information.

[0133] Step S41: Determine whether the output information has finished playing.

[0134] In this step, it is determined whether the output information has finished playing. When the output information has finished playing, the server needs to control the digital human to switch from a non-silent state to a silent state, so that the digital human can enter a silent state after the playback ends.

[0135] If the output information has not finished being broadcast according to the judgment result, step S40 is executed, and the server controls the digital human to continue broadcasting the remaining output information.

[0136] If the output information broadcast ends, the server executes step S42 and subsequent steps to continue broadcasting the remaining output information.

[0137] When the digital human finishes broadcasting, step S42 is executed to control the digital human to switch from a non-silent state to a silent state, so that the digital human enters a silent state after broadcasting the output information.

[0138] Step S42: Determine the currently playing action frame as the current frame, and take the starting silent frame of the original video sequence of the digital human as the target frame.

[0139] After the digital human finishes broadcasting, the currently playing action frame is taken as the current frame, and the starting silent frame of the original video sequence of the digital human is taken as the target frame to switch to.

[0140] After obtaining the current frame and the target frame to be switched to, the server selects at least one switching frame corresponding to the current frame from the original video sequence of the digital human according to the current frame (action frame) through steps S43-S47. Specifically, this includes the first action switching frame and the second action switching frame corresponding to the current frame, as well as the silent switching frame corresponding to the second action switching frame.

[0141] Step S43: Obtain at least one frame after the current frame in the original video sequence as the first candidate frame.

[0142] In this step, firstly, the n video frames following the current frame are obtained from the original video sequence of the digital human, and are designated as the n first candidate frames, denoted as T1, T2, ..., Tn. Here, n is a positive integer, which can be configured and adjusted according to actual application requirements.

[0143] Step S44: From the video frames contained in the action frame sequence pool, obtain the video frame with the smallest pose distance to each first candidate frame, and use it as the second candidate frame corresponding to the first candidate frame. The first candidate frame and the corresponding second candidate frame constitute a candidate frame pair.

[0144] In this step, from the action frames contained in each action frame sequence in the action frame sequence pool, the action frame with the closest pose distance to each first candidate frame is found, resulting in n second candidate frames corresponding to the n first candidate frames, denoted as U1, U2, ..., Un. Each first candidate frame and its corresponding second candidate frame form a candidate frame pair, denoted as (T1, U1), (T2, U2), ..., (Tn, Un).

[0145] Step S45: Determine the candidate frame pair with the smallest attitude distance based on the attitude distance between the candidate frame pairs.

[0146] In this step, the attitude distance between two candidate frames in n candidate frame pairs (T1, U1), (T2, U2), ..., (Tn, Un) is determined, and the candidate frame pair with the smallest attitude distance between the two candidate frames is selected, denoted as (Tk, Uk).

[0147] Step S46: Select the first and second candidate frames from the candidate frame pair with the smallest attitude distance as the first action switching frame and the second action switching frame corresponding to the current frame, respectively.

[0148] In this step, the first candidate frame Tk in the candidate frame pair (Tk, Uk) with the smallest pose distance between the two candidate frames is taken as the first action switching frame, and the second candidate frame Uk is taken as the second action switching frame.

[0149] The first motion transition frame is a video frame in the original video sequence that follows the current frame. The second motion transition frame is the motion frame in the original video sequence that has the smallest pose distance from the first motion transition frame.

[0150] Step S47: Obtain the first frame in the action frame sequence where the second action switching frame is located, and obtain the silent switching frame corresponding to the second action switching frame.

[0151] In this step, the silent transition frame corresponding to the second action transition frame is obtained. Specifically, the first frame in the action frame sequence where the second action transition frame is located is obtained to obtain the silent transition frame corresponding to the second action transition frame. The silent transition frame and the second action transition frame can be smoothly transitioned through several frames in the action frame sequence.

[0152] Optionally, the server can also find the silent transition frame with the shortest interval between it and the second action transition frame in the original video sequence before the second action transition frame, and use it as the silent transition frame corresponding to the second action transition frame. The silent transition frame and the second action transition frame can be smoothly transitioned between the video frames in the original video sequence between the silent transition frame and the second action transition frame.

[0153] After obtaining at least one switching frame corresponding to the current frame (including the first action switching frame and the second action switching frame corresponding to the current frame, and the silent switching frame corresponding to the second action switching frame), through steps S47-S411, video frame interpolation technology is used to generate a state switching sequence that starts from the current frame, transitions through at least one switching frame, and ends at the target frame.

[0154] Step S48: Obtain the fourth sequence from the current frame to the first action switching frame from the original video sequence; use video frame interpolation technology to generate the fifth sequence from the first action switching frame to the second action switching frame; obtain the sixth sequence from the silence switching frame to the second action switching frame from the action frame sequence pool, and generate the reverse sequence of the sixth sequence; use video frame interpolation technology to generate the seventh sequence from the silence switching frame to the starting silence frame; connect the fourth sequence, the fifth sequence, the reverse sequence of the sixth sequence and the seventh sequence to obtain the state switching sequence.

[0155] In this step, the sequence from the current frame to the first action switching frame is obtained from the original video sequence, denoted as the fourth sequence. This fourth sequence enables a smooth transition from the current frame to the first action switching frame.

[0156] By employing video frame interpolation technology, at least one frame is inserted between the first action switching frame and the second action switching frame to generate a sequence that transitions from the first action switching frame to the second action switching frame via at least one inserted frame, denoted as the fifth sequence. This fifth sequence enables a smooth transition from the first action switching frame to the second action switching frame.

[0157] From the action frame sequence pool containing the second action switching frame, extract the sequence from the first frame (i.e., the silent switching frame corresponding to the second action switching frame) to the second action switching frame; this is denoted as the sixth sequence. This sixth sequence enables a smooth transition from the silent switching frame to the second action switching frame. Furthermore, generate the inverse sequence of the sixth sequence to achieve a smooth transition from the second action switching frame to the silent switching frame.

[0158] In this context, the reverse sequence of a frame sequence refers to rearranging the frames in the original frame sequence in the reverse order. For example, suppose the sixth sequence includes three action frames, denoted as (F1, F2, F3), then the reverse sequence of the sixth sequence is (F3, F2, F1).

[0159] By employing video frame interpolation technology, at least one frame is inserted between the silent transition frame and the initial silent frame, generating a sequence that transitions from the silent transition frame to the initial silent frame via at least one inserted frame, denoted as the seventh sequence. This seventh sequence enables a smooth transition from the silent transition frame to the initial silent frame.

[0160] Furthermore, by sequentially connecting the inverse sequences of the fourth, fifth, and sixth sequences and the seventh sequence, a state switching sequence can be obtained that starts from the current frame, smoothly transitions through at least one switching frame, and ends at the target frame, thus achieving a smooth transition from the current frame to the target frame.

[0161] It should be noted that when connecting the fourth, fifth, and sixth sequences in reverse order and the seventh sequence, the end frame of the previous sequence and the start frame of the next sequence are the same frame. The end frame of the previous sequence can be removed before connecting it to the next sequence, or the previous sequence can be connected to the next sequence after removing the start frame. In this way, the connected state switching sequence does not contain repeated silent switching frames and repeated action switching frames.

[0162] For example, the end frame of the fourth sequence and the start frame of the fifth sequence are the same frame (the first action switching frame), and one of the first action switching frames can be removed before connecting them.

[0163] Alternatively, the inverse sequences of the complete fourth, fifth, and sixth sequences and the seventh sequence can be concatenated sequentially, and then the consecutive silent switching frames, consecutive first action switching frames, and consecutive second action switching frames in the concatenation result can be deduplicated, retaining only one silent switching frame, one first action switching frame, and one second action switching frame to obtain the final state switching sequence.

[0164] In this embodiment, for the current frame, the first action switching frame, the second action switching frame, the silent switching frame, and the target frame (the initial silent frame), a smooth transition is achieved between discontinuous but similarly oriented frames (such as the first action switching frame and the second action switching frame, and the silent switching frame and the target frame) using video frame interpolation technology; and a smooth transition is achieved between continuous but dissimilarly oriented frames (such as the current frame and the first action switching frame, and the second action switching frame and the silent switching frame) using a subsequence or the reverse sequence of the original video sequence, thereby achieving a smooth transition of the overall state switching.

[0165] Step S49, Playback state switching sequence.

[0166] After obtaining the state switching sequence, the server plays the state switching sequence to switch the digital human from a non-silent state that broadcasts output information to a silent state.

[0167] Step S410: Play the silent frame sequence from the original video sequence.

[0168] The silent frame sequence contains at least one previous frame from the original video sequence, and all of the previous frames are silent frames.

[0169] After switching the digital human from a non-silent state to a silent state where it is broadcasting information, the silent frame sequence is played in a forward and reverse loop to keep the digital human in a silent state.

[0170] In this embodiment, when controlling the digital human to switch from a non-silent state to a silent state, the currently playing action frame is determined as the current frame, and the starting silent frame of the original video sequence of the digital human is taken as the target frame. The first action switching frame, the second action switching frame, and the silent switching frame corresponding to the second action switching frame are selected from the original video sequence of the digital human. Using video frame interpolation technology, a state switching sequence is generated starting from the current frame, transitioning through the first action switching frame, the second action switching frame, and the silent switching frame corresponding to the second action switching frame, and ending at the target frame. By adding the first action switching frame, the second action switching frame, and the silent switching frame corresponding to the second action switching frame as intermediate transition frames between the current frame and the target frame, and by utilizing video frame interpolation technology and the reverse sequence of the video sequence from the current frame to the first action switching frame and the video sequence from the silent switching frame to the second action switching frame in the original video sequence, a smooth transition between discontinuous frames is achieved. This results in a smooth transition of the digital human from a non-silent state to a silent state, making the state transition of the digital human more natural and fluid.

[0171] Figure 5 is a flowchart of real-time interaction of a digital human provided in another exemplary embodiment of this disclosure. As shown in Figure 5, the overall flow of real-time interaction of the digital human is as follows:

[0172] After the real-time interactive digital human is activated, it enters a silent state. The server maintains this silent state by playing a sequence of silent frames in a loop. When a user inputs information via a device (such as a mobile terminal), the device sends the input to the server. Upon receiving the input, the server retrieves the response information as output by calling a response interface. The server then switches the digital human from a silent to a non-silent state and plays the output information. After the output information playback ends, the server switches the digital human back to a silent state, returning it to silence after the playback.

[0173] Furthermore, during the process of the digital human broadcasting output information, the corresponding emotion category can be identified. Different content segments within the output information may correspond to different emotion categories. For example, the emotion categories that the digital human can express include, but are not limited to: calmness, confusion, anger, joy, sadness, excitement, and surprise. While broadcasting content segments of different emotion categories, the digital human is in a non-silent state, but needs to make gestures that reflect different emotions, requiring the digital human to switch from one non-silent state to another. Through the scheme disclosed herein, the digital human can be switched from one non-silent state to a silent state first, and then from a silent state to another non-silent state, thus achieving the switching between different non-silent states.

[0174] Figure 6 is a schematic diagram of a server structure provided in an embodiment of this disclosure. As shown in Figure 6, the server includes a memory 601 and a processor 602. The memory 601 is used to store computer execution instructions and can be configured to store various other data to support operations on the server. The processor 602 is communicatively connected to the memory 601 and is used to execute the computer execution instructions stored in the memory 601 to implement the technical solutions provided in any of the above method embodiments. Their specific functions and the technical effects they can achieve are similar and will not be repeated here.

[0175] Optionally, as shown in Figure 6, the server may also include other components such as a firewall 603, a load balancer 604, a communication component 605, and a power supply component 606. Figure 6 only schematically illustrates some components and does not imply that the server only includes the components shown in Figure 6. Figure 6 uses a cloud server deployed in the cloud as an example for illustrative purposes; the server can also be deployed locally, and this embodiment does not impose specific limitations.

[0176] This disclosure also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the method of any of the foregoing embodiments. The specific functions and technical effects to be achieved are not described here.

[0177] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the methods of any of the foregoing embodiments. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. The execution of the computer program by the at least one processor causes the server to perform the technical solutions provided in any of the above method embodiments; the specific functions and achievable technical effects are not elaborated here.

[0178] This disclosure provides a chip, including a processing module and a communication interface. The processing module is capable of executing the technical solutions of the server in the aforementioned method embodiments. Optionally, the chip further includes a storage module (e.g., a memory), which stores instructions. The processing module executes the instructions stored in the storage module, and the execution of the instructions stored in the storage module causes the processing module to execute the technical solutions provided in any of the aforementioned method embodiments.

[0179] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this disclosure.

[0180] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), a graphics processing unit (GPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules in at least one processor.

[0181] The memory may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0182] The aforementioned storage device can be object storage service (OSS).

[0183] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0184] The aforementioned communication components are configured to facilitate wired or wireless communication between the device containing the communication components and other devices. The device containing the communication components can access wireless networks based on communication standards, such as mobile hotspots (WiFi), second-generation (2G), third-generation (3G), fourth-generation (4G) / Long Term Evolution (LTE), fifth-generation (5G), or combinations thereof. In one exemplary embodiment, the communication components receive broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication components also include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), infrared, Ultra Wide Band (UWB), Bluetooth, and other technologies.

[0185] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0186] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0187] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside within an application-specific integrated circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components within an electronic device or host device.

[0188] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0189] The order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order; however, it should be clearly understood that these operations may not be executed in the order they appear herein or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. "Multiple" means two or more, unless otherwise explicitly specified.

[0190] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solutions of this disclosure, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this disclosure.

[0191] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0192] The above are merely preferred embodiments of this disclosure and do not limit the patent scope of this disclosure. Any equivalent structural or procedural transformations made using the content of this disclosure and its drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this disclosure.

Claims

1. A method for controlling a digital human, wherein, include: When controlling the switching state of the digital human, the current frame being played and the target frame to be switched to are determined, wherein one of the current frame and the target frame is a silent frame and the other is an action frame; Based on the action frames in the current frame and the target frame, at least one switching frame corresponding to the action frame is selected from the original video sequence of the digital human; Using video frame interpolation technology, a state switching sequence is generated starting from the current frame, passing through at least one switching frame, and ending at the target frame; Play the state switching sequence.

2. The method according to claim 1, wherein, The step of determining the currently playing frame and the target frame to be switched to when controlling the digital human's switching state includes: When controlling the digital human in a silent state to broadcast output information, determine the currently playing silent frame as the current frame; Select the action frame corresponding to the output information from the original video sequence of the digital human as the target frame.

3. The method according to claim 2, wherein, The step of selecting at least one switching frame corresponding to the action frame from the original video sequence of the digital human based on the action frame in the current frame and the target frame includes: Select similar action frames from the original video sequence of the digital human as the action switching frames corresponding to the target frame; Obtain the silent switching frame corresponding to the action switching frame in the original video sequence, wherein the silent switching frame is a video frame whose pose distance from the starting silent frame in the original video sequence is less than or equal to a distance threshold.

4. The method according to claim 3, wherein, The method of using video frame interpolation to generate a state transition sequence starting from the current frame, passing through at least one switching frame, and ending at the target frame includes: Using video frame interpolation technology, a first sequence from the current frame to the silent switching frame is generated; Obtain a second sequence from the original video sequence, from the silent switching frame to the action switching frame; A third sequence from the action switching frame to the target frame is generated using video frame interpolation technology; The first sequence, the second sequence, and the third sequence are connected sequentially to obtain the state switching sequence.

5. The method according to claim 3, wherein, The step of selecting similar motion frames from the original video sequence of the digital human as motion switching frames corresponding to the target frame includes: Based on the pose distance between the target frame and each video frame in the original video sequence, the action frame with the smallest pose distance to the target frame is selected from the action frames contained in the action frame sequence pool and used as the action switching frame corresponding to the target frame. The action frame sequence pool includes multiple action frame sequences, and the starting frame of the action frame sequence is a silent switching frame.

6. The method according to claim 5, wherein, The step of obtaining the silent switching frame corresponding to the action switching frame in the original video sequence includes: Obtain the first frame in the action frame sequence where the action switching frame is located, and obtain the silent switching frame corresponding to the action switching frame.

7. The method according to claim 2, wherein, After playing the state switching sequence, the method further includes: Play the original video sequence starting from the target frame in the original video sequence, and broadcast the output information.

8. The method according to any one of claims 1-7, wherein, The step of determining the currently playing frame and the target frame to be switched to when controlling the digital human's switching state includes: When the digital human finishes broadcasting, and the digital human is switched to a silent state, the currently playing action frame is determined as the current frame, and the starting silent frame of the original video sequence of the digital human is taken as the target frame.

9. The method according to claim 8, wherein, The step of selecting at least one switching frame corresponding to the action frame from the original video sequence of the digital human based on the action frame in the current frame and the target frame includes: Obtain at least one frame following the current frame in the original video sequence as the first candidate frame; From the video frames contained in the action frame sequence pool, the video frame with the smallest pose distance to each of the first candidate frames is selected as the second candidate frame corresponding to the first candidate frame. The first candidate frame and the corresponding second candidate frame constitute a candidate frame pair. Based on the attitude distance between the candidate frame pairs, determine the candidate frame pair with the smallest attitude distance; The first and second candidate frames in the candidate frame pair with the smallest attitude distance are respectively used as the first action switching frame and the second action switching frame corresponding to the current frame.

10. The method according to claim 9, wherein, The step of selecting at least one switching frame corresponding to the action frame from the original video sequence of the digital human based on the action frame in the current frame and the target frame further includes: Obtain the first frame in the action frame sequence where the second action switching frame is located, and obtain the silent switching frame corresponding to the second action switching frame.

11. The method according to claim 10, wherein, The method of using video frame interpolation to generate a state transition sequence starting from the current frame, passing through at least one switching frame, and ending at the target frame includes: Obtain a fourth sequence from the current frame to the first action switching frame from the original video sequence; A fifth sequence is generated, from the first action switching frame to the second action switching frame, using video frame interpolation technology. Obtain the sixth sequence from the silent switching frame to the second action switching frame from the action frame sequence pool, and generate the reverse sequence of the sixth sequence; A seventh sequence from the silence switching frame to the initial silence frame is generated using video frame interpolation technology. The state switching sequence is obtained by connecting the fourth sequence, the fifth sequence, the reverse sequence of the sixth sequence, and the seventh sequence.

12. The method according to claim 8, wherein, After playing the state switching sequence, the method further includes: Play the silent frame sequence in the original video sequence, the silent frame sequence containing at least one previous frame in the original video sequence, the at least one previous frame being a silent frame.

13. The method according to any one of claims 1-12, wherein, Also includes: Obtain the original video sequence of the digital human, wherein the initial state of the digital human in the original video sequence is a silent state, and take the first frame of the original video sequence as the starting silent frame; Obtain the pose information of the digital human in each video frame of the original video sequence; Calculate the distance between the pose information of the digital human in any two video frames to obtain the pose distance between any two video frames; Based on the pose distance between any two video frames, video frames whose pose distance to the starting silent frame is less than or equal to a distance threshold are selected from the original video sequence to obtain silent switching frames.

14. The method according to claim 13, wherein, Also includes: If there are consecutive silent switching frames in the obtained silent switching frames, then one of the consecutive silent switching frames is taken as the retained frame, and the silent switching frames other than the retained frame in the consecutive silent switching frames are deleted. In the original video sequence, the video frame sequence from any of the silent switching frames to the frame with the largest attitude distance from the silent switching frame is obtained as the action frame sequence corresponding to the silent switching frame, and an action frame sequence pool is constructed.

15. The method according to claim 14, wherein, After constructing the action frame sequence pool, the following is also included: Based on at least one emotion category that the digital human can express, corresponding action frames are bound to each of the at least one emotion category.

16. The method according to claim 13, wherein, The step of obtaining the pose information of the digital human in each video frame of the original video sequence includes: A human pose estimation algorithm is used to detect human pose in each video frame of the original video sequence, thereby obtaining the pose information of the digital human in each video frame.

17. The method according to claim 13, wherein, The step of calculating the distance between the pose information of the digital human in any two video frames to obtain the pose distance between any two video frames includes: The pose information of the digital human in each video frame is stored as a vector to obtain the pose vector of each video frame; Calculate the distance between the pose vectors of any two video frames to obtain the pose distance between the two video frames.

18. A server, wherein, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the server to perform the method according to any one of claims 1-17.

19. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-17.

20. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-17.

Images