Training sample generation method and apparatus, storage medium, and electronic device

Abstract

The present disclosure provides a training sample generation method and apparatus, a storage medium, and an electronic device. The method comprises: acquiring a motion description information set, wherein the motion description information set comprises a plurality of pieces of motion description information, and object motions respectively indicated by at least two of the plurality of pieces of motion description information are different object motions; when performing a clustering operation on the basis of the motion description information set to determine at least one motion description information subset, acquiring first motion description information and second motion description information from the at least one motion description information subset; on the basis of the first motion description information and the second motion description information, performing a motion capture operation to obtain multi-frame transition sample motion information; and determining a target training sample according to the first motion description information, the second motion description information, and the multi-frame transition sample motion information. The present disclosure resolves the technical problem of the low quality of training samples obtained by related methods.

Description

Methods and apparatus for generating training samples, storage media and electronic devices

[0001] This application claims priority to Chinese Patent Application No. 2025104881957, filed on April 17, 2025, entitled “Method and apparatus for generating training samples, storage medium and electronic device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of computer science, and more specifically to the generation of training samples. Background Technology

[0003] When designing and rendering the motion of virtual objects, the relevant technologies first capture motion data in keyframes. Then, using motion prediction models, the transition motion between keyframes is inferred based on these keyframe motions.

[0004] Before applying action prediction models to predict transitional actions, it is usually necessary to train the models with training samples to optimize their performance. To enhance the training effect of action prediction models, action samples are usually randomly selected from regular action datasets as training samples. However, since these randomly selected action samples differ significantly from the transitional actions in real-world application scenarios, existing training samples suffer from low sample quality.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a method and apparatus for generating training samples, a storage medium, and an electronic device to at least solve the technical problem of low training sample quality in related fields.

[0007] According to one aspect of the embodiments of this application, a method for generating training samples is provided, including...

[0008] A set of action description information is obtained, wherein the set includes multiple action description information, and at least two of the multiple action description information indicate different object actions; when at least one subset of action description information is determined by performing a clustering operation based on the set of action description information, a first action description information and a second action description information are obtained from the at least one subset of action description information, wherein the first action description information is used to indicate a first object action, the second action description information is used to indicate a second object action, and the first object action and the second object action are different object actions; based on the first action description information and the second action description information, an action acquisition operation is performed to obtain multiple frames of transition sample action information, wherein the multiple frames of transition sample action information are used to indicate multiple frames of transition object actions that match the first object action and the second object action; a target training sample is determined based on the first action description information, the second action description information, and the multiple frames of transition sample action information, wherein the target training sample is used to train the transition action prediction model.

[0009] According to another aspect of the embodiments of this application, an apparatus for generating training samples is also provided, comprising:

[0010] A first acquisition unit is configured to acquire an action description information set, wherein the action description information set includes multiple action description information, and at least two of the multiple action description information indicate different object actions; a second acquisition unit is configured to acquire first action description information and second action description information from at least one subset of action description information after performing a clustering operation based on the action description information set to determine at least one subset of action description information, wherein the first action description information indicates a first object action, the second action description information indicates a second object action, and the first object action and the second object action are different object actions; a collection unit is configured to perform an action collection operation based on the first action description information and the second action description information to obtain multiple frames of transition sample action information, wherein the multiple frames of transition sample action information indicate multiple frames of transition object actions that match the first object action and the second object action; a determination unit is configured to determine a target training sample based on the first action description information, the second action description information, and the multiple frames of transition sample action information, wherein the target training sample is used to train a transition action prediction model.

[0011] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described method for generating training samples when running.

[0012] According to another aspect of the embodiments of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the training sample generation method as described above.

[0013] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the training sample generation method described above through the computer program.

[0014] In the above method for generating training samples, firstly, an action description information set containing multiple action description information is obtained, wherein at least two of the multiple action description information indicate different object actions; nextly, by performing a clustering operation, at least one subset of action description information is determined from the action description information set, and after obtaining at least one subset of action description information, first action description information and second action description information are extracted from these subsets; furthermore, an action acquisition operation is performed based on the first action description information and the second action description information to obtain multi-frame transition sample action information; finally, the target training sample is determined based on the first action description information, the second action description information, and the multi-frame transition sample action information.

[0015] Through the above-described embodiments of this application, the combination of clustering and action acquisition operations not only improves the generalization effect and quality of the samples but also significantly reduces the number of action acquisition operations. On one hand, by performing clustering operations on the action description information set, similar action description information is grouped into one category, facilitating the subsequent selection of representative action description information. This avoids repeated acquisition of similar actions, significantly improving the generalization effect of the samples. On the other hand, since the action acquisition operation is based only on the representative first and second action description information obtained from the clustering operation, without needing to acquire actions from every action description information in the subset of action description information, the generation efficiency of training samples is further improved while ensuring the quality of training samples, thereby solving the technical problem of low training sample quality in the prior art. Attached Figure Description

[0016] Figure 1 is a schematic diagram of the hardware environment of an optional training sample generation method according to an embodiment of this application;

[0017] Figure 2 is a flowchart of an optional method for generating training samples according to an embodiment of this application;

[0018] Figure 3 is a schematic diagram of an optional method for predicting transition actions according to an embodiment of this application;

[0019] Figure 4 is a schematic diagram of another optional method for predicting transition actions according to an embodiment of this application;

[0020] Figure 5 is a schematic diagram of an optional object node according to an embodiment of this application;

[0021] Figure 6 is a schematic diagram of an optional clustering method according to an embodiment of this application;

[0022] Figure 7 is a schematic diagram of another optional clustering method according to an embodiment of this application;

[0023] Figure 8 is a flowchart of another optional method for generating training samples according to an embodiment of this application;

[0024] Figure 9 is a flowchart of another optional method for generating training samples according to an embodiment of this application;

[0025] Figure 10 is a schematic diagram of another optional object node according to an embodiment of this application;

[0026] Figure 11 is a schematic diagram of another optional clustering method according to an embodiment of this application;

[0027] Figure 12 is a flowchart of an optional motion capture method according to an embodiment of this application;

[0028] Figure 13 is a schematic diagram of an optional motion capture method according to an embodiment of this application;

[0029] Figure 14 is a schematic diagram of another optional motion capture method according to an embodiment of this application;

[0030] Figure 15 is a scene diagram of an optional motion capture method according to an embodiment of this application;

[0031] Figure 16 is a schematic diagram of an optional data augmentation method according to an embodiment of this application;

[0032] Figure 17 is a schematic diagram of another optional data augmentation method according to an embodiment of this application;

[0033] Figure 18 is a flowchart of another optional method for generating training samples according to an embodiment of this application;

[0034] Figure 19 is a schematic diagram of an optional action prediction model training method according to an embodiment of this application;

[0035] Figure 20 is a schematic diagram of an optional training sample generation device according to an embodiment of this application;

[0036] Figure 21 is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] It should be noted that in the relevant embodiments of this application, the acquisition and processing of relevant data information of the target account are all obtained in advance through methods that comply with the relevant normative legal documents, and the authorization permission of the corresponding account subject must be obtained before acquiring the above-mentioned relevant information and data.

[0040] The following explains the technical terms used in this application:

[0041] BVH format files: BVH files contain the character's skeleton and limb joint rotation data. It is a universal human feature animation file format that is widely supported by various popular animation software today.

[0042] FBX (Filmbox) format files: FBX files are a standard file format for 3D data exchange. They are widely used in game development, film and television production, and other 3D design fields, supporting the integration and transmission of various data types, including models, textures, and animations.

[0043] According to one aspect of the embodiments of this application, a method for generating training samples is provided. This method can be applied, but is not limited to, a training sample generation system in the hardware environment shown in FIG. 1. The training sample generation system may include, but is not limited to, a terminal device 102, a network 104, a server 106, and a database 108. The terminal device 102 runs an information processing client for processing a set of action description information. The terminal device 102 includes a human-computer interaction screen, a processor, and a memory. The human-computer interaction screen in the terminal device 102 can be used to provide a human-computer interaction interface to receive human-computer interaction operations for controlling the set of action description information. The processor is used to generate interaction instructions in response to the human-computer interaction operations and send the interaction instructions to the server. The memory is used to store relevant data, such as target training samples.

[0044] Furthermore, server 106 and database 108 can form a distributed network. Server 106 includes a processing engine used to perform store or read operations on database 108. Specifically, the processing engine reads a set of action description information from database 108.

[0045] Assuming that the terminal device 102 in Figure 1 is running an information processing client that processes the action description information set, the specific process of this embodiment is as follows: The terminal device 102 executes steps S102 to S108, i.e., obtains the action description information set, wherein the action description information set includes multiple action description information, and at least two of the multiple action description information indicate different object actions; When at least one subset of action description information is determined by performing a clustering operation based on the action description information set, a first action description information and a second action description information are obtained from the at least one subset of action description information, wherein the first action description information is used to indicate a first object action, and the second action description information is used to indicate a second object action, and the first object action and the second object action are different object actions; Based on the first action description information and the second action description information, an action acquisition operation is performed to obtain multi-frame transition sample action information, wherein the multi-frame transition sample action information is used to indicate multi-frame transition object actions that match the first object action and the second object action; A target training sample is determined based on the first action description information, the second action description information, and the multi-frame transition sample action information, wherein the target training sample is used to train the transition action prediction model.

[0046] Next, terminal device 102 executes S110, which sends the target training samples to server 106 via network 104. Finally, server 106 executes S112, which determines the target training sample set based on the target training samples.

[0047] Optionally, in this embodiment, the terminal device 102 may be a terminal device configured with a target client, which may include, but is not limited to, at least one of the following: mobile phone (such as Android phone, iOS phone, etc.), laptop computer, tablet computer, PDA, MID (Mobile Internet Device), PAD (portable android device), desktop computer, smart TV, etc. The target client may be a data operation client that provides operations on a set of action description information. The network may include, but is not limited to, wired network and wireless network, wherein the wired network includes: local area network, metropolitan area network and wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that enable wireless communication. The server may be a single server, a server cluster composed of multiple servers, or a cloud server. The above is only an example, and no limitation is made in this embodiment.

[0048] Optionally, in this embodiment, the action description information involved in the above-described training sample generation method can be, but is not limited to, applied to rendering based on object actions of a target virtual object in a virtual scene. The target virtual object can be a virtual object involved in a game-type terminal application (APP) used to complete a predetermined interactive game task.

[0049] The training samples obtained by the above-described method can be used to train a transitional action generation model. This transitional action generation model can generate transitional action information indicating a transitional object action based on first action description information indicating a first object action and second action description information indicating a second object action, wherein the transitional object action is at least one object action between the first object action and the second object action.

[0050] The aforementioned first action description information, second action description information, and transition action information can be applied to render the object actions of virtual objects in Multiplayer Online Battle Arena (MOBA) applications. In these MOBAs, players can control virtual objects to release character skills within a virtual scene, thereby creating virtual objects within that scene. The aforementioned interactive game tasks can be, but are not limited to, game tasks completed by the current player through virtual social interaction between virtual objects in the generated virtual scene and virtual objects controlled by other players, generated from human-computer interaction training samples. The method for generating these training samples can also be applied to massively multiplayer online role-playing games (MMORPGs) on terminal applications. In these games, the current player can complete interactive game tasks from the first-person perspective of a virtual object through role-playing, such as completing game tasks together with other virtual objects. These interactive game tasks can, but are not limited to, run as plugins or mini-programs within an application (such as a non-standalone game app), or run within a game engine within an application (such as a standalone game app). The types of game applications mentioned above may include, but are not limited to, at least one of the following: two-dimensional (2D) game applications, three-dimensional (3D) game applications, virtual reality (VR) game applications, augmented reality (AR) game applications, and mixed reality (MR) game applications. This is merely an example, and this embodiment does not impose any limitations on it.

[0051] In the above method for generating training samples, firstly, an action description information set containing multiple action description information is obtained, wherein at least two of the multiple action description information indicate different object actions; nextly, by performing a clustering operation, at least one subset of action description information is determined from the action description information set, and after obtaining at least one subset of action description information, first action description information and second action description information are extracted from these subsets; furthermore, an action acquisition operation is performed based on the first action description information and the second action description information to obtain multi-frame transition sample action information; finally, the target training sample is determined based on the first action description information, the second action description information, and the multi-frame transition sample action information.

[0052] Through the above-described embodiments of this application, the combination of clustering and action acquisition operations not only improves the generalization effect and quality of the samples but also significantly reduces the number of action acquisition operations. On one hand, by performing clustering operations on the action description information set, similar action description information is grouped into one category, facilitating the subsequent selection of representative action description information. This avoids repeated acquisition of similar actions, significantly improving the generalization effect of the samples. On the other hand, since the action acquisition operation is based only on the representative first and second action description information obtained from the clustering operation, without needing to acquire actions for every action description information in the subset of action description information, the generation efficiency of training samples is further improved while ensuring the quality of training samples, thereby solving the technical problem of low training sample quality in the prior art.

[0053] As an optional implementation, as shown in Figure 2, the above-described method for generating training samples can be applied to a server or terminal, and may specifically include the following steps:

[0054] S202, Obtain an action description information set, wherein the action description information set includes multiple action description information, and at least two of the multiple action description information indicate different object actions;

[0055] S204, when at least one subset of action description information is determined by performing a clustering operation based on the action description information set, first action description information and second action description information are obtained from the at least one subset of action description information.

[0056] S206, based on the first action description information and the second action description information, perform an action acquisition operation to obtain multi-frame transition sample action information, wherein the multi-frame transition sample action information is used to indicate multi-frame transition object actions that match the first object action and the second object action.

[0057] S208, determine the target training sample based on the first action description information, the second action description information and the action information of the multi-frame transition sample, wherein the target training sample is used to train the transition action prediction model.

[0058] First, the application scenarios of the target training samples obtained in S202 to S208 of the above embodiments will be explained. In an optional manner, the target training samples obtained in S202 to S208 can be used to train a transition motion prediction model. The transition motion prediction model can be used to predict the transition object motion between any two frame frames, such as keyframes. The following explanation will use keyframes as an example. It should be noted that when the keyframe is used to render a virtual scene, and the virtual scene includes virtual objects performing object actions, the keyframe can include motion description information indicating the object actions. For example, the keyframe can include scene rendering data, and it can include object rendering data for rendering virtual objects. In the object rendering data, the data indicating the object actions can be determined as motion description information, which may include, but is not limited to, the position information, velocity information, or rotation information of each driving node in the object-driven model corresponding to the virtual object.

[0059] Furthermore, in the two keyframes used to predict the action of the transition object, the keyframe with the earlier timestamp can be described as the start frame, in which the virtual object performs the action of the start object in the virtual scene indicated by the start frame; the keyframe with the later timestamp can be described as the end frame, in which the virtual object performs the action of the end object in the virtual scene indicated by the end frame.

[0060] Furthermore, the transition object actions predicted by the aforementioned transition action prediction model can be applied to a game application to determine and render the transition object actions of virtual game objects in the virtual scene of the game application. Furthermore, the method for determining the transition object actions can be a real-time determination method during the operation of the game application, or it can be used during the development of the aforementioned game application to determine the transition actions in the various object display animations corresponding to different virtual game objects.

[0061] Taking the above-mentioned target training samples as an example in the training process of the transition action prediction model, and the above-mentioned transition action prediction model as an example in the case of predicting object actions in a game application, the relationship between the first object action and the second object action in step S202 will be further explained below from the perspective of the action type of the object action.

[0062] It should be noted that the first object action and the second object action mentioned above can be two actions performed by a virtual game object in the process of performing virtual object operations in a virtual game scene. For example, when a virtual game object triggers an attack, the first object action and the second object action can be two different attack actions during the attack process; when a virtual game object triggers a skill release, the first object action and the second object action can be two different skill actions during the skill release process; when a virtual game object is attacked by other virtual game objects, the first object action and the second object action can be two different hit actions executed in response to being hit by an attack from another virtual game object; when a virtual game object triggers a movement operation in the virtual game scene, the first object action and the second object action can be two movement actions during the movement process; when it is detected that a virtual game object switches from a standby state to a hit state in the virtual scene, the first object action can be a standby action corresponding to the standby state, and the second object action can be a hit action corresponding to the hit state, wherein the standby state is a state in which an object does not receive a control command and remains stationary, and the hit state is a state in which it is attacked by other virtual game objects; optionally, when a virtual game object switches back from the hit state to the standby state, the first object action can also be a hit action, and the second object action can also be a standby action. The above examples are merely illustrative. In this embodiment, the specific types of the first object action and the second object action are not limited.

[0063] In another alternative approach, when the aforementioned transition motion prediction model is applied to an animation development application, the application can be used to generate and render virtual animation objects in an animation scene. After determining the object actions corresponding to the two key animation frames of the virtual animation object, the transition object actions rendered and displayed in the transition animation frames between the key animation frames can be determined using the transition motion prediction model trained based on the target training samples generated in steps S202 to S208.

[0064] In another alternative approach, when the aforementioned transitional motion prediction model is applied to a 3D modeling scene, this 3D modeling scene can generate CG (Computer-Generated Animation) based on multiple 3D coordinate signal sequences realistically acquired using motion capture technology. Since the frame rate of the multiple 3D coordinate signal sequences acquired by motion capture technology is relatively low, the aforementioned transitional motion prediction model can predict the transitional CG actions between multiple CG actions.

[0065] The following description, in conjunction with Figure 3, illustrates the method for predicting the transitional object actions based on the transitional action prediction model in the aforementioned scenario. As shown in Figure 3, after training the transitional action prediction model 308 using the training samples obtained in steps S202 to S208, the starting object action 304 and ending object action 306 of the current virtual object can be obtained based on the interaction operations between virtual objects in the virtual scene 302. The starting object action 304 can be the object action executed by the current virtual object in a static state or a state where no operation command has been received, and the ending object action 306 can be the object action executed by the current virtual object when it is attacked by other virtual objects, indicating the attacked state.

[0066] Next, the transition action prediction model 308 can predict a series of transition actions between the two actions based on the action description information of the starting object action 304 and the ending object action 306, thereby obtaining the object action sequence 309. Through the object action sequence 309, the process of the current virtual object smoothly transitioning from the starting object action 304 to the ending object action 306 can be displayed.

[0067] Finally, in the virtual scene 310, a series of object actions performed by the current virtual object when it is attacked by other virtual objects can be displayed based on the above object action sequence 309.

[0068] It should be noted that, in the above embodiments, the transition action prediction model 308 can be a neural network model among Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and Convolutional Neural Network (CNN), or a neural network model obtained by combining multiple neural networks in a certain way. In this embodiment, the specific model type of the transition action prediction model 308 is not limited.

[0069] In an optional implementation, the transition action prediction model 308 can directly predict the object action sequence 309 based on the action description information of the ending object action 306 and the action description information of the starting object action 304.

[0070] In another optional implementation, as shown in Figure 4, when the transition object action 406 is determined based on the starting object action 402 and the ending object action 404 according to the aforementioned transition action prediction model 308, the transition object action 408 can be further determined based on the transition object action 406 and the ending object action 404. Finally, the starting object action 402, the transition object action 406, the transition object action 408, and the ending object action 404 are determined as the object action sequence 410. It can be understood that in the above implementation, the above object actions can all be represented as action description information in the above transition action prediction model, thereby realizing the prediction of each transition action in the transition action sequence according to the transition action prediction model 308.

[0071] The above implementation methods are merely illustrative examples and do not limit the specific application scenarios of the transition action prediction model determined in step S208.

[0072] Furthermore, when the above-described embodiments are applied to a game scenario, steps S202 to S208 can be applied to a game client or a game server. When applied to a game client, the above steps can be implemented by an action generation thread that runs in parallel with the game screen rendering thread in the terminal; when applied to a game server, the terminal can obtain the determination result of the transition action information set of the game server in real time via the network, and perform screen rendering in the terminal based on the obtained transition action information set.

[0073] Furthermore, the aforementioned action description information set includes multiple action description information pieces, each used to describe an object action. Moreover, at least two of the multiple action description information pieces indicate different object actions; that is, at least two action description information pieces in the action description information set are different, meaning at least two object actions are different. It should be noted that the aforementioned action description information set can be a pre-acquired sample dataset, including a limited number of sample data. Through the subsequent steps in the above embodiments of this application, high-quality training samples can be generated based on the limited number of sample data. It is understood that by reusing the methods described in steps S204 to S208, or by recombining one or more steps in steps S204 to S208, a large number of high-quality training samples can be obtained, thereby solving the technical problem of low-quality training samples obtained by existing methods.

[0074] In one optional approach, the aforementioned action description information set can be a sample dataset obtained from a publicly available dataset, including first object actions and second object actions corresponding to multiple virtual objects in multiple virtual scenes. In another optional approach, the aforementioned action description information set can be a sample dataset designed based on a specific game scene. For example, when the virtual object is specifically a game character, the aforementioned first object actions and second object actions can be a set of first object actions and second object actions designed for each of multiple game characters such as character a, character b, character c, etc. in game A. The embodiments of this application do not limit the method of obtaining the aforementioned action description information set.

[0075] Furthermore, the multiple action description information in the action description information set can be divided into multiple action description information groups. Each action description information group includes action description information for describing the actions of two different objects. For example, the action description information group includes first action description information for indicating the action of a first object and second action description information for indicating the action of a second object. The first object action and the second object action are different object actions.

[0076] In other words, in step S202 above, the action description information group may include first action description information for indicating the first object action and second action description information for the second object action. When the above embodiment is applied to a game application, the first object action and the second object action in step S202 may be two different object actions performed by a virtual game object displayed in a virtual game scene. Optionally, the first object action may be a starting object action, for example, an object action performed by the current virtual object in a static state or a state where no operation command has been received; the second object action may be a ending object action, for example, an object action performed by the current virtual object to indicate a hit state when attacked by other virtual objects.

[0077] The following details the information content that may be included in the first action description information used to indicate the action of the first object and the second action description information used to indicate the action of the second object in the above embodiments. In the above embodiments of this application, the first action description information and the second action description information may include state information of multiple object nodes used to drive the rendering of the target object, such as spatial position information, linear velocity information, or angular velocity information of the object nodes. The object nodes may be hierarchical object nodes included in a three-dimensional driving model used to render and display the target object in virtual space. The three-dimensional driving model may include a root node used to determine driving information according to control instructions, object nodes used to transmit driving information, and a set of skinned mesh objects for determining the spatial displacement of each point on the object surface of the target object according to the driving information. In the process of determining the spatial displacement of each skinned mesh according to the driving information, the spatial displacement of each skinned mesh object can be further determined by combining the skin weight information with the driving information.

[0078] The following description, in conjunction with Figure 5, illustrates an optional 3D driving model. As shown in Figure 5, the 3D driving model includes at least an object node set 504 and a skinned mesh 505. The object node set 504 receives control commands 502 and transmits and transforms driving information based on the connection relationships between object nodes. Multiple root nodes that can directly receive control commands 502 can transmit the control commands 502, combined with weight information, to connected object nodes. The skinned mesh 505 determines the displacement information of the target skinned mesh rendered on the target object based on the driving information output by node 1. Since the aforementioned first action description information and second action description information can include the relevant state information of the various object nodes, the object actions of the target object in the virtual scene can be displayed based on the aforementioned information set.

[0079] In one specific implementation, the information included in the first and second motion description information may be the displacement information and quaternion rotation information corresponding to each of the multiple object nodes in the object model used to render the target object. Optionally, the aforementioned first and second motion description information may be a set of information determined based on a BVH (Biovision Hierarchy Data Format) file. In the aforementioned BVH file, motion data is stored based on a tree structure of virtual object joints. The file records the offset of human skeleton nodes (i.e., a specific object node) from their parent nodes, and the motion information consists of Euler angle rotation in the local parent coordinate system and displacement of the root bone (i.e., a specific root node).

[0080] Furthermore, having obtained the action description information set and the action description information groups included therein based on step S202 above, one of the following steps can be performed before executing step S204 above:

[0081] Clustering Method 1: Determine the reference action description matrix corresponding to each of the multiple action description information groups in the action description information set; perform clustering operation on the multiple action description information groups according to the spatial distance relationship between the multiple reference action description matrices to obtain at least one subset of action description information;

[0082] In the first clustering method described above, multiple action description information groups can be clustered directly based on their respective reference action description matrices. In other words, in this method, clustering can be performed directly using action description information groups as units, where the reference action description matrix indicates the relevant features contained within an action description information group. Then, a clustering operation is performed based on the spatial distance relationships between the reference action description matrices to obtain a clustering result containing multiple subsets of action description information, where each subset is still considered an action description information group.

[0083] The following explanation of clustering method one, with reference to Figure 6, illustrates this approach. As shown in Figure 6, the action description information set 602 contains multiple action description information groups, among which action description information group 603 is included. It can be understood that each action description information group includes first action description information and second action description information (identified by square and elliptical graphics, respectively). During the clustering process, each action description information group indicates the relevant features it contains based on its corresponding reference action description matrix.

[0084] The above clustering method one can cluster based on action description information groups. As shown in Figure 6, after clustering the action description information set 602, multiple action description information subsets can be obtained, including action description information subset 604, action description information subset 606, and action description information subset 608. The units included in the above action description information subsets are still action description information groups.

[0085] Clustering Method 2: Determine at least one subset of first action description information based on the first action description information included in each of the multiple action description information groups, and determine at least one subset of second action description information based on the second action description information included in each of the multiple action description information groups. The first subset of action description information includes multiple first action description information that satisfy the second clustering condition, the second subset of action description information includes multiple second action description information that satisfy the third clustering condition, and at least one subset of action description information includes both the first subset of action description information and the second subset of action description information.

[0086] In the second clustering method described above, the first and second action description information included in multiple action description information groups can be clustered separately. In other words, in this second clustering method, clustering can be performed using action description information as the unit. During the clustering process, clustering operations can be performed based on the spatial distance relationship between the information matrices corresponding to each of the aforementioned action description information, resulting in a clustering result containing multiple subsets of action description information. The units included in each subset of action description information are either the first or second action description information.

[0087] The second clustering condition is used to cluster multiple first action description information, such as ensuring that the similarity of first action description information belonging to the same subset of first action description information is greater than 60% based on the second clustering condition. The third clustering condition is used to cluster multiple second action description information, such as ensuring that the similarity of second action description information belonging to the same subset of second action description information is greater than 60% based on the third clustering condition.

[0088] The following explanation of clustering method two, with reference to Figure 7, illustrates this concept. As shown in Figure 7, the action description information set 702 contains multiple action description information groups, among which action description information group 703 is included. It is understood that each of these multiple action description information groups includes first action description information and second action description information (identified by square and elliptical graphics, respectively).

[0089] The second clustering method described above can cluster based on action description information as units. In other words, the first action description information and the second action description information can be clustered separately. As shown in Figure 7, after clustering the action description information in the action description information set 702, multiple subsets of first action description information and multiple subsets of second action description information can be obtained. The multiple subsets of first action description information include subsets 704, 706, and 708; the multiple subsets of second action description information include subsets 710, 712, and 714. Through the second clustering method described above, the units included in the multiple subsets of action description information are updated to individual action description information.

[0090] Clustering method three involves clustering multiple action descriptions within the action description set. This means clustering based on the action description itself, without requiring the first and second action descriptions to form a single group. Subsequently, the first action description can be obtained from one subset of action descriptions, and the second action description can be obtained from another subset.

[0091] Furthermore, if multiple subsets of action description information are obtained based on the above clustering method one or clustering method two, step S204 as described above can be continued to obtain first action description information and second action description information from at least one subset of action description information. It is understood that the methods for obtaining the first action description information and second action description information from at least one subset of action description information may differ depending on the clustering method.

[0092] For example, regarding the first clustering method described above, since the units included in the subset of action description information obtained by clustering constitute an action description information group, an action description information group can be directly obtained from one subset of action description information, and the first action description information and the second action description information can be determined based on the action description information group. As shown in Figure 6, the first action description information and the second action description information can be determined based on an action description information group in the action description information subset 604.

[0093] For example, regarding the second clustering method described above, since the action description information subsets obtained from clustering include two types—a first action description information subset and a second action description information subset—each unit in the first action description information subset is a first action description information, and each unit in the second action description information subset is a second action description information. Therefore, a first action description information can be obtained from a first action description information subset, and a second action description information can be obtained from a second action description information subset. As shown in Figure 7, a first action description information can be obtained based on action description information subset 704, and a second action description information can be obtained from action description information subset 714.

[0094] Furthermore, after obtaining the first action description information and the second action description information through the above method, the action acquisition operation can be further performed based on the above step S206 to obtain multi-frame transition sample action information for indicating the action of multi-frame transition objects.

[0095] In one alternative implementation, the aforementioned transition sample motion information can be motion information determined based on the transition motion design draft. Specifically, the aforementioned transition motion design draft can be the motion design result obtained based on the art design operation.

[0096] In another alternative implementation, the aforementioned transitional sample motion information can also be obtained based on 3D video information output by a generative video model. For example, the first frame corresponding to the first object action indicated by the first motion description information and the second frame corresponding to the second object action indicated by the second motion description information can be input into the generative video model. The generative video model can then be prompted by relevant prompts to output a target video generated based on the first and second frames. Subsequently, the transitional sample motion information can be extracted from multiple intermediate frames included in the target video.

[0097] In another optional implementation, the aforementioned transitional sample motion information can also be motion information obtained based on motion capture technology. Correspondingly, in this implementation, the motion acquisition operation is an operation that collects transitional sample motions between two object actions. For example, the motion acquisition operation can specifically be a motion capture operation. It should be noted that motion capture technology can capture the performer's movements and map these movement data onto virtual objects, thereby determining the aforementioned multi-frame transitional sample motion information based on the motion description information of the virtual objects. In other words, in this implementation, while the performer demonstrates the smooth transition between the first object action and the second object action according to the actual motion performance operation, the relevant motion information of the performer can be collected to obtain the aforementioned multi-frame transitional sample motion information.

[0098] Having obtained multi-frame transition sample information through the above step S206, the target training sample is further obtained by combining the above first action description information and the second action description information as described in step S208.

[0099] In an optional implementation, if the target training samples are obtained based on the above steps S202 to S208, data augmentation operations can be performed based on the target training samples to obtain a larger number of training samples.

[0100] The main steps involved in a complete embodiment of this application are described below with reference to Figure 8. A complete embodiment of this application may include the following main steps:

[0101] S800, export game resources;

[0102] First, by exporting game resources in step S800, an initial dataset, namely the action description information set, can be obtained. This step provides the foundational data for subsequent data processing and analysis.

[0103] S802, Data Analysis and Screening;

[0104] The above steps mainly involve clustering operations to conduct in-depth analysis of the initial dataset and to effectively filter the data through clustering methods in order to identify and extract key action features.

[0105] S804, video motion capture;

[0106] The above steps capture the process of the performer executing transitional movements, thereby determining the sequence of transitional movements in the training samples and providing real motion sampling data for model training.

[0107] S806, enhanced motion data;

[0108] By implementing relevant data augmentation operations, a larger number of high-quality training samples can be generated, thereby enriching the training dataset and improving the model's generalization ability.

[0109] S808, model training.

[0110] Finally, the transition action prediction model is trained using the data acquired and processed in the previous steps, enabling it to accurately predict and generate transition actions.

[0111] The embodiments described above in this application provide a systematic method for processing and utilizing motion data to generate high-quality transition animations for applications such as 3D fighting games. This method not only improves the efficiency and quality of motion generation but also enhances the predictive ability and generalization performance of the model through data augmentation and model training.

[0112] Specifically, in the sample generation method based on the above clustering method one, similar action description information groups are grouped into one category by clustering action description information groups. That is, similar combinations of first and second action description information are grouped into one category. Thus, an action description information group can be obtained from a cluster to obtain a combination of first and second action description information used to represent the category. By performing subsequent action acquisition operations using the obtained action description information group, the acquired samples can exhibit high generalization, without having to repeat the action acquisition operation on other action description information groups in the same category. Thus, while ensuring the quality of training samples, the sample acquisition efficiency can be significantly improved.

[0113] For example, given the clustering results shown in Figure 6, one action description information group can be obtained from each of the action description information subsets 604, 606, and 608 to perform subsequent action acquisition operations, thereby obtaining three high-quality target training samples and improving the acquisition efficiency of training samples.

[0114] In the sample generation method based on the above-mentioned clustering method two, by performing clustering operations on the first action description information and the second action description information respectively, a first object action representation of the first action cluster can be obtained from a first object action cluster, and a second object action representation of the second action cluster can be obtained from a second object action cluster. On the one hand, it is not necessary to obtain other object actions in the same action cluster to perform subsequent action collection operations, thus improving the sample collection efficiency; on the other hand, when the first object action and the second object action are clustered separately, the richness of the samples can be improved by permuting and combining different first object action clusters and second object action clusters, thereby increasing the number of high-quality training samples.

[0115] For example, given the clustering results shown in Figure 7, a first action description can be obtained from action description information subsets 704, 706, and 708 respectively; and a second action description can be obtained from action description information subsets 710, 712, and 714 respectively. Then, based on the three first action descriptions and three second sample actions, nine sample action information groups are obtained. Subsequent action acquisition operations are then performed based on these nine sample action information groups, resulting in nine high-quality target training samples. This improves the efficiency of training sample acquisition and further enriches the number of training samples.

[0116] The following section provides further details on the implementation of each step involved in Figure 8.

[0117] In the clustering process, taking the above-mentioned clustering method one as an example, the method for determining at least one subset of action description information is explained. First, the reference action description matrix corresponding to each of the multiple action description information groups in the action description information set is determined, including:

[0118] S1, obtain the first action description matrix corresponding to the first action description information in the current action description information group, and the second action description matrix corresponding to the second action description information in the current action description information group;

[0119] S2, determine the action difference matrix based on the first action description matrix and the second action description matrix, and determine the third action description matrix based on the action difference matrix and the first action description matrix, wherein the first action description matrix is ​​used to indicate the first object action performed by the virtual object at the first moment, and the second action description matrix is ​​used to indicate the second object action performed by the virtual object at the second moment, which is later than the first moment.

[0120] S3, the projection matrix obtained by spatially projecting the third action description matrix onto the target vector space is determined as the reference action description matrix, wherein the first matrix dimension of the reference action description matrix is ​​lower than the second matrix dimension of the third action description matrix.

[0121] It is understood that, in the embodiments of this application, since the matrices corresponding to the first action description information and the second action description information are both high-dimensional matrices, in the above embodiments, the principal component analysis method can be used to perform dimensionality reduction processing on the third action description matrix used to characterize the information features of the current action description information group to obtain the above reference action description matrix.

[0122] The following provides a detailed explanation of the dimensionality reduction method described above, which may include the following steps:

[0123] S1, sample principal component learning;

[0124] Specifically, in the case where the action description information set includes n1 action description information groups, n1 sample matrices may be determined according to the offset result between the first action description matrix of the corresponding first action description information and the second action description matrix of the second action description information in each of the n1 action description information groups, that is, the third action description matrix, which correspond to a sample set, and each sample is represented by x i . Then the entire sample set can be expressed as:

[0125] First, mean removal processing is performed on the entire sample set, and the average value of the sample set is defined as follows:

[0126] Then the result after mean removal of the entire sample set is as follows:

[0127] The is mathematically subjected to SVD decomposition, which can obtain:

[0128] Wherein, U P and V P are two orthogonal matrices, Σ P is a diagonal matrix, and the singular values on the diagonal are arranged in descending order;

[0129] Assume: Σ P =diag(σ1,σ2,…,σ m )

[0130] Obtain a principal component coefficient a, where 0<a<1, which is used to indicate the feature proportion of the spatial feature indicated by the principal component matrix in the original feature space, and may generally be selected as 0.8 or 0.9. Then, a positive integer L is found p , such that both of the following inequalities hold:

[0131] It can be understood that, in the above inequality, after the singular values in Σ P are arranged in order of size, the minimum number of singular values is found, such that the sum of squares occupied by these singular values is a times the sum of squares of all the singular values. Since the value of a is usually close to 1, that is, this part of singular values can approximately replace all the singular values.

[0132] Further, the matrix dimension of the principal component matrix that can represent the original sample space can be determined correspondingly through this value, that is:

[0133] S2, sample projection calculation;

[0134] For each action description information group, its corresponding third action description matrix is ​​projected onto the sample principal component space based on the principal component analysis results described above. Sample x i The projection onto the principal component space of positive samples is calculated as follows: First, subtract the mean of the sample space, then multiply by the transpose of the principal component space matrix, i.e.:

[0135] Through the above-described embodiments of this application, high-dimensional action description information can be reduced in dimensionality while retaining its main information features, thereby significantly reducing the computational load in the subsequent clustering process and improving data processing efficiency.

[0136] Furthermore, after performing dimensionality reduction and obtaining the reference action description matrix corresponding to each action description information group, the method based on the reference action description matrix can be a clustering operation performed when the number of clusters is predetermined. When the number of clusters is determined, it can be a hard clustering method (i.e., a sample is only allowed to belong to one cluster) or a soft clustering method (i.e., a sample is allowed to belong to multiple clusters). Optionally, when the number of clusters cannot be predetermined, clusters can be formed based on the sample density.

[0137] In an optional implementation, given a predetermined number of clusters, the above-mentioned clustering method one performs a clustering operation on multiple action description information groups based on the spatial distance relationship between multiple reference action description matrices to obtain at least one subset of action description information, including:

[0138] S1, obtain multiple reference action description matrices and multiple current centroid matrices;

[0139] S2, based on multiple reference action description matrices and multiple current centroid matrices, determine the clustering distance information corresponding to each of the multiple reference action description matrices, wherein the clustering distance information includes multiple matrix distances between the multiple current centroid matrices and the reference action description matrices;

[0140] In the above steps, given the current centroid matrix, multiple matrix distances can be determined based on the multiple current centroid matrices and the reference action description matrix. Then, based on the matrix distance threshold, the reference action description matrix included in the cluster corresponding to each centroid matrix can be updated.

[0141] S3, based on the clustering distance information corresponding to each of the multiple reference action description matrices, update the clustering label matched by each reference action description matrix, where the clustering label is used for the action description information subset to which the action description information group currently belongs;

[0142] Understandably, in the current step, the cluster to which each reference action description matrix belongs can be re-determined by combining the distance threshold and the matrix distance between each reference action description matrix and each current centroid matrix, and then the corresponding cluster labels can be updated.

[0143] S4. Update the multiple current centroid matrices based on the reference action description matrices matched by each current centroid matrix to obtain the updated multiple current centroid matrices.

[0144] S4-1, If ​​multiple current centroid matrices satisfy the first clustering condition, determine at least one subset of action description information based on the clustering labels matched by each reference action description matrix.

[0145] S4-2, If multiple current centroid matrices do not satisfy the first clustering condition, repeat steps S1 to S4 above.

[0146] Optionally, the first clustering condition mentioned above may include, but is not limited to, one of the following: the number of iterations of the current centroid matrix is ​​greater than or equal to the target number; the spatial distance between the current centroid matrix and the historical centroid matrix corresponding to the previous round is less than or equal to the target threshold.

[0147] Through the above-described embodiments of this application, by calculating the spatial distance between the reference action description matrix and the current centroid matrix, the information group cluster to which each action description information group belongs can be accurately determined; at the same time, the clustering labels corresponding to each action description information group and the centroid matrix corresponding to each information group cluster will be dynamically updated according to the latest clustering distance information, thereby continuously approaching the optimal clustering result through the clustering labels, improving the flexibility and adaptability of clustering.

[0148] It should be noted that the implementation of the above-described clustering method two can also refer to the above implementation method. First, perform dimensionality reduction processing on the matrix used to represent action description information, and then perform clustering operations on the first action description information and the second action description information respectively based on the above implementation method. The specific implementation method can be implemented with reference to the above implementation method, and will not be repeated here.

[0149] In one specific embodiment of this application, the action combinations between the first object action and the second object action can be classified based on data feature analysis. Then, clustering is performed in the feature space to group actions with similar features into the same group. During action acquisition, sampling a certain number of actions within different groups allows for more efficient coverage of more similar actions with a smaller number of actions. The specific implementation process of the above clustering method one is described below with reference to Figure 9.

[0150] S900, Euler angle conversion 6D representation;

[0151] S902, Calculate the bias of the first object action and the second object action;

[0152] In this embodiment, a bias operation is performed on the first object action and the second object action of the character. Specifically, the difference between the action description information corresponding to the first object action and the second object action is taken, and the sum of the difference and the first object action is used as the bias result. Among them, the lower body nodes include nodes 1, 2, 3, 4, 5, 6, 7, and 8, and the remaining nodes are upper body nodes, as shown in Figure 10.

[0153] When the first object action is a standby action and the second object action is a hit action, the Euler angle data (α,β,γ) in the action is first converted into a six-dimensional two-axis rotation matrix vector V_6D based on the standby actions and hit action offsets of all characters. This vector contains an upward and a forward three-dimensional vector: V 6D =(R 11 ,R 12 ,R 13 ,R 21 ,R 22 ,R 23 )

[0154] in,

[0155] This embodiment can use a skeleton model with 22 joints, so the dimensions of the upper and lower body pose models are 84 (14*6) and 54 (9*6) respectively (both the upper and lower body include the hip joint).

[0156] S904 performs dimensionality reduction on the bias result;

[0157] During the dimensionality reduction process, principal component analysis can be performed first, followed by dimensionality reduction. Specifically, the data can be reduced to three dimensions for visualization. For details on the implementation, please refer to the above embodiments.

[0158] S906, Clustering Processing;

[0159] S908, classify data based on clustering results;

[0160] In this embodiment, the K-means clustering method can be used to classify the action combinations of different characters. Figure 11 shows a classification result in the feature space, where different three-dimensional points represent the positions of different human poses after dimensionality reduction in the feature space, and different filling methods are used to indicate different clustering results.

[0161] Understandably, during the data acquisition phase, motion capture operations are performed on the transitional movements of some characters based on the above classification in order to obtain target sample data.

[0162] Through the above-described embodiments of this application, in the sample generation method based on the above-described clustering method one, by performing clustering operations on action information groups, similar combinations of first action description information and second action description information are grouped into one category. Thus, an action description information group can be obtained from a cluster to represent the combination of first action description information and second action description information of that category. By performing subsequent action acquisition operations through the obtained action description information group, the acquired samples can exhibit high generalization, without having to repeatedly perform action acquisition operations on other action description information groups in the same category. Thus, while ensuring the quality of training samples, the sample acquisition efficiency can be significantly improved.

[0163] The following describes a further implementation of step S804 in Figure 8, namely, video motion capture. As shown in Figure 12, in this embodiment, the implementation of video motion capture may include three steps: exporting motion keyframe assets from the game engine, importing the motion keyframe assets into the motion capture software, and acquiring motion capture data of the actor. The steps of exporting motion keyframe assets from the game engine and importing them into the motion capture software correspond to S1 in the above embodiment, and the steps of acquiring motion capture data of the actor correspond to S2 and S3. The following provides a further explanation of each of these three steps.

[0164] In one optional implementation, the above-mentioned motion acquisition operation, based on the first motion description information and the second motion description information, to obtain multi-frame transition sample motion information, includes:

[0165] S1, display the first virtual object to perform the first object action according to the first action description information, and display the second virtual object to perform the second object action according to the second action description information, wherein the first virtual object is displayed according to the first style, and the second virtual object is displayed according to the second style;

[0166] S2, in response to the action acquisition operation, based on the action acquisition device acquiring the current sampled action information, displays the third virtual object to perform the action of the current sampled object, wherein the third virtual object is displayed according to the third style;

[0167] S3, if the collected multi-frame current sampling action information meets the collection conditions, determine the multi-frame transition sample action information that matches the first action description information and the second action description information based on the multi-frame current sampling action information.

[0168] In the above steps, two virtual objects performing different object actions (first object action and second object action) can be displayed in the display interface according to preset styles. For example, the first virtual object can be displayed according to the first style, and the second virtual object can be displayed according to the second style. This makes it easier for the actor object to determine the currently aligned action version according to different display styles during the performance of transition actions, thereby improving the matching degree between the action performed by the actor object and the action indicated by the action description information, thus ensuring that the acquisition of transition action sequences is completed more efficiently based on the actor object.

[0169] The first style and the second style mentioned above can be different color styles. For example, the first style is red and the second style is blue. The current action of the red virtual object indicates the action of the first object, and the current action of the blue virtual object indicates the action of the second object.

[0170] For example, the first style and the second style can also be a dashed stroke style and a solid stroke style. Specifically, the first object action of a virtual object with a dashed stroke can indicate that the first object action is the starting action; the second object action of a virtual object with a solid stroke can indicate that the second object action is the ending action.

[0171] For example, the first style and the second style can also be a transparent display style and an opaque display style. Specifically, the first object action of the virtual object with the transparent display style can indicate that the first object action is the starting action; the second object action of the virtual object with the opaque display style can indicate that the second object action is the ending action.

[0172] Furthermore, during motion capture operations, in response to motion acquisition, a third virtual object can be displayed. This third virtual object performs the action of the currently sampled object and is displayed according to a third style. It is understood that the action display of the third virtual object is a simulation or mapping of the user's actual actions, and the third style may differ from the first and second styles to distinguish actions acquired in real time.

[0173] Furthermore, the acquisition conditions satisfied in S3 above can be either the first type of conditions used to indicate the continuity of the transition action, or the second type of conditions used to indicate the stability of the transition action.

[0174] Through the above-described embodiments of this application, a highly efficient and intuitive motion capture and matching method is achieved by displaying virtual objects of different styles in a virtual environment to perform specific actions. In step S1, based on preset first and second motion description information, the actions of two virtual objects are displayed in first and second styles respectively. This visual distinction helps the motion capture actor clearly identify and understand the characteristics of each sample action. Next, in step S2, in response to the motion capture operation, the motion capture device captures the motion information of the motion capture actor in real time, and displays the current sampled action performed by the third virtual object in a third style. This not only provides the motion capture actor with immediate motion feedback, but also facilitates the motion capture actor to compare its own actions with sample actions. Finally, in step S3, when the acquired multi-frame motion information meets preset conditions, the system can accurately identify the transitional motion information that matches the sample action from the acquired actions, thereby significantly improving the efficiency and quality of motion capture and processing.

[0175] The following section further explains the process of exporting motion keyframe assets from a game engine and importing those assets into motion capture software.

[0176] In one optional implementation, displaying a first virtual object performing a first object action based on first action description information, and displaying a second virtual object performing a second object action based on second action description information, includes:

[0177] S1, in the target rendering engine, the first sample object action executed by the reference virtual object is rendered according to the first action description information, and the second sample object action executed by the reference virtual object is rendered according to the second action description information at the same virtual position.

[0178] S2, based on the actions of the first sample object and the second sample object, sample the first reference action information and the second reference action information respectively;

[0179] S3, initialize the root bone position information and rotation information in the first reference motion information and the second reference motion information to obtain the updated first reference motion information and the second reference motion information.

[0180] S4, in the motion acquisition application, the first virtual object is displayed to perform the first object action according to the updated first reference motion information, and the second virtual object is displayed to perform the second object action according to the second reference motion information.

[0181] The above implementation method will be described below with reference to Figure 13. For each transitional object action to be collected, the object action of two frames, namely the start frame and the end frame, needs to be exported. The object action of the start frame can be the object action corresponding to the initial time, and the object action of the end frame can be the object action corresponding to the end time, that is, corresponding to the first action description information and the second action description information, respectively.

[0182] Since directly using fixed art assets may not accurately reflect the real game performance—for example, knockback animations created by the art team may not include displacement data—in the embodiments described above, the motion assets are all exported from the target rendering engine. This allows the program to control the relevant data during game runtime, making the displacement of the motion capture data closer to the display effect in the virtual scene, thereby improving the training results of the transition motion prediction model.

[0183] Specifically, as shown in Figure 13, when the first object movement is a hit action (i.e., receiving an attack command) and the second object movement is a standby action (i.e., not receiving a control operation), such as in the first virtual scene 1302, the hit-end frame object movement can be selected as the current hit action. Then, in the second virtual scene 1303, the standby action of the character is sampled based on the screen rendered by the game engine at the current character position. Next, the root bone position and rotation of these two action poses are cleared to zero and redirected to a general character skeleton. Finally, the action is exported as an FBX format file, as shown in the action description information group 1304.

[0184] The following section further explains how to obtain motion capture data for actor objects.

[0185] In one optional implementation, after displaying the first virtual object performing a first object action based on the first action description information and displaying the second virtual object performing a second object action based on the second action description information, the method further includes at least one of the following:

[0186] The first adjustment method, in response to the first angle adjustment operation, displays the first virtual object and performs the first object action at the adjusted first display angle;

[0187] The second adjustment method, in response to the second angle adjustment operation, displays the second virtual object and performs the second object action at the adjusted second display angle;

[0188] The third adjustment method, in response to the third angle adjustment operation, displays the third virtual object performing the current sampling object action at the adjusted third display angle.

[0189] It is understood that, in the above embodiments, during the motion capture data acquisition process, the display angles of the first virtual object, the second virtual object, and the third virtual object can be adjusted respectively in response to different adjustment methods, so as to facilitate the alignment of the motion capture actor's movements with the movements of different objects during the performance.

[0190] For example, during the motion capture data acquisition process, the exported motion pose data can be imported into the motion capture software. These poses will retain the original position and rotation information and can be displayed from four different perspectives (front, back, left, and right) so that motion capture actors can observe and align their movements from multiple angles.

[0191] Furthermore, to help motion capture actors distinguish between the start and end frames of their poses, different poses can be marked with different colors and different fill methods. Assume that the first virtual object 1404 in Figure 14 is marked using the first fill method, the second virtual object 1406 is marked using the second fill method, and the third virtual object 1406 is marked using the third fill method. Specifically, the first virtual object 1404 indicates the starting action (the first object action), the third virtual object 1408 indicates the ending action (the second object action), and the second virtual object 1406 indicates the current object action performed by the actor object 1402.

[0192] When actors actually perform transitional movements, they can start from the pre-placed markers, assume the starting frame pose, and then naturally transition to the ending frame pose. By displaying different object movements according to different display styles, and providing a flexible method to adjust the display angle, the motion alignment process during motion capture is greatly simplified, and the accuracy is significantly improved.

[0193] In an optional implementation, before determining the multi-frame transition sample action information that matches the first action description information and the second action description information based on the multi-frame current sample action information, the method further includes: determining that the multi-frame current sample action information meets the acquisition conditions if the first matching degree between the starting sample action information and the first action description information in the multi-frame current sample action information is greater than or equal to a first threshold, and the second matching degree between the ending sample action information and the second action description information in the multi-frame current sample action information is greater than or equal to a second threshold.

[0194] In the above embodiments, a method is provided for determining whether multi-frame motion information meets specific acquisition conditions during motion capture and analysis. The multi-frame currently sampled motion information is used to indicate a series of motion data continuously acquired during motion capture.

[0195] By defining the acquisition conditions as described above, motion data highly similar to preset sample actions can be automatically filtered out, thereby improving the accuracy and reliability of motion capture. Specifically, by setting a matching threshold, the acquired motion information is filtered and verified to ensure that only motion data highly matching the preset sample actions is used. This method can improve the accuracy of motion capture, reduce the impact of erroneous data, and thus improve the quality of the acquired motion capture samples.

[0196] Figure 15 illustrates a schematic diagram of an optional motion capture scene setup, including at least two devices: a motion capture device and a motion preview device. It also provides the actor's range of motion for motion capture. The motion capture device may include multiple high-speed cameras that capture marker points on the actor from different angles, recording their movement trajectories to capture the actor's movements within their range of motion. The motion preview device can be used to display or play back the captured motion data in real time, helping the motion-captured actor to instantly check the effectiveness of the motion capture and ensuring the accuracy and quality of the motion data.

[0197] In one alternative approach, during motion capture, the actor performs actions within a pre-defined range of motion, with markers on their body captured by multiple motion capture devices. The captured motion data is recorded by a script supervisor and displayed and reviewed in real-time on a motion preview device. The actor can also observe the content displayed on the motion preview device in real-time and adjust their movements accordingly. Ultimately, this data is used to generate high-quality motion assets, serving as an important component of the target training samples.

[0198] The following explains the further implementation of step S806 in Figure 8, which is motion data augmentation.

[0199] In one optional implementation, after determining the target training sample based on the first action description information, the second action description information, and the multi-frame transition sample action information, the method further includes at least one of the following:

[0200] Enhancement Method 1: Obtain sample action information sequence from the current training sample, which includes multiple target training samples. The sample action information sequence includes first action description information, second action description information, and multi-frame transition sample action information.

[0201] Perform mirror transformation operations on multiple action description information in the sample action information sequence to obtain the first action description information sequence;

[0202] The sequence of first action description information is used as the new target training sample.

[0203] It is understandable that in the above enhancement method one, a mirror transformation operation can be used on the already generated target training samples to obtain new training samples, which can then be added to the sample set.

[0204] Enhancement Method 2: Obtain sample action information sequence from the current training samples including multiple target training samples, wherein the sample action information sequence includes first action description information, second action description information and multi-frame transition sample action information;

[0205] A velocity transformation operation is performed on the sample action information sequence to obtain a second action description information sequence;

[0206] The second action description information sequence is used as a new target training sample.

[0207] Understandably, in the above-mentioned enhancement method two, a velocity transformation operation can be applied to the already generated target training samples to obtain new training samples, which can then be added to the sample set.

[0208] Enhancement Method 3: Obtain sample action information sequence from the current training samples including multiple target training samples, wherein the sample action information sequence includes first action description information obtained from the first action description subset, second action description information obtained from the second action description subset, and multi-frame transition sample action information.

[0209] The first action description information, the second action description information, and the action information of the multi-frame transition samples are used as new target training samples.

[0210] It should be noted that the above three enhancement methods can be used in combination. For example, when the first training sample is obtained based on enhancement method one, enhancement method two can also be executed based on the first training sample to obtain the second training sample. The embodiments of this application do not limit the specific combination of the above three enhancement methods.

[0211] It is understandable that in the above enhancement method three, when the target training sample is obtained according to clustering method two, a new training sample can be obtained by combining the collected transition action sequence and the clustering results of the first object action and the second object action.

[0212] The following description, in conjunction with Figure 16, illustrates a data augmentation method based on clustering method two. As shown in Figure 16, the first action description information subset 1602 includes multiple first action description information obtained from the clustering operation, and the second action description information subset 1604 includes multiple second action description information obtained from the clustering operation. When a motion capture operation is performed based on the first action description information a1 in the first action description information subset 1602 and the second action description information b1 in the second action description information subset 1604 to obtain a transitional action sequence 1606, the target training sample can be determined based on the first action description information a1, the second action description information b1 in the second action description information subset 1604, and the transitional action sequence 1606.

[0213] Furthermore, the newly added target training sample can be determined based on the first action description information a2 in the first action description information subset 1602 and the second action description information b1 in the second action description information subset 1604, combined with the aforementioned transition action sequence 1606.

[0214] Furthermore, the newly added target training sample can be determined based on the first action description information a3 in the first action description information subset 1602 and the second action description information b2 in the second action description information subset 1604, combined with the aforementioned transition action sequence 1606.

[0215] Understandably, in the second clustering method described above, since the first object action and the second object action can be clustered separately, the similarity between the first object actions within the same cluster can exceed a certain threshold. Based on this, in the obtained target training samples, the first object action and the second object action can be replaced by the first object action or the second object action within the same cluster, such as mirror transformation operations or velocity transformation operations, thereby quickly obtaining new samples and improving the efficiency of training sample generation.

[0216] In the above enhancement method one, a mirror transformation operation is performed on multiple sample action information in the sample action information sequence to obtain a first action description information sequence, including:

[0217] S1, Obtain the current sample action information from multiple sample action information;

[0218] S2, perform the first mirror symmetry operation on the target rotation information of the first part included in the current sample action information;

[0219] S3, perform the second mirror symmetry operation on the three-dimensional coordinate information of the second part included in the current sample action information;

[0220] S4, the updated current sample action information is determined as the first action description information included in the first action description information sequence.

[0221] In the first enhancement method described above, to ensure that the enhanced action does not disrupt the rhythm of the original action, mirror symmetry operations can be performed on the left and right lateral joints (first part) and the hip joint (second part). Specifically, as shown in Figure 17, firstly, the character's frontal orientation in the starting frame is adjusted to the positive y-axis. Then, according to the storage method of BVH data, the rotation of the left and right lateral joints and the three-dimensional coordinates of the hip joint are mirrored along the yz plane to obtain mirrored actions. As shown in Figure 17, after performing mirror operations on each object action in the first action sequence 1702 in the above manner, the newly added training samples, namely the second action sequence 1704, can be obtained.

[0222] In the above enhancement method two, a velocity transformation operation is performed on the sample action information sequence to obtain a second action description information sequence, including:

[0223] S1, Obtain the velocity description information corresponding to the velocity transformation operation;

[0224] Optionally, the speed description information described above can be described by an acceleration or deceleration factor δ.

[0225] S2, determine the number of second action description information included in the second action description information sequence based on the speed description information, and the sequence identifier corresponding to each of the multiple action description information in the second action description information sequence;

[0226] S3, determine the weight information between every two adjacent second action descriptions based on the velocity description information;

[0227] S4. Based on the sequence identifiers corresponding to each of the multiple action description information and the weight information between every two adjacent second action description information, interpolation processing is performed on the multiple action description information to obtain the second action description information sequence.

[0228] Since the animation playback speed is the same, either 30 frames per second or 60 frames per second, in the above implementation, interpolation can be used to reduce / increase the total number of frames of the original animation while ensuring the consistency of the animation performance, thereby achieving acceleration / deceleration effects.

[0229] In this embodiment, an acceleration or deceleration factor δ is first defined according to requirements. Then, the number of frames after acceleration / deceleration is determined by division based on the acceleration / deceleration factor, and the sequence number i of each new frame after interpolation is determined accordingly. inter (corresponding to the original sequence number), and the interpolation weights corresponding to each newly generated frame.

[0230]

[0231] Where i represents the original frame number, i inter Indicates the frame number after acceleration / deceleration. Indicates the i-th value after interpolation inter Frame and i inter +1 frame weighting coefficient This indicates rounding down to the nearest integer.

[0232] Furthermore, based on the sequence identifiers corresponding to each of the multiple action description information pieces and the weight information between every two adjacent second action description information pieces, the above-mentioned interpolation processing is performed on the multiple action description information pieces to obtain the second action description information sequence, including:

[0233] S1, Multiple action description information undergoes coordinate transformation to obtain global action information of multiple samples in the global coordinate system;

[0234] S2, perform linear interpolation based on the global position information in the global action information of multiple samples to obtain multiple global position information corresponding to the second action description information sequence;

[0235] S3, perform spherical linear interpolation based on the global rotation information in multiple action description information to obtain multiple global rotation information corresponding to the second action description information sequence;

[0236] S4. Determine the sequence of second action description information based on multiple global position information and multiple global rotation information.

[0237] In this embodiment of the application, each frame is determined by the root node position and the local joint rotation R. local Composition. Since interpolation calculations need to be performed in the global coordinate system, it is necessary to first use FK (Forward Kinematics) to convert the original joint rotation data R, stored in local coordinates, into a single coordinate system. local Convert to global rotation R global and use R global Perform interpolation.

[0238] Wherein, the Root position (P) root The global rotation R can be obtained using linear interpolation. global The Slerp method, a spherical linear interpolation technique, is used to obtain the result. The specific calculation process is as follows:

[0239] Where p and q are quaternion representations, and t is the weighting coefficient. This is the root position after interpolation. This is the rotation after interpolation.

[0240] Through the above-described embodiments of this application, by adjusting the magnitude of different acceleration / deceleration factors δ, action sequences with different rhythms can be generated.

[0241] The following describes a complete embodiment of this application with reference to FIG18. S1800, obtain the action description information set; then perform a judgment, S1802, whether to adopt clustering method one. If it is determined to adopt clustering method one, first cluster the action description information group according to clustering method one as shown in FIG6, then execute S1804, obtain the first action description information and the second action description information from the action description information group;

[0242] If clustering method one is not executed, i.e. clustering method two is executed, the action description information is first clustered according to clustering method two shown in Figure 7 to obtain multiple first action description information subsets and multiple second action description information subsets. Then, S1804-1 is executed to obtain the first action description information and the second action description information from the two clusters respectively.

[0243] Next, step S1806 is executed to acquire multi-frame transition motion sample information through motion capture. In this embodiment of the application, the above-mentioned acquisition of multi-frame transition motion sample information through motion capture may include three steps: exporting motion keyframe assets from the game engine, importing the motion keyframe assets into motion capture software, and acquiring motion capture data of the actor object.

[0244] The first number of target training samples can be obtained through the above step S1806.

[0245] Next, execute judgment S1808 to determine whether clustering method one was used. If clustering method one was used, execute S1810 to perform data augmentation on the training samples using data augmentation method one and data augmentation method two; if clustering method one was not used, i.e., clustering method two was used, execute S1810-1 to perform data augmentation on the training samples using data augmentation method one, data augmentation method two, and data augmentation method three.

[0246] In the first enhancement method, a mirror transformation operation can be applied to the already generated target training samples to obtain new training samples, which are then added to the sample set. In the second enhancement method, a velocity transformation operation can be applied to the already generated target training samples to obtain new training samples, which are then added to the sample set. In the third enhancement method, when the target training samples are obtained according to the second clustering method, new training samples can be obtained by combining the collected transition action sequences and the clustering results of the first object action and the second object action.

[0247] S1812, Determine the target sample set. Through the above implementation method, a second number of target training samples can be obtained, wherein the second number is greater than the first number.

[0248] After obtaining the target sample set, Figure 19 below illustrates the training method of an action prediction model. During model training, a complete action segment is randomly sampled from the dataset. The beginning and end of the action are selected as the first action description information corresponding to the start frame and the second action description information corresponding to the end frame, respectively, and used as inputs to the transition action prediction model. The model is optimized by performing difference learning between the model's prediction results and the original high-quality transition actions in the dataset.

[0249] As shown in Figure 19, firstly, a complete action sequence is randomly sampled from the original dataset. Then, the beginning (starting action) and end (ending action) of the action are selected as the first action description information corresponding to the starting frame and the second action description information corresponding to the ending frame, respectively. These two frames serve as input to the AI ​​model (i.e., the transition action prediction model), used to train the model to understand and predict action transitions. That is, the first and second action description information are input into the transition action prediction model, which learns the action transition between these two frames to predict the entire transition action. During model training, the model's prediction results are compared with existing high-quality transition actions in the dataset, and difference learning is used to optimize the transition action prediction model.

[0250] The entire process is shown in Figure 19. Through continuous data sampling, model training, and optimization learning, a transition action prediction model that can accurately predict the actions of transitional objects is finally trained. This method not only improves the efficiency of action generation but also ensures the naturalness and realism of the actions.

[0251] The above-described embodiments of this application can reduce the amount of data required to construct a dataset for transition actions in fighting games, improve the motion capture efficiency of transition actions in 3D fighting games, thereby saving human resources and time costs. At the same time, it can ensure the high quality of model training results and is more suitable for training specific tasks than public datasets.

[0252] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0253] According to another aspect of the embodiments of this application, an apparatus for generating training samples for implementing the above-described training sample generation method is also provided. As shown in FIG20, the apparatus includes:

[0254] The first acquisition unit 2002 is used to acquire an action description information set, wherein the action description information set includes multiple action description information, and at least two of the multiple action description information indicate different object actions.

[0255] The second acquisition unit 2004 is used to acquire first action description information and second action description information from at least one subset of action description information when performing a clustering operation based on the above action description information set to determine at least one subset of action description information. The first action description information is used to indicate a first object action, and the second action description information is used to indicate a second object action. The first object action and the second object action are different object actions.

[0256] The acquisition unit 2006 is used to perform an action acquisition operation based on the first action description information and the second action description information to obtain multiple frames of transition sample action information, wherein the multiple frames of the transition sample action information are used to indicate multiple frames of transition object actions that match the first object action and the second object action.

[0257] The determining unit 2008 is used to determine target training samples based on the first action description information, the second action description information and the action information of multiple frames of transition samples, wherein the target training samples are used to train the transition action prediction model.

[0258] Optionally, the above-mentioned training sample generation device further includes one of the following: a first clustering unit, configured to determine reference action description matrices corresponding to each of the plurality of action description information groups in the action description information set; perform clustering operations on the plurality of action description information groups according to the spatial distance relationship between the plurality of reference action description matrices to obtain at least one subset of the action description information, wherein the action description information groups include action description information for indicating the action corresponding to two different object actions respectively; a second clustering unit, configured to determine at least one first action description information subset based on the first action description information included in each of the plurality of action description information groups, and determine at least one second action description information subset based on the second action description information included in each of the plurality of action description information groups, wherein the first action description information subset includes a plurality of first action description information satisfying a second clustering condition, the second action description information subset includes a plurality of second action description information satisfying a third clustering condition, the at least one subset of the action description information includes the first action description information subset and the second action description information subset, and the action description information groups include action description information for indicating the action corresponding to two different object actions respectively.

[0259] Optionally, the first clustering unit is configured to: obtain a first action description matrix corresponding to the first action description information in the current action description information group, and a second action description matrix corresponding to the second action description information in the current action description information group; determine an action difference matrix based on the first action description matrix and the second action description matrix, and determine a third action description matrix based on the action difference matrix and the first action description matrix, wherein the first action description matrix is ​​used to indicate the first object action performed by the virtual object at a first moment, and the second action description matrix is ​​used to indicate the second object action performed by the virtual object at a second moment, the second moment being later than the first moment; and determine the projection matrix obtained by spatially projecting the third action description matrix into the target vector space as the reference action description matrix, wherein the first matrix dimension of the reference action description matrix is ​​lower than the second matrix dimension of the third action description matrix.

[0260] Optionally, the first clustering unit is configured to: obtain multiple reference action description matrices and multiple current centroid matrices; determine clustering distance information corresponding to each of the multiple reference action description matrices based on the multiple reference action description matrices and the multiple current centroid matrices, wherein the clustering distance information includes multiple matrix distances between the multiple current centroid matrices and the reference action description matrices; update the clustering labels matched by each of the multiple reference action description matrices based on the clustering distance information corresponding to each of the multiple reference action description matrices, wherein the clustering labels are used for the action description information subset to which the action description information group currently belongs; update the multiple current centroid matrices based on the reference action description matrices matched by each of the multiple current centroid matrices to obtain updated multiple current centroid matrices; if the multiple current centroid matrices satisfy the first clustering condition, determine at least one subset of the action description information based on the clustering labels matched by each of the multiple reference action description matrices; if the multiple current centroid matrices do not satisfy the first clustering condition, repeat the above steps.

[0261] Optionally, the second acquisition unit 2004 includes a first acquisition module and a second acquisition module. The first acquisition module is used to: after performing a clustering operation on multiple action description information groups based on the spatial distance relationship between multiple reference action description matrices to obtain at least one subset of action description information, the module further includes: acquiring the action description information group from the subset of action description information, and acquiring the first action description information and the second action description information from the subset of action description information. The second acquisition module is used to: after determining at least one first action description information subset based on the first action description information included in each of the multiple action description information groups, and determining at least one second action description information subset based on the second action description information included in each of the multiple action description information groups, the module further includes: acquiring the first action description information from the subset of first action description information, and acquiring the second action description information from the subset of second action description information.

[0262] Optionally, the acquisition unit 2006 is configured to: display a first virtual object performing the first object action according to the first action description information, and display a second virtual object performing the second object action according to the second action description information, wherein the first virtual object is displayed according to a first style and the second virtual object is displayed according to a second style; in response to an action acquisition operation, if the action acquisition device acquires current sampled action information, display a third virtual object performing the current sampled object action according to the current sampled action information, wherein the third virtual object is displayed according to a third style; and if the acquired multiple frames of current sampled action information meet the acquisition conditions, determine multiple frames of transitional sampled action information that match the first action description information and the second action description information based on the multiple frames of current sampled action information.

[0263] Optionally, the acquisition unit 2006 is configured to: render a first sample object action performed by a reference virtual object in the target rendering engine according to the first action description information, and render a second sample object action performed by the reference virtual object at the same virtual position according to the second action description information; sample first reference action information and second reference action information according to the first sample object action and the second sample object action; initialize the root bone position information and rotation information in the first reference action information and the second reference action information to obtain updated first reference action information and second reference action information; and in the action acquisition application, display the first virtual object performing the first object action according to the updated first reference action information, and display the second virtual object performing the second object action according to the second reference action information.

[0264] Optionally, the acquisition unit 2006 is further configured to perform at least one of the following adjustment methods: a first adjustment method: in response to a first angle adjustment operation, displaying the first virtual object performing the first object action at the adjusted first display angle; a second adjustment method: in response to a second angle adjustment operation, displaying the second virtual object performing the second object action at the adjusted second display angle; and a third adjustment method: in response to a third angle adjustment operation, displaying the third virtual object performing the currently sampled object action at the adjusted third display angle.

[0265] Optionally, the acquisition unit 2006 is further configured to: determine that the current sampling action information of multiple frames satisfies the acquisition conditions when the first matching degree between the starting sampling action information in the current sampling action information of multiple frames and the first action description information is greater than or equal to a first threshold, and the second matching degree between the ending sampling action information in the current sampling action information of multiple frames and the second action description information is greater than or equal to a second threshold.

[0266] Optionally, the training sample generation apparatus further includes at least one of the following enhancement methods: a first sample enhancement unit, configured to obtain a sample action information sequence from the current training samples included in the plurality of target training samples, wherein the sample action information sequence includes the first action description information, the second action description information, and multiple frames of transition sample action information; perform mirror transformation operations on the plurality of action description information in the sample action information sequence to obtain a first action description information sequence; and use the first action description information sequence as a newly added target training sample; a second sample enhancement unit, configured to obtain a sample action information sequence from the current training samples included in the plurality of target training samples, wherein the sample action information sequence includes the first action description information, the second action description information, and multiple frames of transition sample action information; perform mirror transformation operations on the plurality of action description information in the sample action information sequence to obtain a first action description information sequence; and use the first action description information sequence as a newly added target training sample; and a second sample enhancement unit, configured to obtain a sample action information sequence from the current training samples included in the plurality of target training samples, wherein the sample action information sequence includes the first action description information, the second action description information, and multiple frames of transition sample action information. The system comprises: a first action description information, the second action description information, and multiple frames of transition sample action information; a velocity transformation operation is performed on the sequence of sample action information to obtain a second action description information sequence; the second action description information sequence is used as a newly added target training sample; and a third sample enhancement unit is used to obtain a sample action information sequence from the current training samples included in the multiple target training samples, wherein the sample action information sequence includes the first action description information obtained from the first action description subset, the second action description information obtained from the second action description subset, and multiple frames of transition sample action information; and the first action description information, the second action description information, and the multiple frames of transition sample action information are used as newly added target training samples.

[0267] Optionally, the first sample enhancement unit is configured to: obtain current sample action information from multiple sample action information; perform a first mirror symmetry operation on the target rotation information of the first part included in the current sample action information; perform a second mirror symmetry operation on the three-dimensional coordinate information of the second part included in the current sample action information; and determine the updated current sample action information as the first action description information included in the first action description information sequence.

[0268] Optionally, the second sample enhancement unit is configured to: acquire velocity description information corresponding to the velocity transformation operation; determine the number of second action description information included in the second action description information sequence and the sequence identifier corresponding to each of the multiple action description information in the second action description information sequence based on the velocity description information; determine the weight information between every two adjacent second action description information based on the velocity description information; and perform interpolation processing on the multiple action description information based on the sequence identifier corresponding to each of the multiple action description information and the weight information between every two adjacent second action description information to obtain the second action description information sequence.

[0269] Optionally, the second sample enhancement unit is configured to: perform coordinate transformation operations on the multiple action description information to obtain multiple sample global action information in the global coordinate system; and perform linear interpolation operations on the global position information in the multiple sample global action information to obtain multiple global position information corresponding to the sequence of the second action description information.

[0270] Perform spherical linear interpolation based on the global rotation information in the global motion information of the above samples to obtain multiple global rotation information corresponding to the above second motion description information sequence; determine the above second motion description information sequence based on the multiple global position information and the multiple global rotation information.

[0271] Optionally, in this embodiment, the implementation of each of the above-mentioned unit modules can be referred to the above-mentioned method embodiments, which will not be repeated here.

[0272] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described method for generating training samples is also provided. This electronic device may be the terminal device or server shown in FIG. 21. This embodiment uses the electronic device as a terminal device as an example for illustration. As shown in FIG. 21, the electronic device includes a memory 2102 and a processor 2104. The memory 2102 stores a computer program, and the processor 2104 is configured to execute the steps in any of the above-described method embodiments through the computer program.

[0273] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0274] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0275] S1, Obtain an action description information set, wherein the action description information set includes multiple action description information, and at least two of the multiple action description information indicate different object actions respectively;

[0276] S2, when at least one subset of action description information is determined by performing a clustering operation based on the above action description information set, first action description information and second action description information are obtained from at least one subset of the above action description information. The first action description information is used to indicate a first object action, and the second action description information is used to indicate a second object action. The first object action and the second object action are different object actions.

[0277] S3, based on the first action description information and the second action description information, perform an action acquisition operation to obtain multiple frames of transition sample action information, wherein the multiple frames of the transition sample action information are used to indicate multiple frames of transition object actions that match the first object action and the second object action.

[0278] S4, determine the target training sample based on the first action description information, the second action description information and the action information of the transition sample in multiple frames, wherein the target training sample is used to train the transition action prediction model.

[0279] Optionally, those skilled in the art will understand that the structure shown in Figure 21 is merely illustrative, and the electronic device may also be a vehicle terminal, smartphone, tablet computer, PDA, mobile internet device (MID), PAD, or other terminal device. Figure 21 does not limit the structure of the aforementioned electronic device. For example, the electronic device may include more or fewer components (such as network interfaces) than shown in Figure 21, or have a different configuration than that shown in Figure 21.

[0280] The memory 2102 can be used to store software programs and modules, such as the program instructions / modules corresponding to the training sample generation method and apparatus in this embodiment. The processor 2104 executes various functional applications and data processing by running the software programs and modules stored in the memory 2102, thereby realizing the training sample generation method described above. The memory 2102 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 2102 may further include memory remotely located relative to the processor 2104, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 2102 may be used, but is not limited to, to store file information such as target logical files. As an example, as shown in FIG21, the memory 2102 may include, but is not limited to, the first acquisition unit 2002, the second acquisition unit 2004, the acquisition unit 2006, and the determination unit 2008 in the training sample generation apparatus described above. In addition, other module units in the above-mentioned training sample generation device may also be included, but will not be described in detail in this example.

[0281] Optionally, the transmission device 2106 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 2106 includes a Network Interface Controller (NIC), which can be connected to other network devices and routers via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 2106 is a radio frequency (RF) module, used for wireless communication with the Internet.

[0282] In addition, the aforementioned electronic device also includes a display 2108 and a connection bus 2110 for connecting various module components in the aforementioned electronic device.

[0283] In other embodiments, the aforementioned terminal device or server can be a node in a distributed system, wherein the distributed system can be a blockchain system, which is a distributed system formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer network, and any form of computing device, such as a server, terminal, or other electronic device, can become a node in the blockchain system by joining this peer-to-peer network.

[0284] According to one aspect of this application, a computer program product is provided, comprising a computer program / instructions containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs various functions provided in embodiments of this application.

[0285] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0286] According to one aspect of this application, a computer-readable storage medium is provided, wherein a processor of a computer device reads computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the above-described method for generating training samples.

[0287] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0288] S1, Obtain an action description information set, wherein the action description information set includes multiple action description information, and at least two of the multiple action description information indicate different object actions respectively;

[0289] S2, when at least one subset of action description information is determined by performing a clustering operation based on the above action description information set, first action description information and second action description information are obtained from at least one subset of the above action description information. The first action description information is used to indicate a first object action, and the second action description information is used to indicate a second object action. The first object action and the second object action are different object actions.

[0290] S3, based on the first action description information and the second action description information, perform an action acquisition operation to obtain multiple frames of transition sample action information, wherein the multiple frames of the transition sample action information are used to indicate multiple frames of transition object actions that match the first object action and the second object action.

[0291] S4, determine the target training sample based on the first action description information, the second action description information and the action information of the transition sample in multiple frames, wherein the target training sample is used to train the transition action prediction model.

[0292] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0293] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0294] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0295] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0296] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0297] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0298] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for generating training samples, said method being applied to a server or terminal, comprising: Obtain an action description information set, wherein the action description information set includes multiple action description information, and at least two of the multiple action description information indicate different object actions; When performing a clustering operation based on the action description information set to determine at least one subset of action description information, first action description information and second action description information are obtained from at least one subset of action description information. The first action description information is used to indicate a first object action, and the second action description information is used to indicate a second object action. The first object action and the second object action are different object actions. Based on the first action description information and the second action description information, an action acquisition operation is performed to obtain multi-frame transition sample action information, wherein the multi-frame transition sample action information is used to indicate multi-frame transition object actions that match the first object action and the second object action. The target training sample is determined based on the first action description information, the second action description information, and the transition sample action information of multiple frames, wherein the target training sample is used to train the transition action prediction model.

2. The method according to claim 1, wherein, before obtaining the first action description information and the second action description information from at least one subset of the action description information after performing a clustering operation based on the action description information set to determine at least one subset of the action description information, it further includes one of the following clustering methods: Clustering Method 1: Determine the reference action description matrix corresponding to each of the multiple action description information groups in the action description information set; perform a clustering operation on the multiple action description information groups according to the spatial distance relationship between the multiple reference action description matrices to obtain at least one subset of the action description information, wherein the action description information group includes action description information used to indicate the action description information corresponding to the actions of two different objects respectively; Clustering Method 2: At least one subset of first action description information is determined based on the first action description information included in each of the multiple action description information groups, and at least one subset of second action description information is determined based on the second action description information included in each of the multiple action description information groups, wherein... The first subset of action description information includes multiple first action description information that satisfy the second clustering condition, the second subset of action description information includes multiple second action description information that satisfy the third clustering condition, at least one subset of action description information includes the first subset of action description information and the second subset of action description information, and the action description information group includes action description information for indicating the actions of two different objects respectively.

3. The method according to claim 2, wherein determining the reference action description matrix corresponding to each of the plurality of action description information groups in the action description information set includes: Obtain the first action description matrix corresponding to the first action description information in the current action description information group, and the second action description matrix corresponding to the second action description information in the current action description information group; An action difference matrix is ​​determined based on the first action description matrix and the second action description matrix, and a third action description matrix is ​​determined based on the action difference matrix and the first action description matrix. The first action description matrix is ​​used to indicate the first object action performed by the virtual object at a first moment, and the second action description matrix is ​​used to indicate the second object action performed by the virtual object at a second moment, which is later than the first moment. The projection matrix obtained by spatially projecting the third action description matrix onto the target vector space is determined as the reference action description matrix, wherein the first matrix dimension of the reference action description matrix is ​​lower than the second matrix dimension of the third action description matrix.

4. The method according to claim 2 or claim 3, wherein performing a clustering operation on the plurality of action description information groups based on the spatial distance relationship between the plurality of reference action description matrices to obtain at least one subset of the action description information includes: Obtain the reference action description matrices and the current centroid matrices; Based on the plurality of reference action description matrices and the plurality of current centroid matrices, clustering distance information corresponding to each of the plurality of reference action description matrices is determined, wherein the clustering distance information includes multiple matrix distances between the plurality of current centroid matrices and the reference action description matrices; Based on the clustering distance information corresponding to each of the multiple reference action description matrices, update the clustering label matched by each of the reference action description matrices, wherein the clustering label is used for the action description information subset to which the action description information group currently belongs; Based on the reference action description matrix matched by each of the multiple current centroid matrices, the multiple current centroid matrices are updated to obtain the multiple updated current centroid matrices; If multiple current centroid matrices satisfy the first clustering condition, at least one subset of action description information is determined based on the clustering labels matched by each of the reference action description matrices. If multiple current centroid matrices do not satisfy the first clustering condition, repeat the above steps.

5. The method according to any one of claims 2-4, After performing a clustering operation on multiple action description information groups based on the spatial distance relationship between multiple reference action description matrices to obtain at least one subset of the action description information, the process includes: Obtain the action description information group from the action description information subset, and obtain the first sample action information and the sample action information from the action description information group; After determining at least one subset of first action description information based on the first action description information included in each of the plurality of action description information groups, and determining at least one subset of second action description information based on the second action description information included in each of the plurality of action description information groups, the process includes: obtaining the first sample action information from the first subset of action description information, and obtaining the sample action information from the second subset of action description information.

6. The method according to any one of claims 1-5, wherein the step of performing an action acquisition operation based on the first action description information and the second action description information to obtain multi-frame transition sample action information includes: According to the first action description information, the first virtual object is displayed to perform the first object action, and according to the second action description information, the second virtual object is displayed to perform the second object action, wherein the first virtual object is displayed according to the first style, and the second virtual object is displayed according to the second style; In response to the motion acquisition operation, based on the current sampled motion information acquired by the motion acquisition device, a third virtual object is displayed to perform the current sampled object's action according to the current sampled motion information, wherein the third virtual object is displayed according to a third style; If the current sampling action information of the collected multiple frames meets the collection conditions, the transition sample action information of the multiple frames that matches the first action description information and the second action description information is determined based on the current sampling action information of the multiple frames.

7. The method according to claim 6, wherein displaying a first virtual object performing the first object action according to the first action description information, and displaying a second virtual object performing the second object action according to the second action description information, comprises: In the target rendering engine, a first sample object action executed by the reference virtual object is rendered according to the first action description information, and a second sample object action executed by the reference virtual object is rendered at the same virtual location according to the second action description information. Based on the actions of the first sample object and the actions of the second sample object, first reference action information and second reference action information are sampled respectively. The root bone position information and rotation information in the first reference motion information and the second reference motion information are initialized to obtain the updated first reference motion information and the second reference motion information. In motion capture applications, the first virtual object is displayed to perform the first object action based on the updated first reference motion information, and the second virtual object is displayed to perform the second object action based on the second reference motion information.

8. The method according to claim 6 or claim 7, wherein after displaying the first virtual object performing the first object action according to the first action description information and displaying the second virtual object performing the second object action according to the second action description information, it further includes at least one of the following adjustment methods: First adjustment method: In response to the first angle adjustment operation, the first virtual object is displayed and the first object action is performed at the adjusted first display angle; Second adjustment method: In response to the second angle adjustment operation, the second virtual object is displayed and the second object action is performed at the adjusted second display angle; Third adjustment method: In response to the third angle adjustment operation, the third virtual object is displayed to perform the action of the current sampled object at the adjusted third display angle.

9. The method according to any one of claims 6-8, before determining the transition sample action information of multiple frames matching the first action description information and the second action description information based on the current sample action information of multiple frames, further comprising: If the first matching degree between the starting sampling action information and the first action description information in the current sampling action information of multiple frames is greater than or equal to a first threshold, and the second matching degree between the ending sampling action information and the second action description information in the current sampling action information of multiple frames is greater than or equal to a second threshold, then the current sampling action information of multiple frames is determined to satisfy the acquisition condition.

10. The method according to claim 2, wherein after determining the target training sample based on the first action description information, the second action description information, and the action information of the transition samples in multiple frames, it further includes at least one of the following enhancement methods: Enhancement Method 1: Obtain a sequence of sample action information from the current training samples, including the multiple target training samples, wherein, The sample action information sequence includes the first action description information, the second action description information, and multiple frames of transition sample action information; a mirror transformation operation is performed on the multiple action description information in the sample action information sequence to obtain the first action description information sequence; the first action description information sequence is used as the newly added target training sample; Enhancement Method 2: Obtain a sequence of sample action information from the current training samples including multiple target training samples, wherein the sequence of sample action information includes the first action description information, the second action description information, and multiple frames of transition sample action information; perform a velocity transformation operation on the sequence of sample action information to obtain a second action description information sequence; use the second action description information sequence as a newly added target training sample; Enhancement Method 3: Obtain a sequence of sample action information from the current training samples included in the multiple target training samples, wherein the sequence of sample action information includes the first action description information obtained from the first action description subset, the second action description information obtained from the second action description subset, and multiple frames of transition sample action information; use the first action description information, the second action description information, and the multiple frames of transition sample action information as newly added target training samples.

11. The method according to claim 10, wherein performing mirror transformation operations on multiple action description information in the sample action information sequence to obtain a first action description information sequence comprises: Obtain the current sample action information from the multiple sample action information; Perform a first mirror symmetry operation on the target rotation information of the first part included in the current sample action information; Perform a second mirror symmetry operation on the three-dimensional coordinate information of the second part included in the current sample action information; The updated current sample action information is determined as the first action description information included in the first action description information sequence.

12. The method according to claim 10 or claim 11, wherein performing a velocity transformation operation on the sample action information sequence to obtain a second action description information sequence comprises: Obtain the velocity description information corresponding to the velocity transformation operation; The number of second action description information included in the second action description information sequence and the sequence identifier corresponding to each of the multiple action description information in the second action description information sequence are determined based on the speed description information. Based on the velocity description information, determine the weight information between every two adjacent second action description information; Based on the sequence identifiers corresponding to each of the multiple action description information and the weight information between every two adjacent second action description information, interpolation processing is performed on the multiple action description information to obtain the second action description information sequence.

13. The method according to claim 12, wherein performing interpolation processing on the plurality of action description information according to the order identifiers corresponding to each of the plurality of action description information and the weight information between every two adjacent second action description information to obtain the second action description information sequence includes: Multiple action descriptions are subjected to coordinate transformation to obtain multiple sample global action information in the global coordinate system; A linear interpolation operation is performed based on the global position information in the global action information of multiple samples to obtain multiple global position information corresponding to the second action description information sequence; Perform a spherical linear interpolation operation based on the global rotation information in the global action information of multiple samples to obtain multiple global rotation information corresponding to the second action description information sequence; The second action description information sequence is determined based on multiple global position information and multiple global rotation information.

14. A training sample generation apparatus, comprising: The first acquisition unit is used to acquire an action description information set, wherein the action description information set includes multiple action description information, and at least two of the multiple action description information indicate different object actions; The second acquisition unit is configured to acquire first action description information and second action description information from at least one subset of action description information when performing a clustering operation based on the action description information set to determine at least one subset of action description information. The first action description information is used to indicate a first object action, and the second action description information is used to indicate a second object action. The first object action and the second object action are different object actions. The acquisition unit is used to perform an action acquisition operation based on the first action description information and the second action description information to obtain multi-frame transition sample action information, wherein the multi-frame transition sample action information is used to indicate multi-frame transition object actions that match the first object action and the second object action. The determining unit is configured to determine a target training sample based on the first action description information, the second action description information, and the transition sample action information of multiple frames, wherein the target training sample is used to train the transition action prediction model.

15. A computer-readable storage medium comprising a stored program, wherein, When the program is executed, it performs the method described in any one of claims 1 to 13.

16. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

17. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor being configured to perform the method of any one of claims 1 to 13 via the computer program.

Images