Method and apparatus for generating three-dimensional character, and device, storage medium and program product

Abstract

A method for generating a three-dimensional character, which method is executed by an electronic device. The method comprises: acquiring a two-dimensional image including a reference character (S210); converting the two-dimensional image into a canonical pose image, and on the basis of the canonical pose image, generating multi-view images of the reference character, wherein the multi-view images include images of the reference character at a plurality of viewing angles (S220); on the basis of the multi-view images, extracting three-dimensional feature information of the reference character, wherein the three-dimensional feature information comprises color information, three-dimensional geometric information and semantic information, and the semantic information is used for describing the composition layers of the reference character (S230); and on the basis of the three-dimensional feature information, generating a three-dimensional model of the reference character (S240).

Description

Methods, apparatus, devices, storage media, and program products for generating 3D characters

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on February 24, 2025, with application number 202510213371.6, entitled "Method and apparatus, device, storage medium and program product for generating three-dimensional characters", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of artificial intelligence technology, specifically to a method and apparatus for generating three-dimensional characters, electronic devices, computer-readable storage media, and computer program products. Background Technology

[0004] With the rapid development of technologies such as virtual reality and augmented reality, 3D characters are increasingly being used in games, movies, social media platforms, and virtual interactions. Users' demands for 3D characters are no longer limited to preset templates; they want personalized customization based on their own preferences and needs.

[0005] Traditional 3D character generation methods typically rely on predefined model libraries or manual modeling. This approach is not only time-consuming and labor-intensive but also struggles to meet users' demands for highly personalized characters. In recent years, deep learning-based generative models have made significant progress in the field of 3D reconstruction and generation, offering new possibilities for the automated generation of personalized 3D characters. However, they generally suffer from inaccuracies in the generated 3D characters, resulting in poor performance in application scenarios.

[0006] Therefore, how to improve the accuracy of 3D character generation is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] Embodiments of this application provide methods for generating three-dimensional characters, apparatus for generating three-dimensional characters, electronic devices, computer-readable storage media, and computer program products.

[0008] One aspect of this application provides a method for generating a three-dimensional character, executed by an electronic device. The method includes: acquiring a two-dimensional image containing a reference character; converting the two-dimensional image into a standard pose image, and generating a multi-view image of the reference character based on the standard pose image, the multi-view image containing images of the reference character from multiple viewpoints; extracting three-dimensional feature information of the reference character from the multi-view image; the three-dimensional feature information including color information, three-dimensional geometric information, and semantic information, the semantic information being used to describe the compositional hierarchy of the reference character; and generating a three-dimensional model of the reference character based on the three-dimensional feature information.

[0009] In another aspect of this application, an apparatus for generating a three-dimensional character is provided. The apparatus includes: a reference module configured to acquire a two-dimensional image containing a reference character; a conversion module configured to convert the two-dimensional image into a standard pose image and generate a multi-view image of the reference character based on the standard pose image, the multi-view image containing images of the reference character from multiple perspectives; an extraction module configured to extract three-dimensional feature information of the reference character from the multi-view image; the three-dimensional feature information including color information, three-dimensional geometric information, and semantic information, the semantic information being used to describe the compositional hierarchy of the reference character; and a generation module configured to generate a three-dimensional model of the reference character based on the three-dimensional feature information.

[0010] Another aspect of this application provides an electronic device, including: one or more processors; and a memory for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the method for generating three-dimensional characters as described above.

[0011] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor of an electronic device, causes the electronic device to perform the method for generating a three-dimensional character as described above.

[0012] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor of an electronic device, implements the method for generating a three-dimensional character as described above.

[0013] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.

[0015] Figure 1 is a schematic diagram of an exemplary implementation environment related to this application;

[0016] Figure 2 is a flowchart illustrating a method for generating a three-dimensional character according to an exemplary embodiment of this application;

[0017] Figure 3 illustrates the detailed process of training a large-scale reconstruction model with semantic awareness;

[0018] Figure 4 illustrates a simplified flowchart of training a large-scale reconstruction model with semantic awareness.

[0019] Figure 5 illustrates another exemplary flowchart for training to obtain a large-scale reconstruction model with semantic awareness;

[0020] Figure 6 is a schematic diagram of the technical architecture for generating three-dimensional characters, illustrating an exemplary embodiment of this application;

[0021] Figure 7 illustrates the processing flow of the semantic-based hierarchical 3D generation module shown in Figure 6.

[0022] Figure 8 illustrates the flowchart for multi-level detail optimization of the 3D model of the reference character;

[0023] Figure 9 is a block diagram illustrating an apparatus for generating a three-dimensional character according to an exemplary embodiment of this application;

[0024] Figure 10 shows a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0028] In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0029] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0030] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0031] Please refer to Figure 1 first, which is a schematic diagram of an exemplary implementation environment related to this application. This implementation environment illustrates an exemplary three-dimensional character generation system, including a terminal 110 and a server 120, which interact with each other through a pre-established wired or wireless communication connection.

[0032] As shown in Figure 1, terminal 110 supports various user-customized inputs, such as text, images, or sketches, all of which can be used as input information for user-customized 3D characters. After obtaining the user's customized input information, terminal 110 uploads it to server 120. The server 120 then uses an artificial intelligence model configured on it to intelligently generate a 3D character that matches the customized input information. Therefore, the 3D character generation system illustrated in Figure 1 not only achieves customized generation of 3D characters but also has a high degree of customization freedom, thus meeting users' customization needs for 3D characters.

[0033] For example, after receiving customized input information uploaded by terminal 110, if the customized input information is text or a sketch lacking detailed features of the 3D character in terms of appearance, clothing, etc., server 120 needs to first generate a 2D image containing a reference character based on the text or sketch, and then generate the 3D character based on the generated 2D image containing the reference character. If the customized input information is already a 2D image containing a reference character, then the 3D character is generated directly based on this 2D image. First, the 2D image containing the reference character needs to be converted into a standard pose image. Then, multi-view images are generated based on the converted standard pose images, such as generating orthographic projections of six standard viewpoints as multi-view images. Then, 3D feature information such as color information, semantic information, and 3D geometric information is extracted from the multi-view images. Finally, the 3D character is generated based on the extracted 3D feature information.

[0034] Because semantic information enables the accurate extraction of different parts of a reference character, and multi-view images generated based on standard pose images can more accurately represent the 3D geometric information of the 3D character, server 120 can more accurately generate 3D characters based on the extracted 3D feature information. The accuracy of the generated 3D character can be understood as the generated 3D character not only possessing high-quality geometric structure and texture, but also being semantically separable, providing great flexibility for subsequent editing and customization of the 3D character, better meeting the needs of industries such as virtual reality, games, and film for highly customized characters. For example, users can individually customize one or more components of the virtual character, such as hair and clothing, thereby satisfying higher customization requirements.

[0035] It should be noted that terminal 110 can be a computer, smartphone, tablet, laptop, smart home appliance, or other device, and there are no restrictions here. Server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services, and there are no restrictions here either.

[0036] Please refer to Figure 2, which is a flowchart illustrating a method for generating a 3D character according to an exemplary embodiment of this application. This method can be applied to the implementation environment shown in Figure 1, for example, it can be specifically executed by server 120, or jointly executed by terminal 110 and server 120. Of course, this method can also be applied to other implementation environments and executed by servers in other implementation environments, or jointly executed by terminals and servers in other implementation environments; this embodiment does not impose any limitations.

[0037] As shown in Figure 2, in an exemplary embodiment, the method for generating a 3D character includes steps S210-S240, which are described in detail below:

[0038] S210, Obtain a two-dimensional image containing the reference character.

[0039] First, it's important to understand that the 3D characters mentioned in this application refer to virtual characters created using 3D technology. 3D characters commonly appear in fields such as games, virtual reality, and video animation, including player characters, non-player-controlled characters (NPCs), and monster characters in games. Through 3D technology, these characters can present realistic appearances, movements, and expressions, providing viewers or players with a more immersive experience.

[0040] In this embodiment, the 3D character to be generated is based on a 2D image containing a reference character. The reference character can be understood as reference information used as the 3D character to be generated. For example, as shown in Figure 1, the server 120 generates a 3D character based on a 2D image input by the user in the terminal 110. This can be understood as generating a 3D character model corresponding to the 2D character image.

[0041] A reference character typically has at least two components, such as the basic human structure (body), hair, and clothing. This means that the reference character mentioned in this embodiment has at least two semantics, with different semantics representing different levels of character composition. That is, the basic human structure, hair, and clothing can all serve as different semantics for the reference character.

[0042] It should also be noted that this embodiment does not limit the specific method of obtaining the 2D image containing the reference character. For example, the 2D image of the reference character can be directly input by the user, or it can be generated based on reference information such as text or sketches that lack detailed features of the reference character. This can also be understood as the generation of 3D characters relying on relatively sufficient reference information, and this embodiment supports generating 2D images that meet the requirements for 3D character generation based on reference information such as text and sketches.

[0043] S220, converts the two-dimensional image into a standard pose image, and generates a multi-view image of the reference character based on the standard pose image. The multi-view image contains images of the reference character from multiple perspectives.

[0044] A standard pose image refers to a standard pose presented by a reference character in an image; a standard pose can also be called a preset pose. For example, a standard pose can be an A-pose, a T-pose, an idle pose, etc. This embodiment does not limit the specific form of the standard pose. The A-pose is characterized by the model's legs slightly apart in an A-shape, with arms hanging naturally, making it more suitable for scenarios showing the model's natural standing posture. The T-pose is characterized by the model's legs straight and shoulder-width apart, with arms straight and perpendicular to the sides of the body, forming a T-shape, making it more suitable for scenarios where motion capture data is needed. The idle pose is more suitable for scenarios showing a character in a idle or resting state. In practical applications, an appropriate standard pose can be selected for 2D image conversion according to actual needs.

[0045] Converting 2D images into standard pose images allows for the elimination of self-occlusion and other issues when generating multi-view images of the reference character, resulting in more accurate multi-view images. These more accurate multi-view images can then be used to extract more accurate 3D feature information, thereby promoting the generation of more accurate 3D characters.

[0046] For example, the multi-view images of the reference character may include orthographic projection images of the reference character from six standard viewpoints, or orthographic projection images from fewer standard viewpoints, or orthographic projection images from other viewpoints. This embodiment does not limit the specific viewpoints. However, it should be understood that the multi-view images need to contain two-dimensional images of the reference character from at least two different viewpoints to ensure accurate generation of the three-dimensional character. Moreover, within a certain range, the number of viewpoints is usually proportional to the accuracy of the three-dimensional character. The multi-view images may also be perspective projection images, etc., and this embodiment does not limit this as well.

[0047] For example, a diffusion model combined with the ReferenceNet network can be used to convert various customized input information into standard pose images. Other AI models or neural networks with similar functions can also be used, and this is not limited. A model based on the U-Net network, such as a fine-tuned Era3D model, can be used. The input image data is fed into the encoder part of the U-Net network, where high-level features are extracted through convolution and downsampling operations. These features are then fed into the decoder part, where deconvolution and upsampling operations restore the spatial resolution of the image. Simultaneously, skip connections are used to fuse feature information from different levels, ultimately generating multi-view images of the reference character. Alternatively, models based on networks such as Diffusion Transformer (DFT) or GAN (Generative Adversarial Networks) can be used to generate multi-view images of the reference character; this embodiment is also not limited in this regard.

[0048] ReferenceNet is a neural network that works in conjunction with a diffusion model in the process of 3D character generation. It plays a role in transforming various customized input information (such as text, images, sketches, etc.) provided by the user into standard pose images. Its specific network structure and working mechanism rely on feature extraction, transformation, and mapping of the input data to ensure that the output standard pose image provides an accurate and consistent foundation for subsequent generation of multi-view images of the reference character, thereby improving the accuracy and stability of the entire 3D character generation process.

[0049] The Era3D model is based on the U-Net network architecture and, after fine-tuning, is specifically designed to generate multi-view images of reference characters. It receives input image data, performs convolution and downsampling operations on the encoder part of the U-Net network to extract high-level features and capture key information and feature patterns. Then, the decoder part performs deconvolution and upsampling operations on these features to restore the spatial resolution of the image. Simultaneously, it utilizes skip connections to effectively fuse feature information at different levels, enabling the output multi-view images to comprehensively and accurately represent the features of the reference character from multiple perspectives, providing rich and reliable data support for subsequent extraction of the reference character's 3D feature information.

[0050] S230, extract the three-dimensional feature information of the reference character from the multi-view images; the three-dimensional feature information includes color information, three-dimensional geometric information and semantic information, and the semantic information is used to describe the compositional hierarchy of the reference character.

[0051] This embodiment extracts 3D feature information from multi-view images, which includes color information, 3D geometric information, and semantic information. At least two semantics exist for each reference character, and different semantics represent different levels of character composition. The semantic information extracted in this embodiment describes the compositional levels of the reference character. In other words, based on the extracted semantic information, the basic human structure, hair, clothing, and other components of the reference character can be identified. This allows the 3D character generated subsequently based on the extracted 3D feature information to be effectively decomposed according to the semantic information, thus decoupling the various components of the 3D character and enhancing its customization capabilities.

[0052] Moreover, since the three-dimensional feature information is extracted based on the multi-view images generated in S220, the multi-view images can provide more comprehensive character reference information, which makes the accuracy of the three-dimensional feature information extracted from the multi-view images higher, thus enabling the generation of more accurate three-dimensional characters.

[0053] For example, image features can be extracted from multi-view images, and then three-dimensional implicit features based on Triplane (a representation method of three-dimensional space) can be extracted based on these image features. These three-dimensional implicit features can then be decoded into display three-dimensional information such as color, volume density, semantics, neural radiance fields (NeRF), and signed distance field (SDF). Of course, this embodiment does not limit the specific method of extracting three-dimensional feature information; it can be selected based on actual application requirements.

[0054] S240, Based on the three-dimensional feature information of the reference character, generate a three-dimensional model of the reference character.

[0055] As mentioned above, since the three-dimensional feature information contains not only color information and three-dimensional geometric information, but also semantic information, the semantic information can enable the accurate extraction of different components of the reference character. Moreover, the multi-view images generated based on the standard pose image can also enable the acquisition of more accurate three-dimensional geometric information and semantic information. This allows the present application to generate a three-dimensional character, i.e. a three-dimensional model of the reference character, more accurately based on the extracted three-dimensional feature information.

[0056] The 3D character generated in this embodiment can be broken down into various parts such as the basic human body model, hair, and clothing based on semantic information, effectively decoupling the various components of the 3D character and thus enhancing its customization capabilities. For example, because the 3D character can be finely divided into different components, user customization needs can be addressed for each part of the 3D character; that is, each part of the 3D character can be personalized by the user. This will undoubtedly greatly enhance the customization function of the 3D character, thereby meeting users' needs for highly personalized characters.

[0057] It is understandable that, in scenarios where a user's customization needs can target at least one part of a 3D character, the semantic information extracted in S230 corresponds to the specific part of the character customized by the user, and thus affects the generation of the 3D model corresponding to the specific part of the character customized by the user. Therefore, this embodiment does not limit the specific type of semantic information extracted in S230.

[0058] In another exemplary embodiment, a Large Reconstruction Model (LRM) refers to a deep learning model that generates an implicit field with 3D geometric and color information from one or more images. It typically employs a highly scalable Transformer-based architecture, has a large number of parameters and a complex network structure, and can capture and utilize the rich information contained in the input data to generate realistic 3D reconstruction results. S230 If a Large Reconstruction Model is used to extract 3D feature information from multi-view images, the large reconstruction model lacks semantic awareness, which cannot meet the high requirements of accuracy and customization for generating 3D models in the embodiments of this application. Therefore, this embodiment proposes a training scheme for the Large Reconstruction Model, enabling it to possess semantic awareness, thereby applying this semantically aware Large Reconstruction Model (hereinafter referred to as the semantically aware Large Reconstruction Model) to the 3D character generation scenario.

[0059] Please refer to Figures 3 and 4. Figure 3 shows a detailed flowchart of training a large-scale reconstruction model with semantic awareness, and Figure 4 shows a simplified flowchart of training a large-scale reconstruction model with semantic awareness.

[0060] As shown in Figure 3, as an exemplary implementation, the process of training a large-scale reconstruction model with semantic awareness includes steps S310-S330, which are described in detail below:

[0061] S310 aims to learn the neural radiation field of single-layer semantics and performs the first stage of training on the initial large-scale reconstruction model to obtain the large-scale reconstruction model after the first stage of training.

[0062] The initial large-scale reconstruction model can be understood as a large-scale reconstruction model without semantic awareness. Single-layer semantics can be understood as the surface of a three-dimensional character. Neural radiation field representation uses a specified viewpoint and volume rendering method to render two-dimensional images. Volume rendering refers to a rendering method that uses cubic voxels for modeling and projects light into the interior of the object for scattering, thereby constructing the final color.

[0063] The first stage of training learns a sufficiently reliable 3D surface and its semantic information to better guide subsequent stages of training.

[0064] For example, as shown in Figure 5, the first stage of training may include the following steps:

[0065] S311, Obtain the three-dimensional feature information extracted by the initial large-scale reconstruction model from the multi-view images of the training character;

[0066] S312, using the three-dimensional surface of the training character as a single-layer semantic, and utilizing the three-dimensional feature information of the training character, render the neural radiation field of the three-dimensional surface of the training character to obtain the first rendering model.

[0067] S313, Calculate the first training loss value based on the first rendering model and the real 3D model;

[0068] S314, optimize the initial large-scale reconstruction model based on the first training loss value to obtain the large-scale reconstruction model after the first stage of training.

[0069] It should be noted that the training sample set used for model training can include a large number of sample pairs corresponding to training characters, and each sample pair consists of multi-view images and a real 3D model.

[0070] This embodiment first uses an initial large-scale reconstruction model to extract 3D feature information from multi-view images of the training character. Then, the 3D surface of the training character is used as a single-layer semantic layer. The 3D feature information of the training character is used to render the neural radiation field of the 3D surface of the training character, thereby obtaining the first rendering model. Therefore, the first rendering model can also be understood as a 3D character model obtained by volume rendering based on the 3D feature information extracted from the initial large-scale reconstruction model. By comparing the difference between the first rendering model and the real 3D model, the first training loss value can be determined. The initial large-scale reconstruction model is optimized based on the first training loss value, that is, the weights of the initial large-scale reconstruction model are adjusted to minimize the training loss, so that the model rendered using the 3D feature information extracted by the optimized model is closer to the real 3D model. This optimization allows the model to continuously learn reliable 3D surfaces and their semantic information.

[0071] For example, the process of rendering the neural radiation field of any semantic layer using three-dimensional feature information can be represented by the following formula:

[0072] in, The color rendered in semantic s represents the color, N represents the number of sampling points, i represents the i-th sampling point, and a i c represents the transparency of the i-th sampling point. i p represents the color value of the i-th sampling point. s,i T represents the probability that the semantic meaning of the i-th sampling point is s, also known as semantic probability. s,i This represents the cumulative historical transparency value corresponding to the i-th sampling point. The cumulative historical transparency value can also be understood as the cumulative transparency before the i-th sampling point. i-1 represents the most recent sampling point before the i-th sampling point, and j represents the j-th sampling point, which is before the i-th sampling point. j p represents the transparency of the j-th sampling point. s,j This represents the probability that the semantics of the j-th sampling point are s.

[0073] Multiple sampling points can be obtained for each semantic layer, and a set of rendering parameters corresponding to each sampling point can be determined. The set of rendering parameters includes transparency, color value, semantic probability, and historical transparency accumulation value. Then, based on the set of rendering parameters corresponding to each sampling point, the product of the rendering parameters for each sampling point is calculated, and the products of each rendering parameter are accumulated to obtain the color rendered under each semantic layer. Thus, the first rendering model is obtained.

[0074] It should be noted that, in order to accelerate the training process and improve model performance, the embodiments of this application can use LoRA (Low-Rank Adaptation) to perform model optimization at each training stage. The core idea of ​​LoRA is to modify the weights of the pre-trained model by adding a low-rank matrix to adapt to the new task. Since the number of parameters in the low-rank matrix is ​​much smaller than that in the original model, LoRA can significantly reduce the number of parameters that need to be trained, which not only speeds up the training process but also reduces the demand for memory and computing resources.

[0075] In addition, as another exemplary implementation, the initial large-scale reconstruction model can be a large-scale language model or vision model based on deep learning, which has pre-trained weights. The pre-trained weights have been trained on a large-scale dataset, so they contain rich prior knowledge. Using the pre-trained weights as the starting point for training the large-scale reconstruction model can greatly shorten the training time and improve the model's performance on specific tasks.

[0076] As an exemplary implementation, pixel loss, mask loss, semantic loss, and perceptual loss can be calculated between the first rendering model and the real 3D model, respectively. The perceptual loss characterizes the similarity of the images in the feature space. Based on this pixel loss, mask loss, semantic loss, and perceptual loss, the loss sum is calculated to obtain the first training loss value.

[0077] Of course, in other exemplary implementations, the first training loss value may also be obtained from one or more of pixel loss, mask loss, semantic loss and perceptual loss, without being specifically limited here.

[0078] Pixel loss characterizes the difference in pixel values ​​between the first rendered model and the real 3D model. For example, pixel loss can be obtained by calculating the image mean square error. By minimizing pixel loss, the model can learn how to generate a rendered model that is as similar as possible to the real 3D model, which prompts the model to extract more accurate 3D feature information.

[0079] Mask loss characterizes the degree of foreground-background separation between the first-rendered model and the real 3D model. By providing mask labels as supervision information, the model can learn how to accurately predict foreground regions in the model. The specific form of mask loss depends on the specific requirements of the mask prediction task and may be binary cross-entropy loss or other suitable loss functions.

[0080] Semantic loss characterizes the semantic difference between the first-rendered model and the real 3D model; it is also known as semantic cross-entropy and is used to measure the model's performance on semantic classification tasks. It calculates the loss by comparing the difference between the semantic labels predicted by the model and the true labels. Minimizing semantic loss can improve the model's ability to understand semantic content.

[0081] Perceptual loss characterizes the similarity between the first-rendered model and the real 3D model in the feature space. For example, the similarity metric can be implemented using LPIPS (Learned Perceptual Image Patch Similarity), which calculates the perceptual differences between images by comparing feature maps in a pre-trained convolutional neural network. Minimizing perceptual loss can improve the model's image perception quality, making images appear more natural and realistic.

[0082] For example, the formula for calculating the first training loss value can be expressed as follows: L1 = L mse +λ lpips L lpips +λ mask L mask +λ sem L sem

[0083] Where L1 represents the first training loss value, L mse L represents pixel loss. lpips λ represents the perceived loss. lpips L represents the weighting coefficient corresponding to the perceptual loss. mask λ represents the masking loss. mask L represents the weight coefficients corresponding to the masking loss. sem λ represents semantic loss. sem These represent the weight coefficients corresponding to the semantic loss. These weight coefficients are adjusted based on the specific task and dataset to optimize the overall performance of the model; no specific restrictions are placed on their values ​​here.

[0084] Therefore, by employing multiple loss functions to calculate the first training loss value, this embodiment can ensure that large-scale reconstruction models can accurately capture and reconstruct 3D scenes and their semantic content.

[0085] S320 aims to learn the neural radiation field of multi-layer semantics. It performs a second-stage training on the large-scale reconstruction model after the first-stage training to obtain the large-scale reconstruction model after the second-stage training.

[0086] Multi-layer semantics is the opposite of the aforementioned single-layer semantics. Single-layer semantics refers to treating the surface of a 3D character as a single layer of semantics, that is, viewing the surface of a 3D character as a whole. Multi-layer semantics, on the other hand, refers to specifically distinguishing the constituent layers of a 3D character, corresponding to the multiple semantics possessed by a 3D character.

[0087] The second stage of training aims to guide the generation of semantic, geometric, and color information within an object by masking a random number of semantic elements and their corresponding 3D information components. This enables large-scale reconstruction models to more accurately reconstruct neural radiation field models with multi-layered semantic information, thereby achieving more refined 3D scene reconstruction and semantic understanding.

[0088] Referring again to Figure 5, for example, the second-stage training process may include the following steps:

[0089] S321, determine the multi-layer semantics of the training character, and render the neural radiation field under the multi-layer semantics by masking a random number of semantics and the corresponding three-dimensional feature information to obtain the second rendering model.

[0090] S322, calculate the second training loss value based on the second rendering model and the real 3D model;

[0091] S323, optimize the large reconstruction model after the first stage of training based on the second training loss value to obtain the large reconstruction model after the second stage of training.

[0092] In this process, the multi-layered semantics of the training character can be determined based on the labeled information of the training samples. By masking a random number of semantic elements and their corresponding 3D feature information, a neural radiation field under multi-layered semantics is rendered. Then, a second training loss value is calculated based on the difference between the obtained second rendering model and the real 3D model. The large-scale reconstruction model after the first stage of training is then optimized based on the second training loss value. The optimization objective can be summarized as minimizing the training loss by guiding the generation of semantic, geometric, and color information within the object, thereby obtaining the large-scale reconstruction model after the second stage of training. Thus, the large-scale reconstruction model after the second stage of training can accurately reconstruct the neural radiation field model of multi-layered semantic information, thereby achieving more refined 3D scene reconstruction and semantic understanding.

[0093] The second stage of training can be understood as an extension based on the first stage of training. Therefore, the calculation method of the second training loss value is similar to that of the first training loss value, but the scope of each loss function is changed from single-layer semantics to multi-layer semantics to adapt to the needs of multi-layer semantics.

[0094] For example, for each layer of semantics contained in the unmasked semantic set, the corresponding pixel loss, perceptual loss, masking loss and semantic loss are calculated respectively. Then, based on the pixel loss, perceptual loss, masking loss and semantic loss corresponding to each layer of semantics, the sum of the multi-layer semantic loss is calculated to obtain the second training loss value.

[0095] The second training loss value can be calculated using the following formula: L2 = L mse,P +λ lpips L lpips,P +λ mask L mask,P +λ sem L sem,P

[0096] Where L2 represents the second training loss value, P represents the unmasked semantic set, and L mse,P L represents the pixel loss in the second training phase. lpips,P L represents the perceptual loss in the second training phase. mask,P L represents the mask loss during the second training phase. sem,P λ represents the semantic loss in the second training phase. lpips , λ mask , λ sem These represent the weight coefficients for perceptual loss, masking loss, and semantic loss, respectively. These weight coefficients are adjusted based on the specific task and dataset to optimize the overall performance of the model, and no specific restrictions are imposed on their values ​​here.

[0097] It should also be noted that the process of rendering the neural radiation field under multi-layer semantics is the same as the method of rendering the neural radiation field for single-layer semantics in the first training stage, and will not be elaborated here.

[0098] S330 aims to learn the signed distance field of multi-layer semantics and performs a third-stage training on the large-scale reconstruction model after the second-stage training to obtain the large-scale reconstruction model after the third-stage training.

[0099] The signed distance field represents the signed distance from a sampling point in 3D space to the nearest surface. In the third training stage, by adding the Flexi Cubes module after the large reconstruction model, higher resolution supervision and more practical 3D mesh output can be achieved, thereby enhancing the reconstruction capabilities of the large reconstruction model. A mesh is a discretized structure used to represent the surface of a 3D object, consisting of a series of interconnected polygons (usually triangles or quadrilaterals). In the 3D character generation scenario of this application, 3D mesh information is obtained by transforming the equivalent signed distance field corresponding to the semantics of each layer. This mesh information can accurately describe the geometry of the 3D character, providing a foundation for subsequent rendering and visualization. In the third training stage of the large reconstruction model, by adding the Flexi Cubes module, higher resolution supervision and more practical 3D mesh output can be achieved, thereby enhancing the reconstruction capabilities of the large reconstruction model for 3D characters.

[0100] For example, still referring to Figure 5, the third stage of training includes the following steps:

[0101] S331, extract the equivalent signed distance field corresponding to each semantic layer based on the three-dimensional feature information of the training character, and convert the equivalent signed distance field corresponding to each semantic layer into three-dimensional mesh information respectively;

[0102] S332, calculates the third training loss value based on 3D mesh information and real mesh information;

[0103] S333 optimizes the large-scale reconstruction model after the second stage of training based on the third training loss value to obtain the large-scale reconstruction model after the third stage of training.

[0104] In the above process, it is important to understand that the equivalent signed distance field representation extracts the distance field of a specific semantic, thereby more effectively distinguishing surfaces with different semantics and making the identification of role components based on semantic information more accurate.

[0105] For example, for each sampling point under each semantic layer, the maximum probability among the semantic probabilities corresponding to other semantics besides the current semantic can be determined, and the probability difference between the maximum probability and the semantic probability corresponding to the current semantic can be calculated. Then, the larger value is selected from the original signed distance field value and the probability difference corresponding to each sampling point as the equivalent signed distance field value of the corresponding sampling point.

[0106] The formula for extracting the equivalent signed distance field can be expressed as follows: f i,s =max(f i ,(max r≠s p i,r )-p i,s )

[0107] Among them, f i,s f represents the equivalent signed distance field value of the i-th sampling point under semantic s. i f represents the original signed distance field value of the i-th sampling point. If the i-th sampling point is inside the model, f i The sign of f is negative if the i-th sampling point is outside the model. i The sign of f is positive if the i-th sampling point is the model surface. i The value is 0, and this original signed distance field value is determined from the three-dimensional feature information. r≠s p i,r It represents the maximum probability among all semantic probabilities except semantic s at the i-th sampling point. It can be interpreted as the probability of the strongest semantic competing with semantic s at the i-th sampling point.

[0108] As can be seen, the above formula aims to compare the original signed distance field values ​​f. i A difference value based on semantic probability (max) r≠s p i,r )-p i,s This difference value represents the maximum probability advantage of other semantics compared to semantic s at the i-th sampling point. If the absolute value of the original signed distance field is greater than this difference value, then the equivalent signed distance field value f is... i,s Keep the original range field value; otherwise, use the equivalent signed range field value f. i,s It is set to this difference value.

[0109] This can be understood as follows: by comparing the distances of the i-th sampling point to different surfaces, it is ensured that the distance of the i-th sampling point to semantic s reflects its relative position with respect to other semantic surfaces. If the i-th sampling point is closer to the surface of semantic s, then f i,s It will be a small positive or negative value; if the i-th sampling point is closer to the surface of other semantics, then f i,sIt will be a large positive value, thus effectively distinguishing surfaces with different semantic meanings.

[0110] The loss function for the third stage of training can include the loss term inherited from the second stage, as well as newly added depth loss, normal vector loss, and standardized loss. First, the depth loss, normal vector loss, and standardized loss are calculated based on the 3D mesh information and the real mesh information, respectively. Then, the loss value obtained from the second stage of training is calculated, and the sum of the depth loss, normal vector loss, and standardized loss is obtained as the third training loss value.

[0111] Depth loss measures the difference between the depth values ​​predicted by the model and the true depth values. Depth information is crucial in 3D reconstruction tasks because it provides accurate information about the distance of objects in the scene from the observer. Depth loss is typically derived by calculating some distance metric between the predicted and true depth maps, such as the L1 or L2 loss function. By minimizing the depth loss, the model can more accurately estimate the depth information in the scene, thereby improving the quality of 3D reconstruction.

[0112] Normal vector loss is used to evaluate the difference between the model's predicted normal vector and the true normal vector. A normal vector is a vector perpendicular to the surface of an object, and normal vector loss captures the model's error in estimating surface orientation. This loss is typically derived by calculating the angular difference or cosine similarity loss between the predicted and true normal vectors. Minimizing normal vector loss helps the model generate smoother and more accurate surfaces.

[0113] Standardization loss measures the difference between the object shape or size predicted by the model and a certain standardized or desired shape. This loss can help ensure that the objects generated by the model maintain consistency and accuracy in size and shape.

[0114] For example, the third training loss value can be calculated using the following formula: L3 = L2 + λ normal L normal +λ depth L depth +λ dev L dev

[0115] Where L2 represents the loss value obtained in the second stage of training, λ normal L represents the weight parameters corresponding to the normal vector loss. normal λ represents the normal vector loss. depth L represents the weight parameters corresponding to the depth loss. depth Denotes depth loss, λ dev L represents the weight parameters corresponding to the standardized loss. dev This represents the standardization loss.

[0116] In some exemplary embodiments, when the reference character contains target semantics prone to rendering incompleteness, such as clothing semantics which are prone to errors like holes or incompleteness, affecting the overall reconstruction quality, a regularization loss term specifically for the target semantics can be added to the calculation formula of the third training loss value. This is specifically used to constrain the reconstruction quality of specific semantic categories such as clothing, preventing errors like holes or incompleteness from occurring.

[0117] For example, the regularization loss can be calculated using the following formula:

[0118] Among them, L reg Indicates regular loss. f represents the set of sampling points corresponding to the target semantics (semantics s in this example). a,s and f b,s All are sets The element in f a,s f represents the equivalent signed distance field value of the a-th sampling point under semantic s. b,s This represents the equivalent signed distance field value of the b-th sampling point under semantic s, where sampling points a and b are adjacent but have different signs, σ represents the sigmoid function, and sign(f b,s ) indicates extracting f b,s The symbol, where sign(f) b,s )<0 means that only f with a negative sign is considered. b,s H represents the cross-entropy function. Minimizing the regularization loss can prevent frequent sign changes in the sign field, thereby preventing fragmentation.

[0119] The cross-entropy function is a function used to measure the difference between two probability distributions. In this application, the cross-entropy function is used when calculating the regularization loss. By comparing the probability distributions corresponding to information such as the equivalent signed distance field values ​​of different sampling points, the degree of difference between the model's prediction and the expected result is measured. Minimizing the cross-entropy loss can make the model's prediction results closer to the reality, which helps to improve the model's reconstruction quality of specific semantics (such as target semantics that are prone to incomplete rendering) and prevents erroneous outputs such as holes and incompleteness.

[0120] Therefore, through the three-stage training of the large-scale reconstruction model provided above, the large-scale reconstruction model can effectively learn multi-layer semantic information, thereby enabling the generation of high-quality 3D character models with detailed semantic decomposition, and providing flexible user-customized applications in fields such as games, virtual reality, and film and television production.

[0121] It should also be noted that, as described above in the three-stage model training process, the large-scale reconstruction model trained in the second stage already possesses multi-layer semantic awareness capabilities. However, its accuracy in perceiving semantic boundaries is weaker than that of the large-scale reconstruction model trained in the third stage. Therefore, in some exemplary embodiments, a semantically aware large-scale reconstruction model can be obtained solely through the first two stages of training, i.e., the first and second stages of training. In other words, the large-scale reconstruction model trained in the second stage can be used as the semantically aware large-scale reconstruction model. However, in application scenarios with higher requirements for the quality of 3D character generation, a three-stage training process can be chosen to obtain the semantically aware large-scale reconstruction model; that is, the large-scale reconstruction model trained in the third stage can be used as the semantically aware large-scale reconstruction model.

[0122] In other exemplary embodiments, during the model inference phase, that is, during the process of extracting the 3D feature information of the reference character from multi-view images using a large-scale reconstruction model with semantic awareness, the sampling points under a specified semantic can be divided into sampling points that need to be thickened and sampling points that do not need to be thickened. The original signed distance field value of the sampling points that need to be thickened is updated to the equivalent signed distance field value, while keeping the original signed distance field value of the sampling points that do not need to be thickened unchanged, so as to obtain the 3D feature information of the reference character.

[0123] The corresponding formula can be expressed as follows:

[0124] Where, d i,surface denoted as the distance from the i-th sampling point to the nearest surface, ∈ represents the distance threshold used to distinguish between inner and outer regions, and di,surface>∈ means that when the i-th sampling point is outside the surface, i.e. in the outer region, its original signed distance field value needs to be adjusted to the equivalent signed distance field value, while when it is inside the surface, i.e. in the inner region, its original signed distance field value remains unchanged.

[0125] To put it simply, thickening is typically used to prevent the breakage or loss of fine structures (such as strands of hair or edges of clothing) during 3D reconstruction. By increasing the "thickness" of these areas, it ensures that these structures are preserved intact in subsequent processing (such as 3D mesh generation), thereby achieving more refined 3D scene reconstruction. Therefore, by thickening specific semantic elements, such as hair or clothing—thin and complex-shaped parts—the integrity of these parts in 3D reconstruction can be enhanced.

[0126] Figure 6 is a schematic diagram of the technical architecture for generating 3D characters according to an exemplary embodiment of this application. As shown in Figure 6, the technical solution for generating 3D characters in this application can be summarized as mainly including a multi-view standardized character generation module, a semantically based hierarchical 3D generation module, and a multi-level optimization module guided by various prior information.

[0127] The multi-view standardized character generation module generates standardized characters from user-specified inputs such as reference text, images, and sketches, providing unified basic viewpoint information for subsequent 3D reconstruction. The semantic-based hierarchical 3D generation module innovatively introduces a semantic field into large-scale reconstruction models, enabling semantic-based hierarchical generation. This effectively decouples the various components of the 3D character (such as body, clothing, and hairstyle), enhancing customization capabilities. A multi-level optimization module guided by various prior information improves the geometry of the 3D character and provides finer texture details.

[0128] The multi-view standardized character generation module takes user-provided text or a single reference character image / sketch as input to generate a set of multi-view images and corresponding normal maps of the 3D character. This can be divided into two steps: converting the user-customized input into an A-pose character, and generating multi-view images and normal maps from the A-pose representation. For example, an enhanced diffusion model combined with the ReferenceNet network can be used to convert various user-customized inputs into A-pose images. These images are then fed into a fine-tuned Era3D model to generate orthogonal projection images and normal maps of six standard views, providing support for subsequent large-scale semantic-aware reconstruction models. To generate multi-view images and normal maps from the A-pose representation, the A-pose image can be fed into a fine-tuned Era3D model. This model uses the U-Net network as a base, and the input A-pose image data is fed into the encoder part of the U-Net network. After convolution and downsampling operations, the high-level features of the image are extracted. These features are then fed into the decoder part, and after deconvolution and upsampling operations, the spatial resolution of the image is restored. At the same time, skip connections are combined to fuse feature information at different levels, and finally, orthogonal projection images and normal maps of six standard views are generated.

[0129] Figure 7 illustrates the processing flow of the semantic-based hierarchical 3D generation module shown in Figure 6. As shown in Figure 7, the semantic-based hierarchical 3D generation module extracts image features from the orthogonal projection images of the six standard viewpoints in the previous stage and inputs them into a semantically aware large-scale reconstruction model to obtain 3D implicit features based on Triplane. These features are further decoded into explicit 3D information such as color, volume density, semantics, neural radiation field, and signed distance field. The training process of the semantically aware large-scale reconstruction model has been described in detail in the aforementioned embodiments and will not be repeated here.

[0130] The multi-level optimization module, guided by multiple prior information sources, adopts a phased and multi-level approach. It iteratively optimizes the generated layered 3D character results using the normal map generated by the multi-view standardized character generation module and multi-view consistency constraints. In each iteration, the error between the current 3D character result and the normal map and multi-view consistency constraints can be calculated. The error calculation formula is as follows: Where E represents the error, n represents the number of samples involved in the calculation, and w i d represents the weight of the i-th sample. i This represents the difference measure between the i-th sample and the constraints. Then, the geometric structure and texture parameters of the 3D character are adjusted based on the error, using the following formula: Where P new P represents the adjusted parameter. old This represents the parameters before adjustment, where α represents the learning rate. This represents the gradient of the error. The above process is repeated until the error meets a preset threshold or the maximum number of iterations is reached.

[0131] Multi-view consistency constraint refers to a constraint condition in the 3D character generation process that requires the generated 3D character to exhibit consistent behavior across different viewpoints. In a multi-level optimization module guided by various prior information, this constraint is compared with the generated layered 3D character results to calculate errors, ensuring that the character's geometric structure and texture features remain consistent and accurate across all viewpoints. This avoids contradictions or mismatches in behavior across different viewpoints, thereby improving the quality and realism of the generated 3D character.

[0132] Figure 8 illustrates the flowchart for multi-level detail optimization of the 3D model of the reference character, including the following steps:

[0133] S810, extract different parts of the reference role based on the specified different semantics;

[0134] S820 focuses on optimizing the basic human body structure.

[0135] S830, which overlays clothing onto an optimized basic human body structure;

[0136] S840, with a fixed basic human body model, only the clothing part is optimized.

[0137] S850, add hair section, and fix the optimized basic human body structure section and clothing section;

[0138] The S860 uses models such as SAM to accurately identify hair regions in order to guide the hair optimization process.

[0139] The SAM model is a powerful image segmentation model that can quickly and accurately segment hair regions by providing on-image prompts (such as clicking, selecting, etc.) and leveraging its strong pre-trained generalization capabilities. Specifically, on the image corresponding to the 3D model, users can click on the general area of ​​hair or select a portion of hair. The SAM model will then generate a precise mask for the hair region based on its learned features and segmentation strategies, thereby guiding the subsequent optimization process for the hair segment.

[0140] Specifically, S820 ensures the accuracy and smoothness of the basic human body structure, S840 ensures the fit and detail of the clothing, and S860 ensures optimal geometric accuracy and texture detail in the hair area. Thus, through the optimization process shown above, a customized layered 3D character model with high geometric and texture accuracy in each semantic layer can be obtained.

[0141] Please refer to Figure 9, which is a block diagram illustrating an apparatus for generating a 3D character according to an exemplary embodiment of this application. This apparatus can be applied to the implementation environment shown in Figure 1, for example, it can be specifically configured on server 120, or jointly configured on terminal 110 and server 120. Of course, this apparatus can also be applied to other implementation environments and specifically configured on servers in other implementation environments, or jointly configured on terminals and servers in other implementation environments; this embodiment does not impose limitations.

[0142] As shown in Figure 9, in an exemplary embodiment, the apparatus for generating a 3D character includes:

[0143] Reference module 910 is configured to acquire a two-dimensional image containing a reference character;

[0144] The conversion module 920 is configured to convert a two-dimensional image into a standard pose image and generate a multi-view image of a reference character based on the standard pose image;

[0145] The extraction module 930 is configured to extract three-dimensional feature information of a reference character from multi-view images; the three-dimensional feature information includes color information, three-dimensional geometric information and semantic information, and the semantic information is used to describe the compositional hierarchy of the reference character;

[0146] The generation module 940 is configured to generate a 3D model of the reference character based on 3D feature information.

[0147] In another exemplary embodiment, the 3D feature information of the reference character is extracted from multi-view images using a large-scale semantically aware reconstruction model, and the apparatus further includes:

[0148] The first training module is configured to train the initial large-scale reconstruction model in the first stage with the goal of learning the neural radiation field of single-layer semantics, and obtain the large-scale reconstruction model after the first stage of training; the neural radiation field representation is rendered in two-dimensional image using a specified viewpoint and volume rendering method.

[0149] The second training module is configured to train the large-scale reconstruction model after the first stage of training with the goal of learning the neural radiation field of multi-layer semantics, so as to obtain the large-scale reconstruction model after the second stage of training.

[0150] In this embodiment, the large-scale reconstruction model trained in the second stage is used as a semantically aware large-scale reconstruction model.

[0151] In another exemplary embodiment, the device further includes:

[0152] The third training module is configured to train the large-scale reconstruction model after the second stage of training with the goal of learning the signed distance field of multi-layer semantics, so as to obtain the large-scale reconstruction model after the third stage of training. The signed distance field represents the signed distance from the sampling point in three-dimensional space to the nearest surface.

[0153] In this embodiment, the large-scale reconstruction model trained in the third stage is used as a semantically aware large-scale reconstruction model.

[0154] In another exemplary embodiment, the first training module further includes:

[0155] The information acquisition submodule is configured to acquire the three-dimensional feature information extracted by the initial large-scale reconstruction model from multi-view images of the training character;

[0156] The single-layer semantic rendering submodule is configured to use the three-dimensional surface of the training character as a single-layer semantic field, and use the three-dimensional feature information of the training character to render the neural radiation field of the three-dimensional surface of the training character to obtain the first rendering model.

[0157] The first loss calculation submodule is configured to calculate the first training loss value based on the first rendered model and the real 3D model.

[0158] The first model optimization submodule is configured to optimize the initial large reconstruction model based on the first training loss value to obtain the large reconstruction model after the first stage of training.

[0159] In another exemplary embodiment, the first loss calculation submodule is further configured to perform the following steps:

[0160] Calculate the pixel loss, mask loss, semantic loss, and perceptual loss between the first rendering model and the real 3D model respectively. The perceptual loss represents the similarity in the feature space.

[0161] The sum of losses is calculated based on pixel loss, mask loss, semantic loss, and perceptual loss to obtain the first training loss value.

[0162] In another exemplary embodiment, the second training module further includes:

[0163] The multi-layer semantic rendering submodule is configured to determine the multi-layer semantics of the training character, and render the neural radiation field under the multi-layer semantics by masking a random number of semantics and the corresponding three-dimensional feature information to obtain the second rendering model.

[0164] The second loss calculation submodule is configured to calculate the second training loss value based on the second rendering model and the real 3D model.

[0165] The second model optimization submodule is configured to optimize the large reconstruction model after the first stage of training based on the second training loss value, so as to obtain the large reconstruction model after the second stage of training.

[0166] In another exemplary embodiment, the second loss calculation submodule is further configured to perform the following steps:

[0167] For each layer of semantics contained in the unmasked semantic set, calculate the corresponding pixel loss, perceptual loss, masking loss and semantic loss respectively;

[0168] Based on the pixel loss, perceptual loss, mask loss and semantic loss corresponding to each semantic layer, the sum of the semantic losses of the multiple layers is calculated to obtain the second training loss value.

[0169] In another exemplary embodiment, both the single-layer semantic rendering submodule and the multi-layer rendering submodule are configured to implement the rendering process using the following steps:

[0170] For each semantic layer, multiple sampling points are obtained, and the set of rendering parameters corresponding to each sampling point is determined; the set of rendering parameters includes transparency, color value, semantic probability, and historical transparency cumulative value;

[0171] Based on the set of rendering parameters corresponding to each sampling point, the product of the rendering parameters for each sampling point is calculated, and the products of each rendering parameter are accumulated to obtain the color rendered under each semantic layer.

[0172] In another exemplary embodiment, the third training module further includes:

[0173] The mesh transformation submodule is configured to extract the equivalent signed distance field corresponding to each semantic layer based on the 3D feature information of the training character, and convert the equivalent signed distance field corresponding to each semantic layer into 3D mesh information respectively;

[0174] The third loss calculation submodule is configured to calculate the third training loss value based on 3D mesh information and real mesh information;

[0175] The third model optimization submodule is configured to optimize the large reconstruction model after the second stage of training based on the third training loss value, so as to obtain the large reconstruction model after the third stage of training.

[0176] In another exemplary embodiment, the mesh transformation submodule is further configured to perform the following steps:

[0177] For each sampling point under each semantic layer, determine the maximum probability among the semantic probabilities corresponding to other semantics besides the current semantic, and calculate the probability difference between the maximum probability and the semantic probability corresponding to the current semantic.

[0178] The larger value is selected from the original signed distance field value and probability difference corresponding to each sampling point to be used as the equivalent signed distance field value of the corresponding sampling point.

[0179] In another exemplary embodiment, the third loss calculation submodule is further configured to perform the following steps:

[0180] The depth loss, normal vector loss, and normalization loss are calculated based on 3D mesh information and real mesh information, respectively.

[0181] The loss value obtained from the second stage of training is summed with the depth loss, normal vector loss, and normalized loss to form the third training loss value.

[0182] In another exemplary embodiment, the third loss calculation submodule is further configured to perform the following steps:

[0183] If the reference character contains target semantics, a regularization loss is also calculated based on 3D mesh information and real mesh information. The loss value obtained in the second stage of training is then combined with the depth loss, normal vector loss, normalization loss, and regularization loss to form the third training loss value.

[0184] In another exemplary embodiment, the device further includes a feature information processing module configured to perform the following steps:

[0185] In the process of extracting the 3D feature information of the reference character from multi-view images using a large-scale reconstruction model with semantic awareness, the specified semantic downsampling points are divided into sampling points that need to be thickened and sampling points that do not need to be thickened.

[0186] The original signed distance field values ​​of the sampling points that need to be thickened are updated to equivalent signed distance field values, while the original signed distance field values ​​of the sampling points that do not need to be thickened remain unchanged, in order to obtain the three-dimensional feature information of the reference character.

[0187] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific manner in which each module and unit performs its operation has been described in detail in the method embodiments, and will not be repeated here. In practical applications, the apparatus for generating three-dimensional characters or the video processing apparatus based on artificial intelligence provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation here.

[0188] Embodiments of this application also provide an electronic device, including: one or more processors; and a memory for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the method for generating three-dimensional characters provided in the above embodiments.

[0189] Figure 10 shows a schematic diagram of a computer system suitable for implementing the electronic device of the present application embodiments. It should be noted that the electronic device may be the terminal 110 or server 120 in the implementation environment shown in Figure 1, or it may be a terminal or server in other implementation environments; no limitation is imposed here. It should also be noted that the computer system 1000 of the electronic device shown in Figure 10 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present application.

[0190] As shown in Figure 10, the computer system 1000 includes a Central Processing Unit (CPU) 1001, which can perform various appropriate actions and processes based on computer programs stored in Read-Only Memory (ROM) 1002 or loaded from storage portion 1008 into Random Access Memory (RAM) 1003, such as performing the methods described in the above embodiments. The RAM 1003 also stores various computer programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An Input / Output (I / O) interface 1005 is also connected to the bus 1004.

[0191] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1010 as needed so that computer programs read from them can be installed into storage section 1008 as needed.

[0192] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs various functions defined in the system of this application.

[0193] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. Computer programs contained on computer-readable media can be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0194] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0195] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0196] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor of an electronic device, implements the method for generating a three-dimensional character as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not incorporated into the electronic device.

[0197] Another aspect of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the method for generating a three-dimensional character provided in the various embodiments described above.

[0198] In summary, this application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for generating 3D characters. By acquiring a 2D image containing a reference character, it provides foundational data for subsequent 3D character generation. Converting the 2D image into a standard pose image eliminates interference from the reference character's pose within the image, avoiding the impact of self-occlusion and other issues on subsequent processing. This results in more accurate multi-view images of the reference character generated from the standard pose image, providing a more reliable basis for accurately extracting 3D feature information. The multi-view image contains images of the reference character from multiple perspectives, providing comprehensive character information from different angles and increasing information completeness. 3D feature information, including color, 3D geometric, and semantic information, is extracted from the multi-view image. The semantic information describes the hierarchical structure of the reference character, helping to accurately identify different parts of the character, enabling effective 3D character segmentation, and enhancing the customization capabilities of the 3D character. The 3D feature information extracted from accurate multi-view images is more accurate, thus enabling more accurate generation of the 3D model of the reference character, improving the accuracy and quality of model generation.

[0199] Furthermore, the 3D feature information of the reference character is extracted from multi-view images using a large-scale reconstruction model with semantic awareness. The model undergoes a first-stage training process targeting a neural radiation field that learns single-layer semantics. This neural radiation field utilizes a specified viewpoint and volume rendering method for 2D image rendering, enabling the model to learn the semantic information of the 3D character's surface, laying the foundation for subsequent multi-layer semantic learning. A second-stage training process then targets a neural radiation field that learns multi-layer semantics, allowing the model to learn multi-layer semantic information. This second-stage trained large-scale reconstruction model serves as the semantically aware large-scale reconstruction model. This phased training approach gradually improves the model's ability to learn and process semantic information at different levels, enabling it to more accurately capture and understand the semantic features of the 3D character, thereby extracting 3D feature information more precisely and enhancing the model's semantic awareness and information extraction capabilities.

[0200] When training a semantically aware large-scale reconstruction model, the goal is to learn a multi-layered semantic signed distance field. A third training stage is performed on the large-scale reconstruction model after the second stage of training to obtain a third-stage trained large-scale reconstruction model, which serves as the semantically aware large-scale reconstruction model. The signed distance field represents the signed distance from a sampling point in 3D space to the nearest surface. Through the third-stage training, a Flexi Cubes module is added after the large-scale reconstruction model, achieving higher-resolution supervision and a more practical 3D mesh output. The 3D mesh information can accurately describe the geometry of the 3D character, providing a foundation for subsequent rendering and visualization, enhancing the large-scale reconstruction model's ability to reconstruct 3D characters, and making the generated 3D character model more geometrically accurate.

[0201] In the first stage of training the initial large-scale reconstruction model, the 3D feature information extracted by the initial large-scale reconstruction model from multi-view images of the training character is obtained. The 3D surface of the training character is used as a single-layer semantic layer. The neural radiation field of the 3D surface of the training character is rendered using the 3D feature information of the training character to obtain the first rendering model. The first training loss value is calculated by comparing the first rendering model with the real 3D model. Based on the first training loss value, the initial large-scale reconstruction model is optimized to obtain the large-scale reconstruction model after the first stage of training. This method enables the model to continuously learn reliable 3D surfaces and their semantic information. By adjusting the model weights to minimize the training loss, the 3D feature information extracted by the model is made closer to the real situation, improving the model's ability to capture and reconstruct the semantic information of 3D surfaces.

[0202] Furthermore, when calculating the first training loss value, the pixel loss, mask loss, semantic loss, and perceptual loss are calculated separately between the first rendering model and the real 3D model. Pixel loss measures the difference in pixel values ​​between the first rendering model and the real 3D model. For example, by calculating the mean square error of the image, the model learns to generate a rendering model with pixel values ​​closer to the real 3D model, improving the model's accuracy in extracting image pixel information. Mask loss measures the degree of foreground and background separation between the first rendering model and the real 3D model. By providing mask labels as supervision information, the model can accurately predict foreground regions, improving the model's ability to process foreground and background information in the image. Semantic loss measures the semantic difference between the first rendering model and the real 3D model. By minimizing the semantic loss, the model's ability to understand semantic content is enhanced. Perceptual loss measures the similarity between the first rendering model and the real 3D model in the feature space. For example, by using LPIPS technology, minimizing the perceptual loss improves the model's image perception quality, making image features more natural and realistic. Based on these losses, the first training loss value is calculated, and the model is optimized from multiple dimensions to ensure that the model can accurately capture and reconstruct the 3D scene and its semantic content.

[0203] In the second stage of training the large-scale reconstruction model after the first stage of training, the multi-layer semantics of the training role are determined. By masking a random number of semantic elements and their corresponding 3D feature information, a neural radiation field under multi-layer semantics is rendered to obtain a second rendering model. A second training loss value is calculated based on the second rendering model and the real 3D model. This second training loss value is then used to optimize the large-scale reconstruction model after the first stage of training, resulting in the large-scale reconstruction model after the second stage of training. This training method simulates the model's processing ability under conditions of partial information loss, enabling the model to learn multi-layer semantic information more flexibly, improving its ability to learn and reconstruct multi-layer semantic information, and enhancing its robustness and adaptability.

[0204] Furthermore, when calculating the second training loss value, for each layer of semantics contained in the unmasked semantic set, the corresponding pixel loss, perceptual loss, masking loss, and semantic loss are calculated separately. Based on these losses corresponding to each layer of semantics, the sum of the multi-layer semantic losses is calculated to obtain the second training loss value. This calculation method comprehensively considers different losses from each layer of semantics, which can more comprehensively measure the model's ability to process multi-layer semantic information, effectively optimize the model's performance in multi-layer semantic scenarios, and improve the accuracy and stability of the model's processing of multi-layer semantic information.

[0205] When rendering the neural radiation field under each semantic layer using the 3D feature information of the trained character, multiple sampling points are acquired for each semantic layer. A set of rendering parameters, including transparency, color value, semantic probability, and historical transparency accumulation, is determined for each sampling point. The product of the rendering parameters for each sampling point is calculated based on this set, and the products are summed to obtain the color rendered under each semantic layer. This rendering method, based on the rendering parameters of multiple sampling points, can more accurately generate the color under each semantic layer according to the 3D feature information, improving the accuracy and realism of the rendering results and providing more reliable color information for generating accurate 3D character models.

[0206] In the third stage of training the large-scale reconstruction model after the second stage of training, the equivalent signed distance field corresponding to each semantic layer is extracted based on the 3D feature information of the training character. This equivalent signed distance field is then converted into 3D mesh information. By comparing the original signed distance field values ​​with the difference values ​​based on semantic probabilities, the equivalent signed distance field effectively distinguishes surfaces with different semantic meanings, making the identification of character components based on semantic information more accurate. A third training loss value is calculated based on the 3D mesh information and the real mesh information. This third training loss value is then used to optimize the large-scale reconstruction model after the second stage of training, resulting in the large-scale reconstruction model after the third stage of training. By calculating the training loss value and optimizing the model, the ability of the large-scale reconstruction model to reconstruct the geometric structure of 3D characters is enhanced, improving the accuracy of the 3D model in terms of geometric shape.

[0207] Furthermore, when extracting the equivalent signed distance field corresponding to each semantic layer and converting it into 3D mesh information, for each sampling point under each semantic layer, the maximum probability among the semantic probabilities corresponding to other semantics besides the current semantic is determined. The probability difference between the maximum probability and the semantic probability corresponding to the current semantic is calculated. The larger value is selected from the original signed distance field value and the probability difference corresponding to each sampling point as the equivalent signed distance field value of the corresponding sampling point. This calculation method considers the competitive relationship between different semantics around the sampling point, which can more accurately reflect the positional relationship of the sampling point relative to different semantic surfaces, effectively distinguish the surfaces of different semantics, and improve the accuracy of identifying the components of a role based on semantic information.

[0208] When calculating the third training loss value, depth loss, normal vector loss, and normalization loss are calculated based on both 3D mesh information and real mesh information. Depth loss measures the difference between the model's predicted depth value and the real depth value. Using L1 or L2 loss functions, this allows the model to more accurately estimate depth information in the scene, improving its ability to process 3D scene depth information. Normal vector loss assesses the difference between the model's predicted normal vector and the real normal vector. By calculating the loss due to angle difference or cosine similarity, it helps the model generate smoother and more accurate surfaces, improving its ability to process object surface normal vector information. Normalization loss measures the difference between the model's predicted object shape or size and a certain standardized or desired shape, ensuring that the objects generated by the model maintain consistency and accuracy in size and shape. The loss value obtained from the second stage of training is summed with the depth loss, normal vector loss, and normalization loss as the third training loss value. This comprehensive optimization of the model from multiple aspects enhances its ability to reconstruct 3D characters.

[0209] When the reference role includes target semantics, a regularization loss is calculated based on 3D mesh information and real mesh information. The loss value obtained in the second stage of training is then summed with the depth loss, normal vector loss, normalization loss, and regularization loss as the third training loss value. Target semantics are prone to incomplete rendering. The regularization loss, through a specific calculation method, such as using the cross-entropy function to compare the probability distributions corresponding to information like the equivalent signed distance field values ​​of different sampling points, minimizes the regularization loss to prevent frequent sign changes in the sign field, thus preventing fragmentation. This specifically constrains the reconstruction quality of target semantics, improves the model's reconstruction quality for specific semantics, and enhances the model's stability when handling complex semantics.

[0210] In the process of extracting 3D feature information of a reference character from multi-view images using a large-scale semantically aware reconstruction model, specified semantic downsampling points are divided into sampling points that need to be thickened and those that do not. The original signed distance field values ​​of the sampling points that need to be thickened are updated to equivalent signed distance field values, while the original signed distance field values ​​of the sampling points that do not need to be thickened remain unchanged, thus obtaining the 3D feature information of the reference character. The thickening process increases the "thickness" of the specified semantic parts in the 3D reconstruction, preventing these parts (such as hair and clothing) from breaking or being lost in subsequent processing (such as 3D mesh generation), enhancing the integrity of the specified semantic parts in the 3D reconstruction, and improving the detail quality of the 3D model.

[0211] In the technical architecture for generating 3D characters, the multi-view standardized character generation module generates standardized characters from user-specified inputs such as reference text, images, and sketches, providing unified basic viewpoint information for subsequent 3D reconstruction. By employing an enhanced diffusion model combined with the ReferenceNet network, various user-customized inputs are converted into A-pose images, which are then fed into a fine-tuned Era3D model to generate orthogonal projection images and normal maps of six standard views. This processing method improves the standardization of input information, providing more accurate and unified basic data for subsequent large-scale semantic-aware reconstruction models, enhancing data availability and model processing efficiency.

[0212] The semantic-based hierarchical 3D generation module innovatively introduces a semantic field into large-scale reconstruction models, enabling semantic-based hierarchical generation and effectively decoupling the various components of a 3D character. By extracting image features from orthogonal projection images of six standard viewpoints and inputting them into a semantically aware large-scale reconstruction model, implicit 3D features based on Triplanes are obtained. These features are further decoded into explicit 3D information such as color, volume density, semantics, neural radiation field, and signed distance field. This hierarchical generation method improves the flexibility and customization capabilities of 3D character generation, allowing the model to process information from different semantic layers more precisely, and enhancing the model's understanding and reconstruction capabilities of the semantic structure of 3D characters.

[0213] A multi-level optimization module guided by various prior information adopts a phased and multi-layered approach. It iteratively optimizes the generated layered 3D character results using normal maps generated by the multi-view standardized character generation module and multi-view consistency constraints. In each iteration, the error between the current 3D character result and the normal map and multi-view consistency constraints is calculated. The geometric structure and texture parameters of the 3D character are adjusted based on the error, and this process is repeated until the error meets a preset threshold or the maximum number of iterations is reached. Multi-view consistency constraints ensure that the character's geometric structure and texture features remain consistent and accurate across different viewpoints, avoiding contradictions or mismatches in appearance. This optimization method improves the quality and realism of the generated 3D character and enhances the model's ability to handle the overall consistency of the 3D character.

[0214] When performing multi-level detail optimization on the 3D model of the reference character, different parts of the reference character are extracted according to different specified semantics. Focusing on optimizing the basic human body structure ensures the accuracy and smoothness of the basic human body structure, improving the model's processing precision for the basic structure. The clothing is superimposed on the optimized basic human body structure, fixing the basic human body model and optimizing only the clothing to ensure the fit and detail of the clothing, enhancing the model's ability to blend different semantic parts and handle details. Hair is added, and the optimized basic human body structure and clothing are fixed. Using models such as SAM to accurately identify the hair region guides the hair optimization process, ensuring that the geometric accuracy and texture detail of the hair region reach the optimal state, improving the model's ability to handle complex details. The final result is a customized layered 3D character model with high geometric and texture accuracy in each semantic layer.

[0215] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

[0216] It is understood that in the specific implementation of this application, data related to characters, images, three-dimensional feature information, etc. are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A method for generating a three-dimensional character, performed by an electronic device, the method comprising: Obtain a 2D image containing the reference character; The two-dimensional image is converted into a standard pose image, and a multi-view image of the reference character is generated based on the standard pose image. The multi-view image contains images of the reference character from multiple perspectives. The three-dimensional feature information of the reference character is extracted from the multi-view images; the three-dimensional feature information includes color information, three-dimensional geometric information, and semantic information, wherein the semantic information is used to describe the compositional hierarchy of the reference character; and A 3D model of the reference character is generated based on the aforementioned 3D feature information.

2. The method according to claim 1, wherein the three-dimensional feature information of the reference character is extracted from the multi-view image using a semantically aware large-scale reconstruction model, and the semantically aware large-scale reconstruction model is obtained through the following training process: With the goal of learning the neural radiation field of single-layer semantics, the initial large-scale reconstruction model is trained in the first stage to obtain the large-scale reconstruction model after the first stage of training; the neural radiation field representation is rendered in two-dimensional image using a specified viewpoint and volume rendering method. With the goal of learning the neural radiation field of multi-layer semantics, a second stage of training is performed on the large-scale reconstruction model after the first stage of training to obtain a large-scale reconstruction model after the second stage of training, which is then used as the large-scale reconstruction model with semantic awareness.

3. The method according to claim 2, wherein the training process of the semantically aware large-scale reconstruction model further includes: With the goal of learning a signed distance field with multi-layer semantics, a third stage of training is performed on the large-scale reconstruction model after the second stage of training to obtain a large-scale reconstruction model after the third stage of training. The large-scale reconstruction model after the third stage of training is used as the large-scale reconstruction model with semantic awareness. The signed distance field represents the signed distance from the sampling point in three-dimensional space to the nearest surface.

4. The method according to claim 2 or 3, wherein the step of training the initial large-scale reconstruction model in the first stage with the goal of learning the neural radiation field of single-layer semantics to obtain the large-scale reconstruction model after the first stage training includes: Obtain the three-dimensional feature information extracted by the initial large-scale reconstruction model from the multi-view images of the training character; The three-dimensional surface of the training character is used as the single-layer semantics. The three-dimensional feature information of the training character is used to render the neural radiation field of the three-dimensional surface of the training character to obtain the first rendering model. Calculate the first training loss value based on the first rendering model and the real 3D model; The initial large-scale reconstruction model is optimized based on the first training loss value to obtain the large-scale reconstruction model after the first stage of training.

5. The method according to claim 4, wherein calculating the first training loss value based on the first rendering model and the real 3D model comprises: Calculate the pixel loss, mask loss, semantic loss, and perceptual loss between the first rendering model and the real 3D model, respectively. The perceptual loss represents the similarity in the feature space. The first training loss value is obtained by calculating the sum of the pixel loss, the mask loss, the semantic loss, and the perceptual loss.

6. The method according to any one of claims 2 to 5, wherein the step of performing a second-stage training on the large-scale reconstruction model trained in the first stage with the goal of learning a neural radiation field of multi-layer semantics to obtain a large-scale reconstruction model trained in the second stage comprises: The multi-layer semantics of the training character are determined, and the neural radiation field under the multi-layer semantics is rendered by masking a random number of semantics and the corresponding three-dimensional feature information to obtain a second rendering model. Calculate the second training loss value based on the second rendering model and the real 3D model; The large reconstruction model after the first stage of training is optimized based on the second training loss value to obtain the large reconstruction model after the second stage of training.

7. The method according to claim 6, wherein calculating the second training loss value based on the second rendering model and the real 3D model comprises: For each layer of semantics contained in the unmasked semantic set, calculate the corresponding pixel loss, perceptual loss, masking loss and semantic loss respectively; Based on the pixel loss, perceptual loss, mask loss and semantic loss corresponding to each semantic layer, the sum of the multi-layer semantic losses is calculated to obtain the second training loss value.

8. The method according to any one of claims 4 to 7, wherein the process of rendering the neural radiation field at each semantic layer using the three-dimensional feature information of the trained character is as follows: For each semantic layer, multiple sampling points are obtained, and a set of rendering parameters corresponding to each sampling point is determined; the set of rendering parameters includes transparency, color value, semantic probability, and historical transparency cumulative value. Based on the set of rendering parameters corresponding to each sampling point, the product of the rendering parameters for each sampling point is calculated, and the products of each rendering parameter are accumulated to obtain the color rendered under each semantic layer.

9. The method according to any one of claims 3 to 5, wherein the step of training the large-scale reconstruction model trained in the second stage with the goal of learning a signed distance field of multi-layer semantics in the third stage to obtain a large-scale reconstruction model trained in the third stage comprises: Based on the three-dimensional feature information of the training character, the equivalent signed distance field corresponding to each semantic layer is extracted, and the equivalent signed distance field corresponding to each semantic layer is converted into three-dimensional mesh information respectively; The third training loss value is calculated based on the three-dimensional mesh information and the real mesh information; The large-scale reconstruction model after the second stage of training is optimized based on the third training loss value to obtain the large-scale reconstruction model after the third stage of training.

10. The method according to claim 9, wherein extracting the equivalent signed distance field corresponding to each semantic layer based on the three-dimensional feature information of the trained character, and converting the equivalent signed distance field corresponding to each semantic layer into three-dimensional mesh information respectively, includes: For each sampling point under each semantic layer, determine the maximum probability among the semantic probabilities corresponding to other semantics besides the current semantic, and calculate the probability difference between the maximum probability and the semantic probability corresponding to the current semantic. The larger value is selected from the original signed distance field value corresponding to each sampling point and the probability difference, and used as the equivalent signed distance field value of the corresponding sampling point.

11. The method according to claim 9 or 10, wherein calculating the third training loss value based on the three-dimensional mesh information and the real mesh information comprises: The depth loss, normal vector loss, and normalization loss are calculated based on the 3D mesh information and the real mesh information, respectively. The loss value obtained from the second stage of training is summed with the depth loss, the normal vector loss, and the normalized loss to form the third training loss value.

12. The method according to claim 11, further comprising: If the reference role contains target semantics, a regularization loss is calculated based on the 3D mesh information and the real mesh information, so that the loss value obtained in the second stage of training is combined with the depth loss, the normal vector loss, the normalization loss and the regularization loss as the third training loss value.

13. The method according to any one of claims 2 to 12, wherein the method further comprises: In the process of extracting the three-dimensional feature information of the reference character from the multi-view image using the semantically aware large-scale reconstruction model, the specified semantic downsampling points are divided into sampling points that need to be thickened and sampling points that do not need to be thickened. The original signed distance field values ​​of the sampling points that need to be thickened are updated to equivalent signed distance field values, while the original signed distance field values ​​of the sampling points that do not need to be thickened remain unchanged, in order to obtain the three-dimensional feature information of the reference character.

14. An apparatus for generating a three-dimensional character, the apparatus comprising: The reference module is configured to acquire a two-dimensional image containing a reference character. The conversion module is configured to convert the two-dimensional image into a standard pose image, and generate a multi-view image of the reference character based on the standard pose image, wherein the multi-view image includes images of the reference character from multiple perspectives; The extraction module is configured to extract three-dimensional feature information of the reference character based on the multi-view images; the three-dimensional feature information includes color information, three-dimensional geometric information, and semantic information, wherein the semantic information is used to describe the compositional hierarchy of the reference character; and The generation module is configured to generate a 3D model of the reference character based on the 3D feature information.

15. An electronic device comprising: One or more processors; A memory for storing one or more computer programs that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-13.

16. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, causes the electronic device to perform the method of any one of claims 1-13.

17. A computer program product comprising a computer program that, when executed by a processor of an electronic device, implements the method as described in any one of claims 1-13.

Images