Three-dimensional model processing method and apparatus, and computer device, computer-readable storage medium and computer program product

Abstract

A three-dimensional model processing method and apparatus, and a computer device, a computer-readable storage medium and a computer program product. The method comprises: acquiring a mesh representation of a three-dimensional model; on the basis of the mesh representation, performing model pre-processing on a first texture optimization model, so as to obtain a second texture optimization model; acquiring view parameters under a plurality of camera views in a world coordinate system, and on the basis of the mesh representation and the view parameters under the plurality of camera views, performing multi-layer texture optimization processing on the second texture optimization model, so as to obtain a third texture optimization model; using the third texture optimization model to perform texture prediction processing on the mesh representation, so as to obtain a target texture representation of the three-dimensional model; and exporting the target texture representation as a texture map resource of the three-dimensional model.

Description

Methods, apparatus, computer equipment, computer-readable storage media, and computer program products for processing three-dimensional models

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202411697612.0, filed on November 25, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of artificial intelligence, and more particularly to a method, apparatus, computer device, computer-readable storage medium, and computer program product for processing three-dimensional models. Background Technology

[0004] With the continuous development of 3D generation technology, some 3D model generation algorithms based on text / image have emerged in the industry. 3D model generation and optimization technologies have been widely used in fields such as digital content creation and virtual reality.

[0005] In related technologies, 3D model generation algorithms typically employ fractional distillation sampling, generating geometric and texture information simultaneously through implicit fields. This method relies on deep neural networks to progressively optimize the input signal, thereby obtaining a representation of the 3D model.

[0006] However, existing methods suffer from long generation times, low-quality 3D models, inconsistent and unstable textures, and significant loss of texture and geometric information when exporting textures as explicit mesh representations. Therefore, achieving stable, efficient, and high-quality 3D model generation has become a research hotspot. Summary of the Invention

[0007] This application provides a method, apparatus, computer device, computer-readable storage medium, and computer program product for processing three-dimensional models, which can efficiently optimize the texture of three-dimensional models to obtain high-quality textured three-dimensional models.

[0008] This application provides a method for processing a 3D model, executed by a computer device. The method includes: obtaining a mesh representation of the 3D model, the mesh representation being used to characterize the texture features of the 3D model in the explicit spatial dimension; the 3D model being obtained by 3D reconstruction of an object in reference information; performing model preprocessing on a first texture optimization model based on the mesh representation to obtain a second texture optimization model; the second texture optimization model being used to predict the texture features of the 3D model in the implicit spatial dimension; obtaining viewpoint parameters from multiple camera perspectives in the world coordinate system, and performing multi-layer texture optimization processing on the second texture optimization model based on the mesh representation and the viewpoint parameters from multiple camera perspectives to obtain a third texture optimization model; the third texture optimization model being used to predict the texture features of the 3D model in the implicit spatial dimension from multiple camera perspectives; performing texture prediction processing on the mesh representation using the third texture optimization model to obtain a target texture representation of the 3D model, the target texture representation being used to characterize the optimized texture features of the 3D model in the implicit spatial dimension; and exporting the target texture representation as a texture map resource of the 3D model.

[0009] This application provides a three-dimensional model processing apparatus, comprising: an acquisition unit configured to acquire a mesh representation of a three-dimensional model, the mesh representation being used to characterize the texture features of the three-dimensional model in the explicit spatial dimension; the three-dimensional model being obtained by three-dimensional reconstruction of an object in reference information; a processing unit configured to perform model preprocessing on a first texture optimization model based on the mesh representation to obtain a second texture optimization model; the second texture optimization model being used to predict the texture features of the three-dimensional model in the implicit spatial dimension; the processing unit further configured to acquire viewpoint parameters from multiple camera perspectives in the world coordinate system, and perform multi-layer texture optimization processing on the second texture optimization model based on the mesh representation and the viewpoint parameters from multiple camera perspectives to obtain a third texture optimization model; the third texture optimization model being used to predict the texture features of the three-dimensional model in the implicit spatial dimension from multiple camera perspectives; the processing unit further configured to perform texture prediction processing on the mesh representation using the third texture optimization model to obtain a target texture representation of the three-dimensional model, the target texture representation being used to characterize the optimized texture features of the three-dimensional model in the implicit spatial dimension; and the processing unit further configured to export the target texture representation as a texture map resource of the three-dimensional model.

[0010] This application provides a computer device, which includes: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the above-described three-dimensional model processing method.

[0011] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs. When these computer-executable instructions or computer programs are loaded and executed by a processor, the above-described method for processing three-dimensional models is implemented.

[0012] This application provides a computer program product, which includes computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, they implement the above-described method for processing three-dimensional models.

[0013] In this embodiment, the computer device acquires a mesh representation of the 3D model to be textured. This mesh representation characterizes the texture features of the 3D model in the explicit spatial dimension, i.e., it characterizes the texture features of the 3D model through mesh vertices. Then, based on the mesh representation of the 3D model, the computer device performs model preprocessing on the first texture optimization model to ensure that the optimized second texture optimization model can accurately predict the texture features of the 3D model in the implicit spatial dimension. Then, based on the mesh representation and the viewpoint parameters from multiple camera perspectives in the world coordinate system, the second texture optimization model undergoes multi-level texture optimization processing to obtain a third texture optimization model. This process uses each camera perspective in the world coordinate system as an optimization level, utilizing the viewpoint parameters from each camera perspective to perform multi-level model optimization on the second texture optimization model. By progressively optimizing the second texture optimization model's ability to predict the texture features of the 3D model from each camera perspective (e.g., frontal view, top view, and bottom view) in the world coordinate system, the optimized third texture model effectively ensures that it possesses the ability to optimize the texture features of the 3D model in the implicit spatial dimension from multiple perspectives. By using the optimized third texture optimization model to perform texture prediction processing on the mesh representation of the 3D model, a high-quality target texture representation of the 3D model can be obtained. This target texture representation characterizes the texture features of the 3D model after multi-view optimization in the implicit spatial dimension. Therefore, the texture map resources of the 3D model derived based on this target texture representation can achieve a comprehensive quality improvement. Furthermore, in this embodiment, after obtaining the mesh representation of the 3D model with coarse texture, the first texture optimization model is directly optimized step by step using the mesh representation of the 3D model. This model optimization method allows the first texture optimization model to be used directly in the model inference stage without prior training or fine-tuning, thus eliminating the need for additional model training steps. This not only reduces the generation cost of the 3D model but also enables rapid high-quality texture optimization of the 3D model. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 is a schematic diagram of a three-dimensional model generation scheme in related technologies;

[0016] Figure 2 is a flowchart illustrating a three-dimensional model processing method provided in an embodiment of this application;

[0017] Figure 3 is a schematic diagram of a texture optimization scene provided in an embodiment of this application;

[0018] Figure 4 is a flowchart illustrating another method for processing three-dimensional models provided in an embodiment of this application;

[0019] Figure 5 is a schematic diagram of user-defined reference information provided in an embodiment of this application;

[0020] Figure 6 is a schematic diagram of a mesh representation of a three-dimensional model provided in an embodiment of this application;

[0021] Figure 7 is a schematic diagram of an implicit representation framework based on an MLP model provided in an embodiment of this application;

[0022] Figure 8a is a schematic diagram of a process for first-stage optimization of an MLP model provided in an embodiment of this application;

[0023] Figure 8b is a schematic diagram of another process for first-stage optimization of an MLP model provided in an embodiment of this application;

[0024] Figure 9 is a schematic diagram of a second-stage optimization process for an MLP model provided in an embodiment of this application;

[0025] Figure 10a is a schematic diagram of sampling from a positive angle provided in an embodiment of this application;

[0026] Figure 10b is a top-view sampling diagram provided in an embodiment of this application;

[0027] Figure 10c is a sampling diagram of a downward-looking perspective provided in an embodiment of this application;

[0028] Figure 10d is a schematic diagram of the positions of multiple camera viewpoints in the world coordinate system provided in an embodiment of this application;

[0029] Figure 11 is a schematic diagram of a second-stage optimization process for an MLP model provided in an embodiment of this application;

[0030] Figure 12a is a schematic diagram of texture optimization and redrawing from a positive viewing angle provided in an embodiment of this application;

[0031] Figure 12b is a schematic diagram of a top-view perspective provided in an embodiment of this application;

[0032] Figure 12c is a schematic diagram of a selected upward viewing angle provided in an embodiment of this application;

[0033] Figure 13 is a schematic diagram of a redrawing region setting provided in an embodiment of this application;

[0034] Figure 14 is a schematic diagram of a texture export framework provided in an embodiment of this application;

[0035] Figure 15 is a schematic diagram of the structure of a three-dimensional model processing device provided in an embodiment of this application;

[0036] Figure 16 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0038] This application proposes a method for processing three-dimensional models. This method is a high-quality and rapid optimization scheme for the texture of three-dimensional models based on artificial intelligence technology. The following is a brief introduction to the technical terms and related concepts involved in the three-dimensional model processing method provided in this application embodiment, wherein:

[0039] (1) Artificial Intelligence (AI): AI refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. The technologies within the field of AI mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning. The three-dimensional model processing method provided in this application primarily involves machine learning within AI technology. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of AI and the fundamental way to give computers intelligence; its applications span all areas of AI. Machine learning typically includes techniques such as deep learning, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0040] The 3D model processing method provided in this application mainly relates to deep learning in machine learning technology. Deep learning is machine learning based on artificial neural networks and methods; its important technical feature is its ability to automatically extract features. Compared with manually designed features, the features extracted by deep learning have stronger and more robust performance. This application mainly relates to multi-layer perceptrons (MLPs) and large reconstruction models (LRMs) in artificial neural networks, where: the MLP model is a basic artificial neural network model. The MLP model is composed of multiple neurons in a hierarchical structure, with each layer of neurons connected to all neurons in the previous layer, forming a fully connected feedforward neural network; according to the function of the neurons, the multi-layer neurons in the MLP model are usually classified into input layers, hidden layers, and output layers. Among them, the input layer receives external input data, and the number of neurons in the input layer usually corresponds to the feature dimension of the input data; if the input data is an image, the number of neurons in the input layer may be equal to the number of pixels in the image. Hidden layers are responsible for extracting latent features from the input data and performing nonlinear transformations. The number of hidden layers and the number of neurons in each hidden layer are important parameters in MLP model design, determining the complexity and learning ability of the MLP model. The output layer receives the latent features output from the hidden layers and generates the final prediction result based on these features. The number of neurons in the output layer usually corresponds to the number of tasks in the model task; for example, in a binary classification task, the output layer may contain only one neuron. LRM models are reconstruction models that generate 3D models from a single input image. An LRM model consists of an encoder and a decoder for reconstructing neural radiance fields (NeRFs) from a single image. The LRM model can encode the input image using a pre-trained visual model, project 2D image features onto a 3D plane through cross-attention, and predict the color and density of points in volume rendering using a multilayer perceptron to achieve 3D model reconstruction. The entire LRM model is trained end-to-end to generate high-quality 3D models.

[0041] (2) Three-dimensional model: A 3D model is a model with three-dimensional data constructed in a virtual three-dimensional space. The virtual three-dimensional space is a three-dimensional environment simulated by computer technology. In this environment, the structure and position of the 3D model can be represented by spatial coordinates (i.e., three-dimensional data) indicated by the world coordinate system (or spatial coordinate system) within the three-dimensional space. From a visual perspective, 3D models in virtual three-dimensional space have a sense of depth and three-dimensionality, creating a feeling of existence in the real world. In contrast to the 3D model is the two-dimensional model (or 2D model). A 2D model is a model on a two-dimensional plane, possessing only length and width information, but no depth information. For example, objects in a two-dimensional plane or two-dimensional image can be considered as 2D models with only length and width information, but no depth information; objects in a 2D image can include, but are not limited to, visually perceptible objects such as people, animals, buildings, and cars.

[0042] (3) Explicit spatial dimension: refers to the direct representation of the geometric structure of a 3D model, usually presented in the form of a mesh, including vertex coordinates, normal vectors, UV mapping and other information. This representation of explicit spatial dimension is easy to be compatible with graphics engines and can be directly exported to 3D file formats (such as object file (Wavefront Object File Format, OBJ), FBX (FilmBox) etc.).

[0043] (4) Implicit spatial dimension: refers to an abstract representation of the surface of a 3D model through neural networks or functions, such as Neural Radiance Fields (NeRF) or MLP. This representation does not depend on a specific mesh structure, but outputs attribute values ​​such as color and density by taking input parameters such as position and orientation, and is suitable for high-precision texture prediction and optimization.

[0044] (5) Camera viewpoint: refers to the observer's position and angle defined in the world coordinate system, used to control the rendering of the 3D model from a specific viewpoint. Each camera viewpoint is determined by parameters such as pitch angle, rotation angle, and camera distance. Different camera viewpoints can be used to capture different parts of the 3D model, thereby improving the comprehensiveness and accuracy of texture optimization.

[0045] (6) Texture Mapping Resources: These refer to the final exported 2D image resources used to provide surface texture information for 3D models. These texture mapping resources can be mapped onto the surface of 3D models using UV mapping, supporting use in graphics engines and offering good compatibility and visualization effects.

[0046] With the rapid development of 3D technology, it is now possible to generate 3D models of objects based on text descriptions or objects in 2D images. The process of 3D model generation in related technologies can be seen in Figure 1. As shown in Figure 1, the process of creating a 3D model may include: Step S01, receiving reference information as a reference for model generation. This reference information describes the object to be generated as a 3D model, such as in text or image form. Based on the input reference information, a 3D model corresponding to the object described in the reference information is generated through an object generation algorithm. Specifically, in Step S02, a geometric framework corresponding to the 3D model is generated; in Step S03, a texture is generated for the geometric framework; and finally, in Step S04, the textured geometric framework is exported as a 3D model. Post-processing is then performed on the 3D model, such as exporting it as a mesh representation. The object generation algorithm is a type of algorithm that can generate a 3D model corresponding to an object based on an object described by reference information through 3D reconstruction technology. Examples of object generation algorithms include 3D model generation (such as Neural Radiance Fields (NeRF)) or 2D distillation dimensionality enhancement algorithms. The style types of 3D models obtained by reconstructing objects from 2D images can be varied, including but not limited to: cartoon style, anime style, realistic style, and animation style. For example, reconstructing a person from a 2D image yields a 3D cartoon portrait of that object in a cartoon style. This application does not limit the style type of the 3D model.

[0047] Practical experience has revealed numerous problems with existing 3D model generation solutions. For instance, 3D reconstruction based on 2D distillation dimensionality enhancement algorithms is time-consuming, making online deployment difficult, and the generated 3D model quality is unstable. Furthermore, since 2D distillation dimensionality enhancement algorithms directly analyze and process reference information, the lack of consistent prior information leads to inconsistent textures in the generated 3D models, making it difficult to provide stable texture effects. Another example is when exporting a 3D model as a mesh representation based on neural radiation fields; the textures or geometries supported by the exported mesh representation differ significantly from those of the original, resulting in substantial loss of texture or geometry in the exported model.

[0048] To improve the texture quality of the 3D model obtained from 3D reconstruction, the 3D model processing method provided in this application emphasizes texture optimization of the already reconstructed coarse 3D model, aiming to obtain high-quality 3D model textures. For example, texture optimization of the 3D model generated by the LRM model further improves the texture quality and stability of the 3D model. The principle of the LRM model in reconstructing an object from an input reference image and generating a coarse 3D model corresponding to that object can be found in the aforementioned introduction to the LRM model, and will not be repeated here.

[0049] The main principle of texture optimization for coarse 3D models in this application is to perform a two-stage model optimization on the texture optimization model, so that the optimized texture optimization model has the ability to predict the optimized texture features of the 3D model in the implicit space dimension. When the optimized texture optimization model is used to perform texture prediction processing on the 3D model, the optimized texture features of the 3D model in the implicit space dimension can be directly obtained, thereby achieving the purpose of converting the coarse texture features of the 3D model into fine texture features. The texture optimization model involved in this application can be the aforementioned MLP model, and the relevant content of the MLP model will not be repeated here. To facilitate the distinction between texture optimization models obtained from different optimization stages, this application supports referring to the texture optimization model before the first stage of optimization as the first texture optimization model, the texture optimization model after the first stage of optimization as the second texture optimization model, and the texture optimization model obtained by performing a second stage of optimization on the second texture optimization model as the third texture optimization model. This is hereby explained.

[0050] The main technical flow of the 3D model processing method provided in this application embodiment can be seen in Figure 2. As shown in Figure 2, this application embodiment mainly includes three parts: S10, model preprocessing; S20, texture optimization and redrawing; S30, texture export. Wherein:

[0051] In the model preprocessing section: Model preprocessing is the first stage of optimization for the first texture optimization model. This first stage of optimization enables the optimized second texture optimization model to correctly predict the texture features of the 3D model. The input data for the first texture optimization model is a coarsely reconstructed 3D model, which is a mesh representation with texture information. After the 3D model is input into the first texture optimization model, the texture information is transformed into a feature representation in the implicit space dimension through a color fitting method, i.e., the texture features in the implicit space represented by the MLP model. The advantage of converting the mesh representation of the 3D model into a feature representation in the implicit space dimension (or implicit representation) is that the implicit representation is a global representation, which facilitates model optimization of the first texture optimization model based on this implicit representation and guided by a loss function.

[0052] In the texture optimization and redrawing section: Texture optimization and redrawing is the second stage of optimization for the texture optimization model, that is, the process of further optimizing the second texture optimization model obtained from the first stage. Through this second stage optimization, the prediction quality of the texture features of the 3D model by the second texture optimization model can be further refined, so that the optimized third texture optimization model can comprehensively optimize the texture feature quality of the entire 3D model. The texture optimization and redrawing stage can combine camera pose control, redrawing algorithms (such as 2D cartoon texture redrawing algorithms), and differentiable rendering algorithms to optimize the texture of the 3D model, so that the optimized texture is mapped to the 3D model, improving the texture quality of the 3D model.

[0053] In the texture export section: Texture export refers to the process of exporting the target texture representation obtained from the texture prediction processing of the 3D model using the optimized third texture optimization model in the second stage as a texture map resource (or simply a texture map). The target texture representation can be a feature representation, which is a color encoding that encodes the texture features of the 3D model. In this embodiment, interpolation is mainly used to map the target texture representation to the UV space of the 3D model, thereby exporting the target texture representation as a texture map of the 3D model. The UV space defines the correspondence between the two-dimensional target texture representation and the 3D curved surface mesh, enabling the texture map resource to be accurately fitted onto the 3D model. The UV space will be elaborated on in the subsequent specific implementation of texture export.

[0054] Through practice, it has been found that the method for texture optimization of 3D models based on differentiable rendering provided in this application has at least the following advantages: On the one hand, after obtaining the mesh representation of the 3D model with coarse texture, the first texture optimization model is directly optimized step by step using the mesh representation of the 3D model. Compared with pre-optimizing the texture optimization model and then using the optimized texture optimization model for texture optimization in the model inference stage, this application adopts a training-free model optimization method, which does not require additional model training. This not only reduces the cost of generating 3D models, but also enables rapid texture optimization of 3D models (in practice, texture optimization can be achieved in just three minutes), achieving high-quality texture optimization, thereby quickly achieving high-quality 3D cartoon portrait model texture optimization. On the other hand, the input data to the first texture optimization model is the mesh representation of the 3D model that has been reconstructed. This not only allows for texture optimization through explicit geometric expression, but also, based on the texture prior of the 3D model that has been reconstructed, optimizes the texture with the global representation of the MLP model, providing a more stable and consistent texture generation effect. Furthermore, by combining multiple camera perspectives and mature redrawing algorithms during the texture optimization and redrawing stages, the second texture optimization model is optimized layer by layer. This improves the ability of the optimized third texture optimization model to predict texture features with richer details and higher quality, thereby greatly improving the texture generation effect and texture consistency of the 3D model. On the other hand, the texture export stage supports the use of interpolation to achieve texture mapping, resulting in less texture loss in the exported texture map resources of the 3D model. Therefore, after applying the texture map resources with less texture loss to the 3D model, it can ensure that the overall viewpoint effect of the 3D model is more detailed and realistic.

[0055] The 3D model processing method provided in this application, as a 3D model texture optimization solution based on artificial intelligence technology, can be applied to various texture optimization scenarios where a rough 3D model has been obtained through 3D reconstruction and texture optimization is required. Texture optimization scenarios may include, but are not limited to: Character generation: This application is applied to character generation scenarios, requiring only the user to provide reference information about the object from a frontal view (such as a frontal view of a human figure, or a textual description of the object's human figure from a frontal view) to generate the corresponding 3D model. Film and game design: In film and television, 3D model character modeling is used to create various special effects, scenes, and characters; in game design, 3D modeling technology is used to create game characters, scenes, and props, providing players with a realistic gaming experience. Industrial design: In the product design stage, 3D modeling technology is used to create the appearance and structural models of products, helping designers to intuitively display design schemes and reduce the cost and time of sample production. Architectural design: In architectural design, 3D modeling technology is used to create 3D models of buildings, realizing the visualization of architecture and helping designers better understand and communicate design schemes. Interior Design: In interior design, 3D modeling technology is used to create 3D models of interior spaces, helping designers better plan and design interior spaces and produce renderings. Medical Field: In the medical field, 3D modeling technology is used to create 3D models of human organs, bones, etc., for surgical planning, medical training, and patient education.

[0056] It should be understood that the above description is merely an exemplary product performance and interactive dialogue scenario provided by the embodiments of this application, and does not limit the product performance and texture optimization scenarios of the 3D model processing method provided by the embodiments of this application. The 3D model processing method provided by the embodiments of this application can help create high-quality textured 3D models in various texture optimization scenarios, demonstrating high value and practicality in various texture optimization scenarios, and helping to improve user experience and satisfaction.

[0057] To facilitate understanding of the 3D model processing method provided in this application embodiment, the texture optimization scene involved in this application embodiment will be briefly introduced below with reference to the scene diagram corresponding to the 3D model processing system shown in Figure 3. As shown in Figure 3, the 3D model processing system includes a terminal 301, a server 302, and a server 303. This application embodiment does not limit the number and naming of the terminal 301, server 302, and server 303.

[0058] Terminal 301 is a smart terminal used by the user. Through this terminal 301, the user can upload reference information to guide the creation of 3D models. The reference information can be in text or image form. Terminal 301 may include, but is not limited to, smartphones (such as smartphones running the Android system or smartphones running the Internetworking Operating System (IOS), tablet computers, portable personal computers, mobile Internet Devices (MIDs), in-vehicle devices, head-mounted devices, intelligent chatbots, and aircraft, etc. This application embodiment does not limit the type of terminal device, which is only stated here.

[0059] Server 302 is the server corresponding to terminal 301, used to interact with terminal 301 to provide computing and application service support. Specifically, server 302 is a background service deployed with an object generation algorithm; for example, if server 302 deploys an LRM model, after receiving reference information transmitted by terminal 301, server 302 can call the LRM model to generate a coarse-textured 3D model based on that reference information. Server 303 is a background service deployed with the 3D model processing method provided in this embodiment; server 303 receives the coarse-textured 3D model transmitted by server 302 and uses the 3D model processing method to optimize the texture of the coarse-textured 3D model to obtain a final refined textured 3D model. Servers 302 and 303 can be independent physical servers, server clusters or distributed networks (or distributed systems) composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals 301, 302, and 303 can be directly or indirectly connected via wired or wireless communication; this embodiment does not impose any limitations on this connection.

[0060] The three-dimensional model processing method provided in the aforementioned embodiments of this application mainly focuses on texture optimization of the reconstructed three-dimensional model. Therefore, the computer device executing the three-dimensional model processing method provided in this application embodiment is server 303 in the system shown in Figure 3. This computer device can serve as a texture optimization module or unit in the three-dimensional model creation process. The complete three-dimensional model creation process is jointly executed by terminal 301, server 302, and server 303. The specific implementation process of three-dimensional model creation jointly executed by terminal 301, server 302, and server 303 will be described below with reference to the system shown in Figure 3.

[0061] First, when a user wants to create a 3D model while using terminal 301, they can upload reference information through the model creation tool on terminal 301. This reference information serves as guidance for the 3D model creation process. The model creation tool is deployed on terminal 301 and can receive the user-uploaded reference information. The model creation tool can be an application, or a plugin / sub-function within an application. The application can run in various ways, including but not limited to: a client running on terminal 301 via a downloaded installation package; a small program that can be used without downloading an installation package, often as a sub-program of the client; or a web (World Wide Web) application opened through a browser running on terminal 301. After receiving the user-uploaded reference information, terminal 301 sends it to server 302.

[0062] Then, after receiving the user-uploaded reference information sent by terminal 301, server 302 calls its deployed object generation algorithm to analyze and process the reference information, automatically creating a 3D model corresponding to the object described in the reference information. At this time, the texture of the 3D model is relatively coarse, and the 3D model is represented by a mesh. This mesh representation is used to characterize the texture features of the 3D model in the explicit spatial dimension. Server 302 transmits the mesh-represented 3D model to server 303. After receiving the mesh representation of the 3D model sent by server 302, server 303 first performs model preprocessing on the first texture optimization model based on the mesh representation, aiming to optimize the first texture optimization model's ability to correctly represent the texture features of the 3D model, obtaining a second texture optimization model. This second texture optimization model has the ability to predict the texture features of the 3D model in the implicit spatial dimension. Then, server 303 obtains the view parameters from multiple camera perspectives in the world coordinate system, and performs multi-layer texture optimization processing on the second texture optimization model through mesh representation and view parameters from multiple camera perspectives. The aim is to optimize the ability of the second texture optimization model to optimize the texture of the three-dimensional model in all aspects, and obtain the third texture optimization model. The third texture optimization model has the ability to predict the texture features of the three-dimensional model in the implicit spatial dimension from multiple camera perspectives.

[0063] Finally, server 303 inputs the mesh representation of the 3D model into the third texture optimization model, enabling the third texture optimization model to perform texture prediction processing on the mesh representation to obtain the target texture representation of the 3D model. This target texture representation can be used to characterize the texture features of the 3D model after optimization in the implicit spatial dimension. In this way, after server 303 exports the target texture representation as the texture map resource of the 3D model, it can ensure that the final exported 3D model has high-quality and high-stability texture effects.

[0064] Therefore, on the one hand, the texture optimization process for 3D models in this application is training-free, effectively improving the speed and efficiency of 3D model texture optimization. On the other hand, combining techniques such as camera pose, redrawing algorithms, and differentiable rendering to optimize the texture of 3D models can comprehensively optimize the texture of 3D models, improving texture effects and stability.

[0065] Based on the above brief introduction of the 3D model processing method and texture optimization system provided in the embodiments of this application, the following points should also be noted: First, the system architecture shown in Figure 3 of this application embodiment is for the purpose of more clearly illustrating the technical solution of this application embodiment and does not constitute a limitation on the technical solution provided in this application embodiment. Those skilled in the art will understand that with the evolution of system architecture and the emergence of new business scenarios, the technical solution provided in this application embodiment is also applicable to similar technical problems. That is to say, the schematic diagram of the texture optimization system architecture shown in Figure 3 is an exemplary schematic diagram. In actual applications, the number and types of devices included in the texture optimization system can change, and this application embodiment does not limit the schematic diagram of the texture optimization system architecture. For example, server 302 and server 303 in the system shown in Figure 3 may be the same server, or server 302 and server 303 may not belong to different distributed networks, but may be two servers belonging to the same distributed network. Second, the data collection and processing in this application embodiment should strictly comply with the requirements of relevant laws and regulations. Obtaining personal information requires the knowledge or consent of the individual (or has a legal basis for information acquisition), and subsequent data use and processing should be carried out within the scope of laws and regulations and the authorization of the personal information subject. For example, when the embodiments of this application are applied to specific products or technologies, such as when generating a 3D model, it is necessary to obtain the user's permission or consent, and the collection, use and processing of related data (such as texture optimization of the 3D model) must comply with the relevant laws, regulations and standards of the relevant regions.

[0066] Based on the three-dimensional model processing method described above, this application proposes a more detailed three-dimensional model processing method. The following will describe the three-dimensional model processing method proposed in this application in detail with reference to the accompanying drawings. Figure 4 shows a flowchart illustrating a three-dimensional model processing method provided in an exemplary embodiment of this application. This processing method can be executed by a computer device, specifically the aforementioned server 303. The three-dimensional model processing method may include, but is not limited to, steps S401 to S405:

[0067] S401: Obtain the mesh representation of the 3D model.

[0068] A 3D model is obtained by reconstructing the object described by reference information using an object generation algorithm. The reference information can be in image or text form, and the object in the reference information can be any object capable of 3D reconstruction. In this embodiment, the reference information can be user-defined or provided by the developer of the model creation tool; this embodiment does not limit the scope of the reference information.

[0069] For example, a schematic diagram of user-defined reference information can be seen in Figure 5. As shown in Figure 5, when a user needs to view a model, they can trigger the model display interface 501 in the model creation tool through the terminal. The model display interface 501 displays various 3D models (such as those published by other users using the model creation tool). The model display interface 501 also displays an option 5011 for triggering the model creation interface 502. When the option 5011 is triggered, the model creation interface 502 is displayed. If the user has already created a 3D model in the past, the created 3D model 5021 is displayed in the model creation interface 502; the model creation interface 502 also includes a model creation option 5022. When the model creation option 5022 is triggered, it indicates that the user wants to create a new 3D model, and the user is redirected to the information input interface 503, where they can input reference information. For example, the input reference information can be a reference image or text. When the reference information is in text format, the computer device needs to use a generative model to generate a corresponding reference image based on the text-based reference information, so as to facilitate subsequent operations such as 3D model creation and texture optimization based on the reference image. It should be noted that Figure 5 is only one way of uploading reference information and does not limit the way reference information is obtained in the embodiments of this application.

[0070] The 3D model obtained by 3D reconstruction based on reference information is represented by a mesh. The mesh representation of a 3D model refers to decomposing the 3D model into a series of polygons, which define the shape, structure, and texture of the 3D model's surface. Polygons include, but are not limited to, triangles, quadrilaterals, pentagons, and irregular shapes. Each polygon consists of vertices, which are connected by edges to form a mesh structure. In other words, each polygon that makes up the 3D model can be considered a mesh, and the basic components of this mesh include: mesh vertices (Vertex), edges (Edge), and faces (Face). The mesh vertices of the 3D model define the position of the mesh in the world coordinate system. Each mesh vertex has a definite coordinate (x, y, z) in the world coordinate system; by using the coordinates of each mesh vertex, the position of the mesh formed by these multiple mesh vertices in the world coordinate system can be determined. The edges of the 3D model are line segments connecting two mesh vertices, defining the boundaries of the mesh. Connecting mesh vertices through edges in the world coordinate system allows us to depict the shape of the mesh, and thus the shapes of multiple meshes can outline the geometric contours of the 3D model. The faces of a 3D model are planar regions enclosed by the edges of a mesh. The world coordinate system mentioned above can be understood as a spatial coordinate system built in a virtual 3D space. Through this spatial coordinate system, the position and geometry of the 3D model in the virtual 3D space can be determined, and by adjusting the vertex coordinates and edge connection methods of the 3D model's mesh, fine control and optimization of the 3D model can be achieved.

[0071] For example, a schematic diagram of the mesh of the 3D model can be seen in Figure 6. As shown in Figure 6, the head of the 3D model is composed of multiple meshes, each with a different shape. For example, mesh 601 is a decagon, and mesh 602 is a quadrilateral. Each mesh includes mesh vertices, edges, and faces; for example, mesh 602 includes four mesh vertices (mesh vertex 6021, mesh vertex 6022, mesh vertex 6023, and mesh vertex 6024), four edges, and one face. Continuing to refer to Figure 6, mesh 602 is the mesh that constitutes the boundary of the head of the 3D model. By adjusting the coordinates of the mesh vertices of mesh 602, the shape of mesh 602 can be changed, thereby adjusting the outline of the head of the 3D model through the change in the shape of mesh 602.

[0072] In this embodiment, each mesh vertex in the mesh representation of the 3D model possesses three vertex attributes: geometric information, texture information, and vertex texture coordinates. Geometric information refers to the 3D coordinates of the mesh vertex in the world coordinate system, used to define the position of the mesh vertex. Thus, the position of the mesh can be determined using the geometric information of multiple mesh vertices. Texture information, also known as vertex color information, describes the surface color of the mesh vertex. During the 3D model rendering process, interpolation is performed within the mesh region based on the vertex color information of multiple mesh vertices to determine the color of each pixel in the entire mesh. Vertex texture coordinates, or simply texture coordinates, define the coordinate system on which the texture is mapped onto the surface of the 3D model. These texture coordinates allow for the mapping of 2D texture maps onto the surface of the 3D model, adding texture and decorative effects. Therefore, mesh vertices in the mesh representation are an important component of the 3D model. Through the three attributes of the mesh vertices, not only can the outline or geometry of the 3D model be determined, but the texture effect of the 3D model's surface can also be defined.

[0073] S402: Perform model preprocessing on the first texture optimization model based on the mesh representation to obtain the second texture optimization model.

[0074] As mentioned earlier, the 3D model output by the LRM model is a mesh representation. This mesh representation characterizes the texture features of the 3D model through the vertex color information of the mesh vertices. The mesh representation is essentially an explicit representation used to characterize the texture features of the 3D model in the explicit spatial dimension. This explicit spatial mesh representation is not conducive to subsequent optimization of the 3D model's texture. Therefore, this application introduces a texture optimization model to convert the explicit representation of the 3D model in the explicit spatial dimension—the pixel representation (which is explicit and can characterize the texture of the mesh vertex colors in the mesh representation)—into an implicit representation of the 3D model in the implicit spatial dimension—the feature representation. The feature representation in the implicit spatial dimension is more conducive to optimizing the texture optimization model through a loss function, thereby achieving texture quality optimization of the 3D model through the optimized texture optimization model.

[0075] The texture optimization model involved in this application embodiment can be an MLP model. The first texture optimization model refers to the unoptimized MLP model, and the second texture optimization model refers to the MLP model obtained by preprocessing the first texture optimization model. The model preprocessing for the first texture optimization model aims to optimize the first texture optimization model's ability to predict the texture features of the mesh vertices in the 3D model by using the global coordinates of the 3D model (i.e., the geometric information of the mesh vertices of the entire 3D model). In other words, by inputting the geometric information of the global mesh vertices of the 3D model into the first texture optimization model, the first texture optimization model learns the ability to predict the implicit representation of the whole world.

[0076] For example, a schematic diagram of the implicit representation framework based on the MLP model can be seen in Figure 7. As shown in Figure 7, regardless of whether it is the first-stage optimization or the second-stage optimization of the MLP model, the global coordinates 701 of the 3D model are used as the output data of the MLP model 70. The output data of the MLP model is the global implicit representation of the global coordinate prediction, which is the texture feature / texture representation 704 of the 3D model mentioned above. The encoder 702 and mapper 703 in the MLP model 70 shown in Figure 7 are both composed of neurons. The encoder 702 is used to extract features from the input data, while the mapper 703 is used to convert the explicit representation into an implicit representation.

[0077] Based on the implicit representation framework based on the MLP model shown in Figure 7, the specific implementation process of the first stage optimization of the MLP model (i.e., the optimization process for the first texture optimization model) is described below; the process of the first stage optimization of the MLP model can be seen in Figure 8a, including but not limited to steps S11 to S14:

[0078] S11: The first texture optimization model is used to perform texture prediction processing on the geometric information to obtain the first predicted texture features of the mesh vertices.

[0079] The geometric information of the global mesh vertices in the mesh representation of the 3D model is used as input data for the first texture optimization model. This first texture optimization model then predicts the first predicted texture feature for each mesh vertex in the global mesh through feature extraction by the encoder and feature mapping by the mapper. Compared to the mesh representation of the 3D model, the first predicted texture feature output by the first texture optimization model is used to characterize the texture features of the 3D model in the implicit spatial dimension.

[0080] S12: Perform differentiable rendering processing on the first predicted texture features to obtain the first predicted rendered image.

[0081] After predicting the global first predicted texture features of the 3D model based on the first texture optimization model, considering that these first predicted texture features are only implicit texture features and have not yet been mapped to the 3D model, this application embodiment supports the use of a differentiable rendering algorithm to perform differentiable rendering processing on the first predicted texture features. The aim is to render the texture effect indicated by the first predicted texture features onto the 3D model to obtain a first predicted rendered image (or first predicted rendered view) corresponding to the 3D model. The first predicted rendered texture image includes the 3D model rendered based on the first predicted texture features, and may include the geometry and texture effects of the 3D model.

[0082] Differentiable rendering is a rendering method based on the principle of mathematical differentiability. It provides gradient feedback during graphics rendering for backpropagation optimization. Differentiable rendering allows the calculation of the gradient of the loss function during rendering, thereby guiding the adjustment of model parameters. In the embodiments of this application, differentiable rendering is applied to the first predicted texture features and the original texture information to generate corresponding predicted and real rendered images. The differentiable rendering algorithm mainly uses the viewpoint parameters in the world coordinate system to render the texture indicated by the first predicted texture features onto the 3D model in the corresponding camera viewpoint. For example, if the camera viewpoint is a frontal view, then in the first predicted rendered image obtained by the differentiable rendering algorithm on the first predicted texture features, the 3D model is the model in the frontal view of the camera; simply put, the first predicted rendered image is a frontal view of the 3D model. The first predicted rendered image is a 2D image generated by differentiable rendering of the first predicted texture features, reflecting the surface texture effect predicted by the model.

[0083] The following describes the specific process of using a differentiable rendering algorithm to perform differentiable rendering on the first predicted texture feature: The computer device acquires the viewpoint parameters under a random camera viewpoint in the world coordinate system. That is, the camera is randomly placed at a certain position in the world coordinate system, and when the camera is facing the origin of the world coordinate system from that position, the viewpoint captured by the camera is the random camera viewpoint. The viewpoint parameters under this random camera viewpoint can be characterized by parameters such as the top angle, rotation angle, and camera distance. Then, based on the viewpoint parameters under this random camera viewpoint, an initial rendered image of the 3D model in the world coordinate system is acquired. This initial rendered image can be understood as the image obtained by taking a picture of the 3D model from the camera position indicated by the viewpoint parameters under the random camera viewpoint, when the 3D model is at the origin of the world coordinate system. Finally, the first predicted texture feature is micro-rendered onto the initial rendered image to obtain the first predicted rendered image.

[0084] S13: Perform differentiable rendering on the texture information to obtain the first realistic rendered image.

[0085] The first realistic rendered image is an image generated by differentiable rendering of texture information, representing the true state of the current model. As described above, the mesh representation of the 3D model includes not only the geometric information of the mesh vertices but also the texture information of the mesh vertices, which is the true texture information of the 3D model. Based on this, this embodiment will adopt the same implementation process as step S12 to perform differentiable rendering of the global geometric information of the 3D model to obtain the first realistic rendered image corresponding to the 3D model. This first realistic rendered image and the first predicted rendered image in step S12 are paired to form a set of data pairs that can optimize the first texture optimization model.

[0086] It should be noted that the specific implementation process of performing differentiable rendering on the texture information of the mesh vertices in the mesh representation of the 3D model is the same as the specific implementation process of performing differentiable rendering on the first predicted texture feature in step S12 above; furthermore, the viewpoint parameters used in the random camera viewpoint when performing differentiable rendering on the global geometric information of the 3D model are also the same as the viewpoint parameters used in the random camera viewpoint when performing differentiable rendering on the first predicted texture feature in step S12 above. For the sake of brevity, the specific implementation process of differentiable rendering on texture information will not be elaborated here.

[0087] S14: Optimize the first texture optimization model in the direction of reducing the difference between the first predicted rendered image and the first real rendered image to obtain the second texture optimization model.

[0088] Based on the aforementioned steps S12 and S13, paired rendering views, namely the first predicted rendering image and the first real rendering image, are constructed. Thus, the difference information between the first predicted rendering image and the first real rendering image can be calculated. This difference information serves as the loss information used in this iteration to optimize the model parameters of the first texture optimization model. Optimizing the first texture optimization model in the direction of reducing the difference between the first predicted rendering image and the first real rendering image means that during multiple iterations of optimization of the first texture optimization model, the model parameters are adjusted in the direction of gradually reducing the loss information in each iteration until the loss information calculated by consecutive iterations is zero, or the multiple loss information values ​​obtained by consecutive iterations tend to stabilize (e.g., the value 1). At this point, the first texture optimization model optimized in the last iteration is used as the second texture optimization model, ensuring that the second texture optimization model obtained in the first stage can be used to predict the texture features of the 3D model in the implicit spatial dimension. In other words, the goal of step S14 is to adjust the parameters of the first texture optimization model by minimizing the difference between the first predicted rendering image and the first real rendering image, so that the second texture optimization model can more accurately generate high-quality texture features. Specifically, during the optimization process, the differences between the first predicted rendered image and the first real rendered image (such as mean squared error, SSIM, etc.) are calculated, and the model parameters are adjusted according to the direction of the differences to gradually approach the ideal texture effect. The second texture optimization model has the ability to accurately predict the texture features of the 3D model in the implicit spatial dimension based on the global coordinates of the 3D model.

[0089] The model optimization process uses optimization algorithms such as gradient descent to iteratively adjust the parameters of the first texture optimization model, making the model output closer to the real texture. Finally, the optimized model becomes the second texture optimization model, which has higher texture generation accuracy and consistency. The optimization strategy proposed in this application significantly improves the model's texture generation capability, reduces manual intervention, and increases generation efficiency and quality. Furthermore, since no additional training is required when applying the optimization strategy proposed in this application, computational costs are reduced, enhancing its practical application value.

[0090] For ease of explanation, the rendered view (such as the first predicted rendered image) obtained by differentiating the implicit representation output by the MLP model using a differentiable rendering algorithm is denoted as Render View I. Render The rendered view (such as the first real rendered view) obtained by differentiating the vertex color texture of the mesh vertices in the mesh representation of the 3D model through a differentiable rendering algorithm is represented as rendered view I. RefTherefore, the consistency loss during the iterative optimization process of the first texture optimization model is expressed by the following formula (1): Loss = MSE(I Render ,I Ref (1).

[0091] Among them, I Render To predict the rendered image (such as the first predicted rendered image), I Ref For the real-world rendered image (such as the first real-world rendered image), Loss is the predicted rendered image I. Render and realistically rendered image I Ref The loss value; mean squared error (MSE) is the loss function.

[0092] To facilitate understanding of the implementation process shown in steps S11 to S14 above, the first-stage optimization process of the first texture optimization model will be further explained below using the framework diagram shown in Figure 8b. As shown in Figure 8b, after inputting the global coordinates 801 of the 3D model 800 into the first texture optimization model (i.e., the MLP model), the first texture optimization model performs texture prediction processing on the global coordinates to obtain the first predicted texture feature 802. Then, using the viewpoint parameters under the random camera view, the first predicted texture feature 802 is subjected to differentiable rendering processing to obtain the first predicted rendered image 803; simultaneously, using the viewpoint parameters under the random camera view, the real texture information 804 of the mesh vertices of the 3D model is subjected to differentiable rendering processing to obtain the first real rendered image 805. The difference between the first predicted rendered image 803 and the first real rendered image 805 is calculated using a loss function, and this difference is used as loss information to adjust the model parameters of the first texture optimization model, resulting in the first texture optimization model after this iteration. Then, the same logic as above can be used to iteratively optimize the first texture optimization model (the viewpoint parameters under random camera viewpoints need to be changed) to obtain the second texture optimization model after the first stage of optimization.

[0093] Therefore, by preprocessing the MLP model, appropriate model parameters can be set for the MLP model, enabling the preprocessed MLP model to accurately predict the texture features of the 3D model in the implicit spatial dimension, thus laying the model foundation for the second stage of optimization of the MLP model for texture optimization and redrawing.

[0094] S403: Obtain the view parameters from multiple camera perspectives in the world coordinate system, and perform multi-layer texture optimization processing on the second texture optimization model based on the mesh representation and the view parameters from multiple camera perspectives to obtain the third texture optimization model.

[0095] Based on the aforementioned first-stage optimization of the MLP model, an MLP model with good predictive ability of texture features in implicit spatial dimension has been obtained (i.e., the second texture optimization model). In order to optimize and improve the texture quality of the three-dimensional model, this application embodiment continues to perform a second-stage optimization (i.e., texture optimization and redrawing) on ​​the second texture optimization model, aiming to ensure that the MLP model optimized in the second stage (i.e., the third texture optimization model) has the ability to comprehensively optimize the texture features of the three-dimensional model.

[0096] For example, a schematic diagram of the optimization framework for the second texture optimization model in the second stage can be seen in Figure 9. As shown in Figure 9, the optimization process in the second stage mainly combines technologies such as camera pose, redrawing algorithm, and differentiable rendering algorithm to achieve model optimization for the second texture optimization model in the second stage. After inputting the 3D model 901, a rendered image 903 is obtained through camera visual sampling 902, and the rendered image 903 is redrawn through a redrawing algorithm to obtain a redrawn image 904. Then, through loss optimization, based on the input reference information, the texture 904 is continuously optimized 905 to obtain the final output texture 906. Here, camera pose refers to the placement posture of the camera at different camera positions in the world coordinate system. To ensure the completeness and comprehensiveness of the texture optimization for the 3D model, this application embodiment supports selecting multiple camera views in the world coordinate system as optimization views for optimizing the second texture optimization model. In this way, the second texture optimization model can be optimized from multiple camera views during the texture optimization process, ensuring that the third texture optimization model optimized in the second stage can comprehensively improve the texture quality of the 3D model.

[0097] The specific sampling process for camera viewpoint sampling will be described below. In the implementation, the position of the 3D model in the virtual 3D space is set as the origin of the world coordinate system. To ensure that the camera viewpoints sampled in the world coordinate system can cover the entire 3D model from all angles, this embodiment supports the acquisition of multiple camera viewpoints around the origin of the world coordinate system. These multiple camera viewpoints may include: one frontal view, N1 top-down viewpoints, and N2 bottom-up viewpoints, where N1 and N2 are both positive integers. For ease of explanation, the sampling process for camera viewpoints will be described below using N1 = 15 and N2 = 4 as an example.

[0098] The implementation process of sampling the orthogonal viewpoint in the world coordinate system is explained below: During implementation, a coordinate point with a pitch angle of 0°, a camera distance of 0°, and a rotation angle of 0° can be selected as the first camera position in the world coordinate system. In other words, the coordinate point with a pitch angle of 0°, a camera distance of 0°, and a rotation angle of 0° is determined as the first camera position in the world coordinate system. Then, when the camera is placed at this first camera position, the viewpoint taken by the camera from this position relative to the origin of the world coordinate system is taken as the orthogonal viewpoint. Next, the viewpoint parameters under the orthogonal viewpoint are obtained. In this case, the viewpoint parameters under the orthogonal viewpoint include: a pitch angle of 0°, a camera distance of 0°, and a rotation angle of 0°.

[0099] The world coordinate system is a three-dimensional spatial reference system used to uniformly describe the position and orientation of objects in space. This application uses a right-handed coordinate system as the world coordinate system to ensure consistency of all camera viewpoint parameters. A pitch angle of 0° indicates that the camera's line of sight is parallel to the horizontal plane, i.e., looking directly forward; a rotation angle of 0° indicates that the camera has not rotated around its vertical axis and is facing the target directly; the camera distance is the first distance, meaning the distance from the camera to the target object (the origin of the world coordinate system) is set to a fixed value for the target object. The value of the first distance is adjusted according to actual modeling needs, such as for generating a standard portrait viewpoint. The orthographic viewpoint refers to the camera viewpoint formed under the above conditions. The orthographic viewpoint provides the most intuitive and distortion-free visual presentation of the target object and is used as the baseline viewpoint for texture optimization. Viewpoint parameters under the orthographic viewpoint include the camera's position, orientation, focal length, and field of view. These parameters determine the quality and stability of the rendered image. By obtaining the viewpoint parameters under the orthographic viewpoint, a consistent basis can be provided for texture optimization of other viewpoints.

[0100] As shown in Figure 10a, the world coordinate system is selected as a right-handed coordinate system. The endpoint of ray 1001 is the origin of the world coordinate system, and the angle between ray 1001 and the xOz plane of the world coordinate system is the pitch angle. The endpoint of ray 1002 is the origin of the world coordinate system, and the angle between ray 1002 and the xOy plane of the world coordinate system is the rotation angle. Based on this, the camera's viewing angle parameters in the orthogonal view are: pitch angle of 0°, camera distance of the first distance, and rotation angle of 0°. For example, when the first distance is 1.5, the first camera position in the world coordinate system is coordinate point A.

[0101] Here is an explanation of the implementation process for sampling N1 top-down viewpoints in the world coordinate system: During implementation, N1 coordinate points with a pitch angle of M°, a camera distance of the second distance, and a rotation angle uniformly sampled within 360° from 0° can be selected in the world coordinate system as N1 second camera positions. That is, N1 coordinate points with a pitch angle of M°, a camera distance of the second distance, and a rotation angle uniformly sampled within 360° from 0° are determined as N1 second camera positions in the world coordinate system. The camera is then placed at each of the N1 second camera positions, and the viewpoint of the origin of the world coordinate system captured by the camera at each second camera position is taken as the top-down viewpoint. Then, the viewpoint parameters under the top-down viewpoint are obtained. Let j be any of the N1 top-down viewpoints, where j is a positive integer less than or equal to N1. Then the viewpoint parameters of the j-th top-down viewpoint among the N1 top-down viewpoints include: the top-down viewpoint is M°, the camera distance is the second distance, and the rotation angle is 360° / (N1+1)*j.

[0102] The pitch angle of M° indicates that the camera's line of sight is tilted upwards at a certain angle, forming a top-down view. This top-down view is used to observe the top area of ​​the target object, helping to capture global structural information. The camera distance is set to a second distance, which can be smaller than the first distance to ensure that the camera can get closer to the target object and obtain higher detail resolution. Rotation angles are uniformly sampled from 0° to 360°, resulting in N1 angles. These angles represent N1 camera positions evenly distributed around the target object to cover the complete annular view, thus enabling the acquisition of multi-angle top-down images and improving the comprehensiveness and consistency of texture optimization. The viewpoint parameters under the top-down view include not only the camera's position and orientation but also the maximum visible area of ​​the target object under the top-down view. The maximum visible area refers to the pixel area of ​​the target object's surface visible to the camera under a specific camera viewpoint, and it determines the key areas for texture optimization under that specific camera viewpoint. By acquiring multiple top-down views, the texture of the 3D model can be refined at different heights and orientations, avoiding the local distortion problems caused by a single viewpoint.

[0103] As shown in Figure 10b, assuming M = 10, the second distance is 1.4, and N1 = 15, then the rotation angle of the j-th top-view angle is 360° / (15+1)*j. During the camera view sampling process, a ray 1003 with a top-view angle of 10° is first emitted from the origin of the world coordinate system and intersects the xOz plane; the coordinate point B0 of the second distance is taken on this ray 1003; then, during the rotation (e.g., counterclockwise) around the origin O along the ray 1003, 16 coordinate points are sampled at 360°, such as B0, B1, B2, ..., B15; the point B0 is discarded, and the remaining 15 coordinate points are retained as the 15 second camera positions. Then, the camera takes a picture of the origin O of the world coordinate system from each second camera position (i.e., each coordinate point in B1, B2, ..., B15) as the top-down view. When j = 1, the second camera position of the first top-down view is point B1, and the rotation angle under this first top-down view is 22.5°. When j = 2, the second camera position of the second top-down view is point B2, and the rotation angle under this second top-down view is 45°. And so on. When j = 15, the second camera position of the 15th top-down view is point B15, and the rotation angle under this 15th top-down view is 337.5°.

[0104] The implementation process of sampling N2 upward viewpoints in the world coordinate system is explained below: In the implementation process, N2 coordinate points with a pitch angle of Q°, a camera distance of the third distance, and a rotation angle of N2 angles uniformly sampled from 0° to 360° are selected in the world coordinate system as N2 third camera positions. That is, N2 coordinate points with a pitch angle of Q°, a camera distance of the third distance, and a rotation angle of N2 angles uniformly sampled from 0° to 360° are determined as N2 third camera positions. The viewpoint of the camera at each third camera position relative to the origin of the world coordinate system is taken as the upward viewpoint, and then the viewpoint parameters under the upward viewpoint are obtained. Let any one of the N2 upward viewpoints be denoted as k, where k is a positive integer less than or equal to N2. Then, the viewpoint parameters under the k-th upward viewpoint of the N2 upward viewpoints include: a pitch angle of M°, a camera distance of the third distance, and a rotation angle of 360° / N2*(k-1).

[0105] A pitch angle of Q° indicates that the camera's line of sight is tilted downwards at a certain angle, forming a low-angle view. This low-angle view, created by a pitch angle of Q°, can be used to observe the bottom area of ​​a target object, especially suitable for optimizing details such as the legs and shoes of a character model. A third distance is set, which can be greater than the first distance, to avoid the camera getting too close to the target and causing occlusion or perspective distortion. Rotation angles are uniformly sampled within the range of 0° to 360° to obtain N² angles, ensuring a wraparound low-angle view around the target object, thereby improving the integrity and natural transition of the bottom texture. The view parameters under the low-angle view also include camera position, orientation, and maximum visible area. Since the low-angle view is easily affected by the ground or other obstructions, the relative relationship between the camera height and the target object needs to be considered when setting the low-angle view to ensure that key texture areas are not obstructed by the ground or other objects. By collecting multiple low-angle views, the deficiencies of the frontal and top-angle views in terms of texture detail can be effectively compensated for, thus achieving comprehensive texture optimization of the target object.

[0106] As shown in Figure 10c, assuming Q = -20, the third distance is 1.2, and N2 = 4, then the rotation angle of the k-th upward view is 360° / 4*(k-1). During the camera view sampling process, a ray with a downward angle of -20° between the origin of the world coordinate system and the xOz plane is first emitted from the origin. The coordinate point C0 of the third distance is taken on this ray. Then, during the rotation of the ray around the origin O (e.g., counterclockwise), four coordinate points are sampled at 360°, such as C1, C2, C3, and C4 in sequence. Then, the camera takes the view of the origin O of the world coordinate system from each third camera position (i.e., each coordinate point in C1, C2, C3, and C4) as the upward view. When k = 1, the third camera position of the first upward view is point C1, and the rotation angle under this first upward view is 0°. When k = 2, the third camera position of the second upward view is point C2, and the rotation angle under this third downward view is 90°, and so on. When j = 4, the third camera position of the fourth upward view is point C4, and the rotation angle under this fourth downward view is 270°.

[0107] In summary, based on the specific implementation processes of camera sampling at the frontal angle (Figure 10a), camera sampling at 15 top-down angles (Figure 10b), and camera sampling at 4 bottom-up angles (Figure 10c), the positions of the cameras corresponding to these 20 camera angles in the world coordinate system can be obtained as shown in Figure 10d. As shown in Figure 10d, coordinate point A in the world coordinate system is the first camera position, and the angle from which this camera captures the origin O is the frontal angle. Coordinate points B1, B2, ..., B15 in the world coordinate system are the 15 second camera positions, and the angle from which this camera captures the origin at each of these second camera positions is the top-down angle. Similarly, coordinate points C1, C2, C3, and C4 in the world coordinate system are the third camera positions, and the angle from which this camera captures the origin at each of these third camera positions is the bottom-up angle.

[0108] Based on the above description and sampling of multiple camera viewpoints (e.g., N1+N2+1=20), this embodiment supports a progressive camera viewpoint optimization method. By advancing the camera viewpoints, the second texture optimization model is optimized layer by layer to obtain the third texture optimization model. As can be seen from the above sampling process of multiple camera viewpoints, this embodiment uses the 3D model as the origin of the world coordinate system for camera sampling, and the sampled multiple camera viewpoints comprehensively cover the 3D model in the world coordinate system. Therefore, this embodiment optimizes the second texture optimization model layer by layer based on multiple camera viewpoints, greatly improving the ability of the second texture optimization model to comprehensively optimize the implicit representation of the 3D model. This ensures that the third texture optimization model after the second stage of optimization can comprehensively predict the texture features of the 3D model in the implicit spatial dimension from multiple camera viewpoints.

[0109] In this embodiment, the MLP model is optimized in the second stage primarily based on a viewpoint alternation strategy and a stacking optimization strategy. The viewpoint alternation strategy, also known as the camera viewpoint alternation optimization strategy, indicates the selection rules for choosing the camera viewpoint during each round of iterative optimization. For example, the selection rules indicated by the viewpoint alternation strategy include: starting from the frontal viewpoint, selecting the next camera viewpoint for iterative optimization by alternating between left and right viewpoints (essentially analyzing the relationship between the rotation angles under the top and bottom viewpoints and the rotation angle under the frontal viewpoint to alternately select the camera viewpoint used for the next round of iterative optimization); and continuing iterative optimization of the second texture optimization model optimized by the previous camera viewpoint based on the viewpoint parameters under the selected camera viewpoint, until all camera viewpoints have been selected for iterative optimization of the second texture optimization model, resulting in the final third texture optimization model.

[0110] The stacking optimization strategy indicates the stacking rules for iterative loss information during each iteration of optimization. For example, the stacking rules indicated by the stacking optimization strategy include: during iterative optimization based on the selected camera viewpoint, the loss information corresponding to each selected camera viewpoint is stacked, and a larger loss weight is assigned to the loss information corresponding to the camera viewpoint selected in the current round. Since the loss information corresponding to the previous camera viewpoint (i.e., the camera viewpoint selected for iterative optimization of the MLP model) is stacked in each iteration of optimization, and based on global optimization of the 3D model, the content consistency between the rendered images corresponding to different camera viewpoints is guaranteed, thus ensuring consistent texture optimization.

[0111] The following explanation, with reference to Figure 11, details the implementation process of the second-stage optimization of the MLP model based on the alternating viewpoint strategy and the overlay optimization strategy, to obtain the third texture optimization model. As shown in Figure 11, assume that any one of the multiple camera views acquired from the world coordinate system is represented as the i-th camera viewpoint. This i-th camera viewpoint serves as the sample camera viewpoint used in the i-th iteration of the second texture optimization model. When the total number of multiple camera views is 1 + N1 + N2, the minimum value of i is 0, and the maximum value is N1 + N2, i.e., i is an integer and 0 ≤ i ≤ N1 + N2. The second-stage optimization of the second texture optimization model may include, but is not limited to, steps S21 to S23, where:

[0112] S21: When i = 0, the second texture optimization model is iteratively optimized based on the mesh representation and the viewpoint parameters under the positive viewpoint to obtain the second texture optimization model corresponding to the positive viewpoint.

[0113] In the second-stage optimization process of the MLP model, the computer equipment selects the positive viewpoint from multiple camera views as the sample camera viewpoint for the first round of iterative optimization, following a viewpoint alternation strategy. That is, the camera viewpoint used for the first round of iterative optimization of the second texture optimization model is the 0th camera viewpoint. Each round of iterative optimization includes multiple iterations, such as 24 iterations.

[0114] In specific implementation, when the sample camera viewpoint used in this round of iterative optimization is a frontal viewpoint, the global coordinates (i.e., the geometric information of all mesh vertices) in the mesh representation of the 3D model are input into the second texture optimization model obtained from model preprocessing. Thus, the computer device uses this second texture optimization model to perform texture prediction processing on the global geometric information, obtaining the global second predicted texture features, specifically the second predicted texture features for each mesh vertex. Then, based on the viewpoint parameters under the frontal viewpoint, the second predicted texture features are subjected to differentiable rendering processing to obtain the second predicted rendered image, which serves as the predicted value for this iterative optimization. The specific implementation process of the differentiable rendering processing can be found in the relevant description of the specific implementation process of the differentiable rendering processing shown in step S402 above, and will not be repeated here.

[0115] Similarly, considering that the reference information (provided by the user or defaulted to by the computer device) in this embodiment describes an object from a frontal view, i.e., the reference information describes an object from a frontal view, such as when the reference image is a portrait image, the object in the portrait image is a frontal portrait. Therefore, the 3D model obtained by using the object generation model to perform 3D reconstruction of the object described by the reference image has relatively accurate texture information from a frontal view. Thus, the computer device performs differentiable rendering processing on the global texture information of the 3D model based on the viewpoint parameters from a frontal view, and the resulting second realistic rendered image is used as the true value for this iteration of optimization. Finally, the second texture optimization model is optimized in the direction of reducing the difference between the second predicted rendered image and the second realistic rendered image to obtain the second texture optimization model corresponding to the frontal view. It should be noted that the specific content of optimizing the second texture optimization model in the direction of reducing the difference between the second predicted rendered image and the second realistic rendered image can be found in the aforementioned description of optimizing the first texture optimization model in the direction of reducing the difference between the first predicted rendered image and the first realistic rendered image, and will not be repeated here.

[0116] As shown in Figure 12a, the global coordinates 1202 of the 3D model 1201 are input into the preprocessed second texture optimization model (i.e., the preprocessed MLP model). This second texture optimization model performs texture prediction processing on the global coordinates 1202 to obtain the second predicted texture feature 1203. Then, using the viewpoint parameters under the positive viewpoint, the second predicted texture feature 1203 is subjected to differentiable rendering processing to obtain the second predicted rendered image 1204. Simultaneously, using the viewpoint parameters under the positive viewpoint, the real texture information 1205 of the mesh vertices of the 3D model is subjected to differentiable rendering processing to obtain the second real rendered image 1206. The loss value (i.e., loss information) between the second predicted rendered image 1204 and the second real rendered image 1206 is calculated using the loss function 1207, and the model parameters of the second texture optimization model are adjusted using this loss value to obtain the second texture optimization model after this iteration. Then, the same logic (viewpoint parameters under the positive viewpoint) is used to perform multiple iterations of model optimization on the second texture optimization model to obtain the second texture optimization model corresponding to the positive viewpoint.

[0117] Therefore, by iteratively optimizing the second texture optimization model based on the viewpoint parameters under the positive viewpoint, the second texture optimization model can be guided to learn the texture optimization capability of the 3D model surface under the positive viewpoint.

[0118] S22: Let i = i+1, select the i-th camera view from N1 top-down views and N2 bottom-up views according to the viewpoint alternation strategy, and iteratively optimize the second texture optimization model corresponding to the i-1 camera view according to the overlay optimization strategy based on the mesh representation, the viewpoint parameters of the i-th camera view and the redrawn image corresponding to the i-th camera view, to obtain the second texture optimization model corresponding to the i-th camera view.

[0119] After performing the first round of iterative optimization on the preprocessed second texture optimization model using the frontal view to obtain the second texture optimization model corresponding to the frontal view, this embodiment of the application selects one top view from N1 top view as the i-th camera view according to the view alternation strategy, and continues to use the i-th camera view to perform the second stage optimization on the second texture optimization model until all N1 top view views have been selected. Then, following the view alternation strategy, one bottom view is selected from N2 bottom view as the i-th camera view, and continues to use the i-th camera view to perform the second stage optimization on the second texture optimization model until all N2 bottom view views have been selected.

[0120] It should be noted that the above only provides a brief introduction to the viewpoint alternation strategy. The following describes the specific implementation process of the selection rules indicated by the viewpoint alternation strategy. The selection rules indicated by the viewpoint alternation strategy include: when 0 < i ≤ N1, based on the positional relationship between the first and second rotation angles, alternately select one top-down viewpoint from N1 top-down viewpoints as the i-th camera viewpoint, where the first rotation angle is the rotation angle under each of the N1 top-down viewpoints, and the second rotation angle is the rotation angle under the front viewpoint. When N1 < i ≤ N1 + N2, based on the positional relationship between the third and second rotation angles, alternately select one bottom-up viewpoint from N2 bottom-up viewpoints as the i-th camera viewpoint, where the third rotation angle is the rotation angle under each of the N2 bottom-up viewpoints.

[0121] When 0 < i ≤ N1, taking the 20 camera views shown in Figure 10d as an example, in the world coordinate system shown in Figure 12b, coordinate point A corresponds to the frontal view, and coordinate points B1, ..., B15 correspond to 15 top-down views. When i = 1, from the 15 top-down views corresponding to coordinate points B1, ..., B15, the top-down view corresponding to coordinate point B1, which is located to the left of coordinate point A corresponding to the frontal view and has the smallest angular difference between its rotation angle and the rotation angle under the frontal view, is selected as the first camera view. When i = 2, considering that a top-down view located to the left of coordinate point A corresponding to the frontal view was selected in the previous iteration, in this iteration, a top-down view is alternately selected from the right of coordinate point A corresponding to the frontal view. This can be the top-down view corresponding to coordinate point B15, which is located to the right of coordinate point A corresponding to the frontal view and has the smallest angular difference between its rotation angle and the rotation angle under the frontal view, as the second camera view. When i=3, considering that a top-down viewpoint located to the right of coordinate point A corresponding to the frontal viewpoint was selected in the previous iteration, this iteration alternately selects a top-down viewpoint located to the left of coordinate point A corresponding to the frontal viewpoint. This could be the top-down viewpoint corresponding to coordinate point B2, located to the left of coordinate point A corresponding to the frontal viewpoint, with the smallest angular difference between its rotation angle and the rotation angle under the frontal viewpoint (excluding the already selected coordinate point B1), as the 3rd camera viewpoint. This continues until the top-down viewpoint corresponding to coordinate point B8 is selected as the 15th camera viewpoint, used for iterative optimization of the second texture optimization model corresponding to the 14th camera viewpoint.

[0122] When N1 < i ≤ N1 + N2, taking the 20 camera viewpoints shown in Figure 10d as an example, in the world coordinate system shown in Figure 12c, coordinate point A corresponds to the frontal viewpoint, and coordinate points C1, ..., C4 correspond to four upward viewpoints. When i = N1 + 1, i.e., i = 16, from the four upward viewpoints corresponding to coordinate points C1, ..., C4, the upward viewpoint corresponding to coordinate point C1 (the rotation angle of coordinate point C1 in the upward viewpoint is 0°, and the rotation angle in the front viewpoint is also 0°, with a 0° difference between the two) is selected as the 16th camera viewpoint. When i = 17, from coordinate points C2, C3, and C4, the upward viewpoint corresponding to C2, which is located to the left of coordinate point C1 and has the smallest angular difference between its rotation angle and the rotation angle in the upward viewpoint corresponding to coordinate point C1, is selected as the 17th camera viewpoint. When i = 18, the 18th camera view is selected from coordinate points C3 and C4, which is located to the right of coordinate point C1 and has the smallest angular difference between its rotation angle and the rotation angle of the corresponding upward view of coordinate point C1. When i = 19, the upward view of the unselected coordinate point C4 is selected as the 19th camera view.

[0123] In summary, according to the perspective alternation strategy provided in the embodiments of this application, when there are 20 camera perspectives as shown in Figure 10d, the second texture optimization model will undergo 20 rounds of iterative optimization, with each round including multiple iterations (e.g., 24 times). Furthermore, the camera perspectives used in the 20 rounds of iterative optimization are as follows: the 0th camera perspective is the frontal view corresponding to coordinate point A, the 1st camera perspective is the top-down view corresponding to coordinate point B1, the 2nd camera perspective is the top-down view corresponding to coordinate point B15, the 3rd camera perspective is the top-down view corresponding to coordinate point B2, the 4th camera perspective is the top-down view corresponding to coordinate point B14, ..., the 15th camera perspective is the top-down view corresponding to coordinate point B8, the 16th camera perspective is the bottom-up view corresponding to coordinate point C1, the 17th camera perspective is the bottom-up view corresponding to coordinate point C2, the 18th camera perspective is the bottom-up view corresponding to coordinate point C4, and the 19th camera perspective is the bottom-up view corresponding to coordinate point C3. It is not difficult to see that, on the one hand, these 20 camera viewpoints can fully cover the 3D model in the world coordinate system. Therefore, by using these 20 camera viewpoints to optimize the second texture optimization model layer by layer, the user can effectively improve the ability of the second texture optimization to comprehensively optimize the texture of the 3D model. On the other hand, as mentioned above, the reference information describes the object from the frontal viewpoint. The texture optimization of the 3D model in this embodiment aims to ensure that the texture of the optimized 3D model is consistent with the texture of the object from the frontal viewpoint. Therefore, this embodiment uses a viewpoint alternation strategy, starting from the frontal viewpoint and alternating between left and right camera viewpoints for layer-by-layer texture optimization. This is equivalent to each iteration using a camera viewpoint close to the frontal viewpoint for optimization, which can effectively ensure that the texture optimized layer by layer is consistent with the texture of the object described in the reference information, thereby improving the consistency between the reconstructed 3D model and the object described in the reference information (e.g., the 3D model from the frontal viewpoint looks like the object described in the reference information).

[0124] In this embodiment of the application, after selecting the i-th camera view from the remaining camera views other than the front view based on the viewpoint alternation strategy described above, the computer device will iteratively optimize the second texture optimization model corresponding to the (i-1)-th camera view according to the overlay optimization strategy based on the mesh representation, the viewpoint parameters of the i-th camera view, and the redrawn image corresponding to the i-th camera view, to obtain the second texture optimization model corresponding to the i-th camera view.

[0125] It should be noted that the above only provides a brief introduction to the overlay optimization strategy. The following section describes the specific implementation process of the overlay rules indicated by the overlay optimization strategy. Specifically, the overlay rules indicated by the overlay optimization strategy stipulate that when iteratively optimizing the second texture optimization model corresponding to the (i-1)th camera viewpoint using the i-th camera viewpoint, the loss information of the preceding camera viewpoints of the i-th camera viewpoint out of 1+N1+N2 camera viewpoints is overlaid for model optimization. For example, if 1+N1+N2=20, the preceding camera viewpoints of the i-th camera viewpoint refer to the camera viewpoints selected for iterative optimization out of the 20 camera viewpoints.

[0126] Following the overlay optimization strategy, based on the mesh representation, the viewpoint parameters of the i-th camera view, and the redrawn image, the second texture optimization model corresponding to the (i-1)-th camera view is iteratively optimized to obtain the specific implementation process of the second texture optimization model corresponding to the i-th camera view. This includes: During the iterative optimization of the second texture optimization model corresponding to the (i-1)-th camera view using the i-th camera view, the first loss information of the second texture optimization model corresponding to the (i-1)-th camera view under the positive view is calculated based on the mesh representation and the viewpoint parameters under the positive view. In specific implementation, the second texture optimization model corresponding to the (i-1)-th camera view can be used to perform texture prediction processing on the global coordinates of the 3D model to obtain predicted reference texture features; based on the viewpoint parameters under the positive view, the predicted reference texture features are subjected to differentiable rendering processing to obtain the predicted reference rendered image. Simultaneously, based on the viewpoint parameters under the positive viewpoint, the global texture information of the 3D model is subjected to differentiable rendering processing to obtain a real reference rendered image. Based on the predicted reference rendered image and the real reference rendered image, the first loss information of the second texture optimization model corresponding to the (i-1)th camera viewpoint under the positive viewpoint is calculated. The specific implementation process of the differentiable rendering processing can be found in the aforementioned description and will not be repeated here. Then, based on the mesh representation, the viewpoint parameters of each of the i selected target camera views out of the N1+N2 camera views, and the redrawn image corresponding to each target camera viewpoint, the computer device calculates the second loss information of the second texture optimization model corresponding to the (i-1)th camera viewpoint under each target camera viewpoint. In other words, for the i-th camera view and the i-1 camera views (excluding the front view) in the preceding camera view of the i-th camera view (i.e., the i-1 camera views selected for iterative optimization from the N1 top view and N2 bottom view), the i target camera views are all used to optimize the second texture optimization model corresponding to the i-1 camera view; the second texture optimization model corresponding to the i-1 camera view predicts the second loss information under each of the i target camera views, resulting in a total of i second loss information.

[0127] In detail, assuming any one of the i target camera viewpoints is represented as the target reference camera viewpoint, the calculation process of the second loss information of the second texture optimization model corresponding to the (i-1)th camera viewpoint under the target reference camera viewpoint includes: using the second texture optimization model corresponding to the (i-1)th camera viewpoint, performing texture prediction processing on the global coordinates (i.e., global geometric information) of the 3D model to obtain the third predicted texture feature of the 3D model, i.e., the third predicted texture feature of each mesh vertex in the 3D model. Then, according to the viewpoint parameters under the target reference camera viewpoint, performing differentiable rendering processing on the third predicted texture feature to obtain the third predicted rendered image, which is used as the predicted value for this iteration optimization. For the true value of this iteration optimization, since the object described by the reference information is under the normal viewpoint and not under the target reference camera viewpoint, this embodiment introduces a redrawing algorithm to redraw the third predicted rendered image to obtain the redrawn image corresponding to the target reference camera viewpoint, which is used as the true value for this iteration optimization. The redrawing algorithm refers to a method of redrawing an image, aiming to modify the original image by changing colors and details to achieve the desired image effect. Therefore, using the redrawing algorithm to redraw the third predicted rendering image results in a redrawn image with better image texture than the third predicted rendering image, making the redrawn image a reliable ground truth. Then, based on the third predicted rendering image and the redrawn image, the second loss information of the second texture optimization model corresponding to the (i-1)th camera viewpoint under the target reference camera viewpoint can be calculated.

[0128] Here, the specific process of optimizing the second texture optimization model corresponding to the (i-1)th camera viewpoint for each of the i target camera viewpoints is the same as the specific process of optimizing the second texture optimization model corresponding to the (i-1)th camera viewpoint described above for the target reference camera viewpoint, except that the target camera viewpoint used in the differentiable rendering process is different; therefore, the specific process of optimizing the second texture optimization model corresponding to the (i-1)th camera viewpoint for each of the i target camera viewpoints will not be described here.

[0129] To improve the image quality of the redrawn images, embodiments of this application support dynamically setting different redrawing regions for the third predicted rendering images corresponding to different camera viewpoints during the progressive selection process. The redrawing strategy / algorithm includes: during the alternating selection of multiple camera viewpoints, for the first W1 selected camera viewpoints, the redrawing region is set to 3 / 4 of the third predicted rendering image, where W1 is an integer and 1 ≤ W1 < N1 + N2. A stable diffusion model combined with prompt word engineering is used to redraw the set 3 / 4 region of the third predicted rendering image, resulting in the redrawn image corresponding to each of the W1 camera viewpoints. For the subsequently selected W2 camera viewpoints, the redrawing region is set to 1 / 2 of the third predicted rendering image, where W1 = 1 + N1 + N2 - W1. A stable diffusion model combined with prompt word engineering is used to redraw the set 1 / 2 region of the third predicted rendering image, resulting in the redrawn image corresponding to each of the W2 camera viewpoints. The stable diffusion model is a generative model based on the diffusion process, capable of denoising noisy images and gradually converting them into non-noisy images containing data. The cue word engineering is used to introduce cue words generated during the image redrawing process of the third predicted rendered image based on the stable diffusion model, helping the stable diffusion model generate redrawn images faster and better. Therefore, this embodiment of the application uses the stable diffusion model as the redrawing basis model during the image redrawing process, and simultaneously performs image redrawing based on cue word engineering, greatly improving the stability of the redrawing effect.

[0130] As shown in Figure 13, assuming the total number of camera views excluding the frontal view which does not require redrawing is 19 (e.g., 15 top-down views and 4 bottom-up views), W1 = 4, W2 = 15, then for the first 4 selected top-down views, the redrawing area is determined to be 3 / 4 of the area in the third predicted rendering image. For example, dividing it by left and right, the right side of the third predicted rendering image, which occupies 3 / 4 of the total area, is used as the redrawing area. For the subsequent 15 selected camera views, including 11 top-down views and 4 bottom-up views, for the 11 top-down views, the right side of the third predicted rendering image, occupying 1 / 2 of the total area, is used as the redrawing area; for the 4 bottom-up views, the top and bottom half of the third predicted rendering image (e.g., the top or bottom half) is used as the redrawing area.

[0131] After obtaining the second loss information under each of the i target camera views, totaling i second loss information, the computer device performs weight calculation on the first loss information and the i second loss information to obtain the target loss information of the second texture optimization model corresponding to the (i-1)th camera view.

[0132] Based on the foregoing, after performing any iteration optimization on the second texture optimization model corresponding to the (i-1)th camera view using the i-th camera view, we can obtain the first loss information of the second texture optimization model corresponding to the (i-1)th camera view under the positive view, and the second loss information under each of the i target camera views. Then, this application embodiment supports assigning different loss weights to the first loss information and the i second loss information in this round of iteration optimization, so as to perform weight calculation on the first loss information and the i second loss information to obtain the target loss information of the second texture optimization model corresponding to the (i-1)th camera view. The loss function for calculating the target loss information according to the loss weight is as follows: (2)

[0133] in, This is the actual rendered image corresponding to the viewpoint of the i-th camera; for example, when i = 0, For the true reference image rendered at the positive viewpoint, when i=3, Redraw the image corresponding to the third camera viewpoint. This is the predicted rendered image corresponding to the i-th camera viewpoint; for example, when i = 0, For the predicted reference rendering image corresponding to the positive viewpoint, when i=3, This is the third predicted rendered image corresponding to the third camera viewpoint. λ i Loss weights are assigned to the loss information corresponding to the i-th camera viewpoint; for example, when i = 0, λ i For the loss weight of the first loss information corresponding to the positive perspective, when i=3, λ i The loss weight is the second loss information corresponding to the third camera view. In this embodiment, the loss information corresponding to the i-th camera view used in this iteration needs to be assigned a larger loss weight compared to the loss information corresponding to the preceding camera view; the loss weight λ i The larger the value of , the greater the guiding role of the i-th camera viewpoint in the optimization of model parameters in this iteration, and the sum of the loss weights of the front viewpoint and the i-th target camera viewpoints is 1.

[0134] It is the set of pixel weights for the largest visible area of ​​the 3D model under the i-th camera view. In this embodiment, normal estimation is supported to calculate the largest visible area of ​​the 3D model that the camera can render under each camera view (i.e., the largest area of ​​the 3D model that can be seen under the current camera view), which is used as the pixel weight under the view corresponding to the camera view (including the predicted rendering view and the real rendering view).

[0135] The following example illustrates the process of calculating pixel weights under the target reference camera viewpoint using normal estimation, and then calculating the second loss information of the second texture optimization model corresponding to the (i-1)th camera viewpoint under the target reference camera viewpoint. The process includes: First, obtaining the geometric normal map of the 3D model under the target reference camera viewpoint in the world coordinate system. This geometric normal map can be understood as the image captured by the camera of the 3D model, which includes the model pose presented by the 3D model under the current target reference camera viewpoint. For example, if the camera is located at coordinate point D, and the 3D character model is located at the origin of the world coordinate system and faces the positive x-axis of the world coordinate system, then the geometric normal map obtained when the camera captures the world coordinate system origin from coordinate point D includes the left side of the 3D character model. Then, based on the geometric normal of each pixel in the geometric normal map (i.e., the normal vector perpendicular to the plane where the pixel is located) and the geometric normal of the camera corresponding to the viewpoint of the target reference camera (i.e., the normal direction of the camera's optical axis), the normal of the geometric normal map is estimated to obtain the binary image corresponding to the geometric normal map.

[0136] In this embodiment, the normal estimation process includes: the larger the angle between the geometric normal of a pixel and the geometric normal of the camera, the worse the camera's accuracy in capturing the pixel from the current target reference camera's viewpoint (e.g., poor color capture effect), and a smaller pixel weight (ranging from 0 to 1, e.g., an estimated pixel weight of 0.3) is estimated for that pixel in the geometric normal map; conversely, the smaller the angle between the geometric normal of a pixel and the geometric normal of the camera, the higher the camera's accuracy in capturing the pixel from the current target reference camera's viewpoint (e.g., the pixel can be completely captured), and a larger pixel weight (e.g., an estimated pixel weight of 1) is estimated for that pixel in the geometric normal map; and so on, normal estimation is performed for each pixel in the geometric normal map to obtain the pixel weight of each pixel in the binary image. In other words, the binary image can be understood as the aforementioned set of pixel weights, which includes the pixel weight of each pixel in the view corresponding to the current target reference camera's viewpoint. In this way, the computer device uses the pixel weight of each pixel in the binary image to perform loss calculation on the redrawn image corresponding to the target reference camera's viewpoint and the third predicted rendered image (i.e. This yields the second loss information from the target reference camera's perspective.

[0137] After obtaining the target loss information of the second texture optimization model corresponding to the (i-1)th camera view, the computer device optimizes the second texture optimization model corresponding to the (i-1)th camera view in the direction of reducing the target loss information, and obtains the second texture optimization model corresponding to the ith camera view.

[0138] Based on the above, the second texture optimization model corresponding to the (i-1)th camera view is obtained. After obtaining the target loss information under the ith camera view, the computer device can adjust the model parameters of the second texture optimization model corresponding to the (i-1)th camera view in the direction of reducing the target loss information. After the current round of camera view iterations ends, the (i+1)th camera view is selected according to the aforementioned view alternation strategy to continue the second stage optimization of the second texture optimization model corresponding to the ith camera view.

[0139] S23: When i>N1+N2, the second texture optimization model corresponding to the N1+N2th camera viewpoint is determined as the third texture optimization model.

[0140] When i > N1+N2, it indicates that all 1+N1+N2 camera views selected in the world coordinate system have been used for the second stage optimization of the MLP model. Then, the MLP model obtained by iteratively optimizing the MLP model when i = N1+N2, that is, the second texture optimization model corresponding to the N1+N2th camera view, is determined as the final third texture optimization model.

[0141] Therefore, as can be seen from the second-stage optimization process shown in steps S21 to S23 above, in the second-stage optimization of the MLP model in this application embodiment, multiple camera perspectives in the world coordinate system that can fully cover the 3D model are used to perform layer-by-layer texture optimization on the MLP model; for example, the frontal view is used to focus on optimizing the front texture of the 3D model, N1 top-down views are used to focus on optimizing the top and side textures of the 3D model, and N2 bottom-up views are used to focus on optimizing the bottom texture of the 3D model; thus, by using multiple specified camera perspectives, the 3D model texture is optimized in all directions, which greatly improves the quality and stability of texture optimization.

[0142] It should also be noted that, considering the potential inconsistency between the geometric contours of the reconstructed 3D model and the geometric contours of the object described in the reference information when mainstream object generation models reconstruct 3D models based on reference information, this embodiment only performs texture optimization on the consistent geometric contours during the texture optimization process of the 3D model. This effectively reduces the resources consumed by texture optimization, further shortens the time required for texture optimization, and improves texture optimization efficiency. Therefore, before performing the second-stage optimization on the MLP model (at this time, the MLP model is the second texture optimization model after model preprocessing), this embodiment supports using the geometric contour map of the 3D model from a frontal view to perform ablation matching processing on the geometric contour map of the object described in the reference information to obtain the alignment region of the 3D model. This alignment region refers to the area where the geometric contour map of the 3D model and the geometric contour map of the object have consistent contours. In this way, during the second-stage optimization, only the geometric information within the alignment region needs to be input into the MLP model for texture optimization; that is, the input data input into the MLP model in each iteration of optimization in the aforementioned steps S21 to S23 is the geometric information of the alignment region.

[0143] Here, the geometric contour map of a 3D model from a frontal view refers to a pure geometric edge image (i.e., an image without texture, retaining only the model's outline) generated through differentiable rendering of the 3D model from a frontal view (e.g., 0° pitch angle, 0° rotation angle). For example, the geometric contour map of a 3D model from a frontal view could be the frontal view geometric contour map of a cartoon character's 3D model. This geometric contour map would show the edge lines of the head, body, and limbs, but would not include hair color, facial texture, etc. The geometric contour map of the object described in the reference information refers to the geometric contour map corresponding to the reference image input by the user, which can be extracted from the reference image using an edge detection algorithm (such as Canny). For example, inputting a frontal view of a cartoon character and using the Canny algorithm to extract its edge lines yields a reference contour map (e.g., a line drawing of "upturned hair and round eyes").

[0144] Ablation matching refers to comparing the geometric contour map of the 3D model with the geometric contour map of the object described by the reference information, removing inconsistent areas, and retaining areas where the contours completely overlap. For example, if the hair in the reference map is a "curved triangle" while the hair in the 3D model is a "flat rectangle," then the "curved triangle" part is inconsistent and needs to be "ablated" (removed). Aligned regions are the areas retained after ablation matching (the parts where the contour of the 3D model completely matches the contour of the reference map), and are effective areas for subsequent texture optimization. For example, if the facial contours (eye and nose positions) of the reference map and the 3D model are consistent, then the face is an aligned region; if the hair is inconsistent, then the hair is an misaligned region.

[0145] In this embodiment of the application, the process of a computer device using the geometric contour map of the object described by the reference information under a normal viewing angle to perform ablation matching processing on the reconstructed 3D model to obtain the aligned region of the 3D model includes: calculating a first mask image of the geometric contour map of the 3D model under a normal viewing angle, and calculating a second mask image of the geometric contour map of the object described by the reference information; subtracting the first mask image and the second mask image to obtain the misaligned region, which is a region that does not require texture optimization; conversely, the remaining region in the geometric contour map of the 3D model other than the misaligned region is the aligned region.

[0146] Here, the first mask image refers to the binary mask corresponding to the frontal view geometric contour map of the 3D model (contour areas are "1", non-contour areas are "0"). The second mask image refers to the binary mask corresponding to the geometric contour map of the reference image (reference contour areas are "1", non-reference contour areas are "0"). The misaligned area refers to the area where the difference between the first and second mask images is 1 (i.e., the area that is present in the reference image but not in the 3D model or the area that is present in the 3D model but not in the reference image), which is an invalid area for texture optimization.

[0147] S404: The third texture optimization model is used to perform texture prediction processing on the mesh representation to obtain the target texture representation of the three-dimensional model.

[0148] S405: Exports the target texture representation as a texture map resource for a 3D model.

[0149] In steps S404 to S405, a third texture optimization model is obtained after two-stage optimization of the MLP model based on the aforementioned steps S401 to S403. This third texture optimization model has the ability to optimize the texture features of the 3D model in the implicit spatial dimension from multiple camera perspectives. Thus, by inputting the global coordinates in the mesh representation of the 3D model into the third texture optimization model, the third texture optimization model can be used to perform texture prediction processing on the mesh representation to obtain the target texture representation of the 3D model. This target texture representation is used to characterize the texture features of the 3D model after comprehensive optimization in the implicit spatial dimension. Considering that the target texture representation is an abstract texture feature, this embodiment of the application also needs to export the target texture representation into the form of a texture map. This texture map can be understood as a pattern on the surface of the 3D model. By applying the texture map to the surface of the 3D model, texture can be assigned to the 3D model.

[0150] In this embodiment, the target texture representation is mapped to the UV space using color encoding and interpolation to obtain texture map resources. As mentioned above, in addition to geometric and texture information, the mesh vertices of the 3D model also possess vertex texture coordinates, which are represented by two-dimensional variables (U, V) in the UV space; where U represents the horizontal axis and V represents the vertical axis, typically ranging from 0 to 1. The U and V coordinates define the 2D position of each mesh vertex on the model surface of the 3D model in the texture image.

[0151] The process by which a computer device maps a target texture representation to UV space using color encoding and interpolation is illustrated in Figure 14. As shown in Figure 14, the computer device performs coordinate unrolling on the 3D model 1401 to obtain a planar image of the 3D model; this coordinate unrolling is called UV unrolling 1402, which aims to map the surface of the 3D model onto a planar image, facilitating texture painting on this planar image during texture mapping. The computer device also performs texture sampling 1404 on the target texture representation 1403 to obtain texture sampling values; texture sampling refers to the process of obtaining texture sampling values ​​for each position in the target texture representation based on the surface position of the 3D model. During texture sampling, for non-mesh vertex positions in the mesh, texture sampling values ​​are set for these positions through interpolation mapping 1405. For example, known texture sampling values ​​of multiple mesh vertices can be used to estimate / assign texture sampling values ​​for positions on the mesh surface. Next, by using UV filling 1406, the texture sample values ​​are filled onto the planar image to obtain the texture map resource 1407 of the 3D model. For example, the texture sample values ​​can be filled onto the corresponding positions on the planar image through UV filling to construct the obtained texture map resource. This texture map resource can be fitted to the corresponding positions in the 3D model, thereby achieving the goal of giving the 3D model skin texture while optimizing the skin texture quality.

[0152] In summary, on the one hand, the embodiments of this application support multi-level model optimization of the second texture optimization model using the viewpoint parameters of each camera viewpoint in multiple camera perspectives. By progressively optimizing the second texture optimization model's ability to predict the texture features of the 3D model under each camera viewpoint in the world coordinate system, it effectively ensures that the optimized third texture model has the ability to comprehensively optimize the texture features of the 3D model in the implicit spatial dimension. Thus, by using the optimized third texture optimization model to perform texture prediction processing on the mesh representation of the 3D model, a high-quality target texture representation of the 3D model can be obtained. This target texture representation characterizes the texture features of the 3D model after comprehensive optimization in the implicit spatial dimension, thereby enabling a comprehensive quality improvement of the texture mapping resources of the 3D model derived from this target texture representation. On the other hand, after obtaining the mesh representation of the coarse-textured 3D model, this embodiment directly uses the mesh representation to perform step-by-step optimization of the first texture optimization model. This training-free (i.e., the first texture optimization model is used directly in the model inference stage without prior training or fine-tuning) model optimization method requires no additional model training, which not only reduces the cost of generating the 3D model but also enables rapid texture optimization of the 3D model, achieving high-quality texture optimization. Furthermore, this embodiment uses interpolation for texture export, which can effectively reduce texture loss during texture export, thereby improving the quality of texture map resources.

[0153] The foregoing has detailed the method for processing three-dimensional models according to embodiments of this application. To facilitate better implementation of the methods described above, a processing apparatus for three-dimensional models according to embodiments of this application is provided below. In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0154] Figure 15 shows a schematic diagram of a three-dimensional model processing device provided in an embodiment of this application. The three-dimensional model processing device can be used to execute some or all of the steps in the method embodiment shown in Figure 4. Please refer to Figure 15. The device includes: an acquisition unit 1501 configured to acquire a mesh representation of a 3D model, the mesh representation being used to characterize the texture features of the 3D model in the explicit spatial dimension; the 3D model is obtained by 3D reconstruction of an object in reference information; a processing unit 1502 configured to perform model preprocessing on a first texture optimization model based on the mesh representation to obtain a second texture optimization model; the second texture optimization model is used to predict the texture features of the 3D model in the implicit spatial dimension; the processing unit 1502 is further configured to acquire viewpoint parameters from multiple camera perspectives in the world coordinate system, and perform multi-layer texture optimization processing on the second texture optimization model based on the mesh representation and the viewpoint parameters from multiple camera perspectives to obtain a third texture optimization model; the third texture optimization model is used to predict the texture features of the 3D model in the implicit spatial dimension from multiple camera perspectives; the processing unit 1502 is further configured to perform texture prediction processing on the mesh representation using the third texture optimization model to obtain a target texture representation of the 3D model, the target texture representation being used to characterize the optimized texture features of the 3D model in the implicit spatial dimension; the processing unit 1502 is further configured to export the target texture representation as a texture map resource of the 3D model.

[0155] In some embodiments, the mesh representation includes geometric information and texture information of the mesh vertices that make up the three-dimensional model; the processing unit 1502 is further configured to perform texture prediction processing on the geometric information using a first texture optimization model to obtain a first predicted texture feature of the mesh vertex; perform differentiable rendering processing on the first predicted texture feature to obtain a first predicted rendered image; and perform differentiable rendering processing on the texture information to obtain a first real rendered image; and perform model optimization on the first texture optimization model in the direction of reducing the difference between the first predicted rendered image and the first real rendered image to obtain a second texture optimization model.

[0156] In some embodiments, the processing unit 1502 is further configured to obtain view parameters under random camera view in the world coordinate system; obtain an initial rendered image of the 3D model in the world coordinate system based on the view parameters under random camera view; and micro-render the first predicted texture features to the initial rendered image to obtain the first predicted rendered image.

[0157] In some embodiments, the multiple camera viewpoints include: a frontal viewpoint, N1 top-down viewpoints, and N2 bottom-up viewpoints, where N1 and N2 are positive integers; the processing unit 1502 is further configured to determine a first camera position in the world coordinate system with a pitch angle of 0°, a camera distance of a first distance, and a rotation angle of 0°, and to determine the viewpoint of the camera at the origin of the world coordinate system from the first camera position as the frontal viewpoint; obtain viewpoint parameters under the frontal viewpoint; the viewpoint parameters under the frontal viewpoint include a pitch angle of 0°, a camera distance of a first distance, and a rotation angle of 0°; determine N1 second camera positions in the world coordinate system with N1 coordinate points at N1 angles uniformly sampled from 0° to 360° with a pitch angle of M°, a camera distance of a second distance, and a rotation angle of M°, and to determine the viewpoint of the camera at the origin of the world coordinate system from each second camera position as the frontal viewpoint. Top-down viewpoint; obtain the viewpoint parameters under the top-down viewpoint; the viewpoint parameters under the j-th top-down viewpoint among N1 top-down viewpoints include a pitch angle of M°, a camera distance of the second distance, and a rotation angle of 360° / (N1+1)*j, where j is a positive integer less than or equal to N1; in the world coordinate system, determine N2 coordinate points under N2 angles uniformly sampled from 0° to 360° with a pitch angle of Q°, a camera distance of the third distance, and a rotation angle of N2, as N2 third camera positions, and determine the viewpoint of the camera at each third camera position that captures the origin of the world coordinate system as the bottom-up viewpoint; obtain the viewpoint parameters under the bottom-up viewpoint; the viewpoint parameters under the k-th bottom-up viewpoint among N2 bottom-up viewpoints include a pitch angle of Q°, a camera distance of the third distance, and a rotation angle of 360° / N2*(k-1), where k is a positive integer less than or equal to N2.

[0158] In some embodiments, any one of the multiple camera views is represented as the i-th camera view, where i is an integer and 0 ≤ i ≤ N1 + N2; the processing unit 1502 is further configured to, when i = 0, iteratively optimize the second texture optimization model based on the mesh representation and the view parameters under the front view to obtain the second texture optimization model corresponding to the front view; let i = i + 1, select the i-th camera view from N1 top view and N2 bottom view according to the view alternation strategy, and iteratively optimize the second texture optimization model corresponding to the (i-1)-th camera view according to the overlay optimization strategy based on the mesh representation, the view parameters of the i-th camera view, and the redrawn image corresponding to the i-th camera view to obtain the second texture optimization model corresponding to the i-th camera view; when i > N1 + N2, determine the second texture optimization model corresponding to the N1 + N2-th camera view as the third texture optimization model.

[0159] In some embodiments, the reference information describes the object from a frontal viewpoint; the mesh representation includes geometric and texture information of the mesh vertices that make up the 3D model; the processing unit 1502 is further configured to use the second texture optimization model obtained by model preprocessing to perform texture prediction processing on the geometric information to obtain second predicted texture features of the mesh vertices; perform differentiable rendering processing on the second predicted texture features according to the viewpoint parameters from the frontal viewpoint to obtain a second predicted rendered image; perform differentiable rendering processing on the texture information according to the viewpoint parameters from the frontal viewpoint to obtain a second real rendered image; and optimize the second texture optimization model in the direction of reducing the difference between the second predicted rendered image and the second real rendered image to obtain a second texture optimization model corresponding to the frontal viewpoint.

[0160] In some embodiments, the processing unit 1502 is further configured to let i = i + 1; when 0 < i ≤ N1, according to the positional relationship between the first rotation angle and the second rotation angle, alternately select one top view from N1 top view angles as the i-th camera view angle; the first rotation angle is the rotation angle under each of the N1 top view angles, and the second rotation angle is the rotation angle under the front view angle; when N1 < i ≤ N1 + N2, according to the positional relationship between the third rotation angle and the second rotation angle, alternately select one bottom view from N2 bottom view angles as the i-th camera view angle; the third rotation angle is the rotation angle under each of the N2 bottom view angles.

[0161] In some embodiments, the processing unit 1502 is further configured to: determine the first loss information of the second texture optimization model corresponding to the (i-1)th camera view under the frontal view based on the mesh representation and the view parameters under the frontal view; select i target camera views from N1+N2 camera views; for each target camera view, determine the second loss information of the second texture optimization model corresponding to the (i-1)th camera view under the target camera view based on the mesh representation, the view parameters under each target camera view in the target camera view, and the redrawn image corresponding to the target camera view; perform weight calculation on the first loss information and the i second loss information to obtain the target loss information of the second texture optimization model corresponding to the (i-1)th camera view; and perform model optimization on the second texture optimization model corresponding to the (i-1)th camera view in the direction of reducing the target loss information to obtain the second texture optimization model corresponding to the ith camera view.

[0162] In some embodiments, the mesh representation includes geometric and texture information of the mesh vertices constituting the 3D model; the processing unit 1502 is further configured to determine the target camera view as the target reference camera view; use the second texture optimization model corresponding to the (i-1)th camera view to perform texture prediction processing on the geometric information to obtain the third predicted texture features of the mesh vertices; perform differentiable rendering processing on the third predicted texture features according to the view parameters under the target reference camera view to obtain the third predicted rendered image; perform image redrawing on the third predicted rendered image to obtain the redrawn image corresponding to the target reference camera view; and determine the second loss information of the second texture optimization model corresponding to the (i-1)th camera view under the target reference camera view based on the redrawn image and the third predicted rendered image.

[0163] In some embodiments, the position of the 3D model in the world coordinate system is the origin of the world coordinate system; the processing unit 1502 is further configured to obtain the geometric normal map corresponding to the 3D model from the perspective of the target reference camera in the world coordinate system; perform normal estimation on the geometric normal map based on the geometric normal of each pixel in the geometric normal map and the geometric normal of the camera corresponding to the perspective of the target reference camera to obtain a binary image corresponding to the geometric normal map; the binary image includes the pixel weight of each pixel; and use the pixel weight of each pixel in the binary image to perform loss calculation on the redrawn image and the third predicted rendering image to obtain the second loss information from the perspective of the target reference camera.

[0164] In some embodiments, the processing unit 1502 is further configured to perform texture sampling on the target texture representation to obtain texture sampling values; perform coordinate unfolding processing on the three-dimensional model to obtain a planar image; and fill the planar image with the texture sampling values ​​to obtain the texture map resource of the three-dimensional model.

[0165] In some embodiments, the apparatus further includes: an ablation matching module configured to, after performing model preprocessing on the first texture optimization model according to the mesh representation to obtain the second texture optimization model, acquire the geometric contour map of the three-dimensional model from a frontal view; based on the geometric contour map of the three-dimensional model, perform ablation matching processing on the geometric contour map of the object described by the reference information to obtain an alignment region of the three-dimensional model; the alignment region is a region where the contours of the geometric contour map of the three-dimensional model and the geometric contour map of the object are consistent; correspondingly, the processing unit 1502 is further configured to perform multi-layer texture optimization processing on the second texture optimization model based on the geometric information in the alignment region, the mesh representation, and the viewpoint parameters under multiple camera views to obtain a third texture optimization model.

[0166] In some embodiments, the ablation matching module is further configured to determine a first mask image of the geometric contour map of the 3D model from a positive perspective, and a second mask image of the geometric contour map of the object described by the reference information; subtract the first mask image and the second mask image to obtain the misaligned region; the misaligned region is the region that does not require texture optimization; in the geometric contour map of the 3D model, the remaining region other than the misaligned region is the aligned region.

[0167] According to one embodiment of this application, the various units in the three-dimensional model processing device shown in FIG15 can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This can achieve the same operation without affecting the technical effect of the embodiment of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the three-dimensional model processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by multiple units working together. According to another embodiment of this application, the three-dimensional model processing device shown in FIG15 and the three-dimensional model processing method of the embodiment of this application can be constructed by running a computer program (including program code) capable of executing the steps involved in the corresponding method shown in FIG4 on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), RAM, and ROM. Computer programs may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computer device via the computer-readable recording medium, and run therein.

[0168] In this embodiment, the computer device uses each camera viewpoint in the world coordinate system as an optimization level. Utilizing the viewpoint parameters of each camera viewpoint, it performs multi-level model optimization on the second texture optimization model. By progressively optimizing the second texture optimization model's predictive ability for the texture features of the 3D model under each camera viewpoint (e.g., frontal view, top view, and bottom view) in the world coordinate system, it effectively ensures that the optimized third texture model possesses the ability to comprehensively optimize the texture features of the 3D model in the implicit spatial dimension. Furthermore, this embodiment directly utilizes the mesh representation of the 3D model to perform step-by-step optimization of the first texture optimization model. This training-free (i.e., the first texture optimization model is used directly in the model inference stage without prior training or fine-tuning) model optimization method eliminates the need for additional model training. This not only reduces the cost of generating the 3D model but also enables rapid texture optimization of the 3D model, achieving high-quality texture optimization.

[0169] Figure 16 shows a schematic diagram of a computer device provided in an exemplary embodiment of this application. Referring to Figure 16, the computer device includes a processor 1601, a communication interface 1602, and a computer-readable storage medium 1603. The processor 1601, communication interface 1602, and computer-readable storage medium 1603 can be connected via a bus or other means. The communication interface 1602 is used to receive and send data. The computer-readable storage medium 1603 can be stored in the memory of the computer device. The computer-readable storage medium 1603 is used to store computer programs, including program instructions. The processor 1601 is used to execute the program instructions stored in the computer-readable storage medium 1603. The processor 1601 (or Central Processing Unit (CPU)) is the computing and control core of the computer device, adapted to implement one or more instructions, specifically adapted to load and execute one or more instructions to implement corresponding method flows or corresponding functions.

[0170] This application embodiment also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by the processor 1601, which may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM memory or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0171] In one embodiment, the computer-readable storage medium stores one or more computer-executable instructions or computer programs; the processor 1601 loads and executes the one or more computer-executable instructions or computer programs stored in the computer-readable storage medium to implement the corresponding steps in the above-described embodiment of the three-dimensional model processing method.

[0172] Based on the same inventive concept, the principle and beneficial effects of the computer device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the three-dimensional model processing method in the embodiments of this application in solving the problem. Please refer to the principle and beneficial effects of the implementation of the method. For the sake of brevity, they will not be repeated here.

[0173] This application also provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the computer device to perform the aforementioned three-dimensional model processing method.

[0174] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0175] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0176] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing a three-dimensional model, the method being executed by a computer device, the method comprising: Obtain a mesh representation of the 3D model, wherein the mesh representation is used to characterize the texture features of the 3D model in the explicit spatial dimension; The three-dimensional model is obtained by reconstructing the object described by the reference information in three dimensions. The first texture optimization model is preprocessed according to the mesh representation to obtain the second texture optimization model; the second texture optimization model is used to predict the texture features of the three-dimensional model in the implicit spatial dimension. The viewpoint parameters from multiple camera perspectives in the world coordinate system are obtained, and the second texture optimization model is subjected to multi-layer texture optimization processing based on the mesh representation and the viewpoint parameters from multiple camera perspectives to obtain a third texture optimization model; the third texture optimization model is used to predict the texture features of the three-dimensional model in the implicit spatial dimension from the multiple camera perspectives. The third texture optimization model is used to perform texture prediction processing on the mesh representation to obtain the target texture representation of the three-dimensional model. The target texture representation is used to characterize the texture features of the three-dimensional model after optimization in the implicit space dimension. The target texture representation is exported as a texture map resource for the 3D model.

2. The method as described in claim 1, wherein, The mesh representation includes geometric and texture information of the mesh vertices that make up the 3D model; the step of preprocessing the first texture optimization model according to the mesh representation to obtain the second texture optimization model includes: The first texture optimization model is used to perform texture prediction processing on the geometric information to obtain the first predicted texture features of the mesh vertices; The first predicted texture feature is subjected to differentiable rendering processing to obtain a first predicted rendered image; and the texture information is subjected to differentiable rendering processing to obtain a first real rendered image. The first texture optimization model is optimized in the direction of reducing the difference between the first predicted rendered image and the first real rendered image to obtain the second texture optimization model.

3. The method as described in claim 1 or 2, wherein, The step of performing differentiable rendering processing on the first predicted texture features to obtain the first predicted rendered image includes: Obtain the viewpoint parameters under a random camera viewpoint in the world coordinate system; Based on the perspective parameters under the random camera view, obtain the initial rendered image of the 3D model in the world coordinate system; The first predicted texture feature is micro-rendered into the initial rendered image to obtain the first predicted rendered image.

4. The method as described in any one of claims 1 to 3, wherein, The multiple camera viewpoints include: one frontal viewpoint, N1 top-down viewpoints, and N2 bottom-up viewpoints, where N1 and N2 are positive integers; obtaining the viewpoint parameters of the multiple camera viewpoints in the world coordinate system includes: In the world coordinate system, the coordinate point with a pitch angle of 0°, a camera distance of the first distance, and a rotation angle of 0° is determined as the first camera position, and the angle from which the camera at the first camera position captures the origin of the world coordinate system is determined as the orthogonal angle; the angle parameters under the orthogonal angle are obtained; the angle parameters under the orthogonal angle include a pitch angle of 0°, a camera distance of the first distance, and a rotation angle of 0°; In the world coordinate system, N1 coordinate points with a pitch angle of M°, a camera distance of the second distance, and a rotation angle uniformly sampled from 0° to 360° are determined as N1 second camera positions. The viewpoint of the camera at each second camera position that captures the origin of the world coordinate system is determined as the top-down viewpoint. The viewpoint parameters under the top-down viewpoint are obtained. The viewpoint parameters under the j-th top-down viewpoint among the N1 top-down viewpoints include a pitch angle of M°, a camera distance of the second distance, and a rotation angle of 360° / (N1+1)*j, where j is a positive integer less than or equal to N1. In the world coordinate system, N2 coordinate points with a pitch angle of Q°, a camera distance of the third distance, and a rotation angle uniformly sampled from 0° to 360° are determined as N2 third camera positions. The viewing angle of the camera at each of the third camera positions, which is taken from the origin of the world coordinate system, is determined as the upward viewing angle. The viewing angle parameters under the upward viewing angle are obtained. The viewing angle parameters under the k-th upward viewing angle among the N2 upward viewing angles include a pitch angle of Q°, a camera distance of the third distance, and a rotation angle of 360° / N2*(k-1), where k is a positive integer less than or equal to N2.

5. The method according to any one of claims 1 to 4, wherein, Any one of the multiple camera viewpoints is represented as the i-th camera viewpoint, where i is an integer and 0 ≤ i ≤ N1 + N2; the step of performing multi-layer texture optimization processing on the second texture optimization model based on the mesh representation and the viewpoint parameters under the multiple camera viewpoints to obtain the third texture optimization model includes: When i = 0, the second texture optimization model is iteratively optimized based on the mesh representation and the view parameters under the positive view to obtain the second texture optimization model corresponding to the positive view. Let i = i+1, select the i-th camera view from N1 top-down views and N2 bottom-up views according to the viewpoint alternation strategy, and based on the mesh representation, the viewpoint parameters of the i-th camera view, and the redrawn image corresponding to the i-th camera view, iteratively optimize the second texture optimization model corresponding to the (i-1)-th camera view according to the superposition optimization strategy to obtain the second texture optimization model corresponding to the i-th camera view. When i>N1+N2, the second texture optimization model corresponding to the N1+N2th camera viewpoint is determined as the third texture optimization model.

6. The method according to any one of claims 1 to 5, wherein, The reference information describes the object from the positive perspective; the mesh representation includes geometric and texture information of the mesh vertices that make up the 3D model; The step of iteratively optimizing the second texture optimization model based on the mesh representation and the viewpoint parameters under the positive viewpoint to obtain the second texture optimization model corresponding to the positive viewpoint includes: Using the second texture optimization model obtained from the model preprocessing, texture prediction processing is performed on the geometric information to obtain the second predicted texture features of the mesh vertices; Based on the viewpoint parameters under the positive viewpoint, the second predicted texture feature is subjected to differentiable rendering processing to obtain the second predicted rendered image; Based on the perspective parameters under the positive perspective, the texture information is subjected to differentiable rendering processing to obtain a second realistic rendering image; The second texture optimization model is optimized in the direction of reducing the difference between the second predicted rendered image and the second real rendered image to obtain the second texture optimization model corresponding to the positive viewpoint.

7. The method according to any one of claims 1 to 6, wherein, The step of setting i = i + 1 and selecting the i-th camera view from N1 top-down views and N2 bottom-up views according to the viewpoint alternation strategy includes: Let i=i+1; When 0 < i ≤ N1, according to the positional relationship between the first rotation angle and the second rotation angle, a top-down view is alternately selected from N1 top-down viewpoints as the i-th camera viewpoint; the first rotation angle is the rotation angle under each of the N1 top-down viewpoints, and the second rotation angle is the rotation angle under the frontal viewpoint; When N1<i≤N1+N2, according to the positional relationship between the third rotation angle and the second rotation angle, an upward viewpoint is alternately selected from N2 upward viewpoints as the i-th camera viewpoint; the third rotation angle is the rotation angle under each of the N2 upward viewpoints.

8. The method according to any one of claims 1 to 7, wherein, The step of iteratively optimizing the second texture optimization model corresponding to the (i-1)th camera viewpoint based on the mesh representation, the viewpoint parameters of the i-th camera viewpoint, and the redrawn image corresponding to the i-th camera viewpoint, according to the overlay optimization strategy, to obtain the second texture optimization model corresponding to the i-th camera viewpoint, includes: Based on the mesh representation and the view parameters under the positive view, determine the first loss information of the second texture optimization model corresponding to the (i-1)th camera view under the positive view; Select i target camera views from N1+N2 camera views; For each target camera viewpoint, based on the mesh representation, the viewpoint parameters under the target camera viewpoint, and the redrawn image corresponding to the target camera viewpoint, the second loss information of the second texture optimization model corresponding to the (i-1)th camera viewpoint under the target camera viewpoint is determined. Weights are calculated for the first loss information and the i second loss information to obtain the target loss information of the second texture optimization model corresponding to the (i-1)th camera viewpoint; In accordance with the direction of reducing the target loss information, the second texture optimization model corresponding to the (i-1)th camera viewpoint is optimized to obtain the second texture optimization model corresponding to the i-th camera viewpoint.

9. The method according to any one of claims 1 to 8, wherein, The mesh representation includes geometric and texture information of the mesh vertices that make up the 3D model; determining the second loss information of the second texture optimization model corresponding to the (i-1)th camera view under the target camera view, based on the mesh representation, the view parameters under the target camera view, and the redrawn image corresponding to the target camera view, includes: The target camera's viewpoint is defined as the target reference camera's viewpoint; The second texture optimization model corresponding to the (i-1)th camera viewpoint is used to perform texture prediction processing on the geometric information to obtain the third predicted texture features of the mesh vertices; Based on the viewpoint parameters under the target reference camera viewpoint, the third predicted texture features are subjected to differentiable rendering processing to obtain the third predicted rendered image. The third predicted rendered image is redrawn to obtain the redrawn image corresponding to the viewpoint of the target reference camera; Based on the redrawn image and the third predicted rendered image, determine the second loss information of the second texture optimization model corresponding to the (i-1)th camera viewpoint under the target reference camera viewpoint.

10. The method according to any one of claims 1 to 9, wherein, The position of the 3D model in the world coordinate system is the origin of the world coordinate system; determining the second loss information of the second texture optimization model corresponding to the (i-1)th camera viewpoint under the target reference camera viewpoint based on the redrawn image and the third predicted rendered image includes: In the world coordinate system, obtain the geometric normal map corresponding to the three-dimensional model from the perspective of the target reference camera; Based on the geometric normals of each pixel in the geometric normal map and the geometric normals of the camera corresponding to the viewpoint of the target reference camera, normal estimation is performed on the geometric normal map to obtain a binary image corresponding to the geometric normal map; the binary image includes the pixel weight of each pixel. Using the pixel weights of each pixel in the binary image, loss calculations are performed on the redrawn image and the third predicted rendering image to obtain the second loss information from the perspective of the target reference camera.

11. The method according to any one of claims 1 to 10, wherein, The step of exporting the target texture representation as a texture map resource of the 3D model includes: The target texture representation is subjected to texture sampling to obtain texture sample values; The three-dimensional model is subjected to coordinate unfolding to obtain a planar image; The texture sampling values ​​are filled into the planar image to obtain the texture map resource of the three-dimensional model.

12. The method as claimed in any one of claims 1 to 11, wherein, After performing model preprocessing on the first texture optimization model based on the mesh representation to obtain the second texture optimization model, the method further includes: Obtain the geometric contour diagram of the 3D model from a positive perspective; Based on the geometric contour map of the 3D model, the geometric contour map of the object described by the reference information is subjected to ablation matching processing to obtain the alignment region of the 3D model; the alignment region is the area where the geometric contour map of the 3D model and the geometric contour map of the object have the same contour. Accordingly, the step of performing multi-layer texture optimization processing on the second texture optimization model based on the mesh representation and the viewpoint parameters under multiple camera views to obtain the third texture optimization model includes: Based on the geometric information in the alignment region, the mesh representation, and the viewpoint parameters under multiple camera views, the second texture optimization model is subjected to multi-layer texture optimization processing to obtain the third texture optimization model.

13. The method as claimed in any one of claims 1 to 12, wherein, The geometric contour map based on the 3D model is used to perform ablation matching processing on the geometric contour map of the object described by the reference information to obtain the alignment region of the 3D model, including: A first mask image is used to determine the geometric contour map of the 3D model under the positive view, and a second mask image is used to determine the geometric contour map of the object described by the reference information; Subtracting the first mask image from the second mask image yields the misaligned region; the misaligned region is the region that does not require texture optimization; in the geometric contour map of the 3D model, the remaining region other than the misaligned region is the aligned region.

14. A three-dimensional model processing device, comprising: The acquisition unit is configured to acquire a mesh representation of a three-dimensional model, wherein the mesh representation is used to characterize the texture features of the three-dimensional model in an explicit spatial dimension; The three-dimensional model is obtained by reconstructing the object in the reference information in three dimensions. The processing unit is configured to perform model preprocessing on the first texture optimization model according to the mesh representation to obtain a second texture optimization model; the second texture optimization model is used to predict the texture features of the three-dimensional model in the implicit spatial dimension. The processing unit is further configured to acquire view parameters from multiple camera perspectives in the world coordinate system, and perform multi-layer texture optimization processing on the second texture optimization model based on the mesh representation and the view parameters from multiple camera perspectives to obtain a third texture optimization model; the third texture optimization model is used to predict the texture features of the three-dimensional model in the implicit spatial dimension from the multiple camera perspectives. The processing unit is further configured to perform texture prediction processing on the mesh representation using the third texture optimization model to obtain the target texture representation of the three-dimensional model. The target texture representation is used to characterize the texture features of the three-dimensional model after optimization in the implicit space dimension. The processing unit is further configured to export the target texture representation as a texture map resource of the three-dimensional model.

15. A computer device, comprising: Memory is used to store executable instructions or computer programs. A processor, configured to execute computer-executable instructions or computer programs stored in the memory, implements the three-dimensional model processing method as described in any one of claims 1 to 13.

16. A computer-readable storage medium storing computer-executable instructions or a computer program, which, when loaded and executed by a processor, implement the method for processing a three-dimensional model as described in any one of claims 1 to 13.

17. A computer program product comprising computer-executable instructions or a computer program, wherein when the computer-executable instructions or the computer program are executed by a processor, they implement the method for processing a three-dimensional model as described in any one of claims 1 to 13.

Images