Method and apparatus for processing mesh model

Abstract

A method for processing a mesh model, which method is executed by an electronic device. The method comprises: acquiring a source two-dimensional image, and on the basis of the source two-dimensional image, determining an embedding feature of the source two-dimensional image (S301); acquiring a target two-dimensional image, and on the basis of the target two-dimensional image, determining an embedding feature of the target two-dimensional image (S302); on the basis of the embedding feature of the source two-dimensional image and the embedding feature of the target two-dimensional image, determining a semantic mapping feature, wherein the semantic mapping feature indicates a correspondence from pixels in the source two-dimensional image to pixels in the target two-dimensional image (S303); on the basis of the semantic mapping feature, iteratively updating a deformation model (S304); and on the basis of the updated deformation model, deforming and rendering a source mesh model, so as to obtain a target mesh model fused with geometric information of the source mesh model (S305).

Description

Methods and apparatus for processing mesh models

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on January 27, 2025, with application number 202510127454.3, entitled "Method and Apparatus for Processing Mesh Models", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of artificial intelligence services, and more specifically to a method for processing mesh models, a method for training neural networks, an apparatus for processing mesh models, an electronic device, and a computer-readable storage medium. Background Technology

[0004] Shape correspondence is an important research topic in the field of computer vision and graphics. Its main purpose is to build a point-to-point mapping between the source mesh model and the target mesh model so as to preserve the geometric features of the source mesh model as completely as possible when generating the target mesh model.

[0005] Shape mapping tasks can serve as subtasks within various tasks within computer vision and graphics. For example, in re-topology tasks, shape mapping can be used to determine the correspondence between mesh models with different topologies, transforming complex source topologies into target topologies more suitable for specific application needs, thereby improving the naturalness of 3D modeling. In attribute transfer tasks, once the correspondence between the source and target mesh models is determined, attributes such as color and texture from the source mesh model can be transferred to the target mesh model, achieving efficient attribute assignment. In shape interpolation tasks, interpolation calculations based on the correspondence between multiple mesh models can generate a series of transitional mesh models to support animation production and achieve special deformation effects.

[0006] However, shape mapping tasks face numerous challenges. For example, non-equidistant deformation often occurs during the mapping process from the source mesh model to the target mesh model, meaning that the distance ratio between corresponding points changes during the deformation process, making it difficult to accurately determine the correspondence. Furthermore, different mesh models vary in resolution, noise level, and local detail, all of which reduce the stability of shape mapping tasks and make them difficult to adapt to complex environments. Summary of the Invention

[0007] This disclosure provides a method for processing mesh models, a method for training neural networks, an apparatus for processing mesh models, an electronic device, and a computer-readable storage medium.

[0008] This disclosure provides a method for processing a mesh model, the method comprising: acquiring a source two-dimensional image and determining embedding features of the source two-dimensional image based on the source two-dimensional image; acquiring a target two-dimensional image and determining embedding features of the target two-dimensional image based on the target two-dimensional image; determining semantic mapping features based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image, the semantic mapping features indicating the correspondence between pixels in the source two-dimensional image and pixels in the target two-dimensional image; iteratively updating a deformation model based on the semantic mapping features; and deforming and rendering the source mesh model based on the updated deformation model to obtain a target mesh model that incorporates the geometric information of the source mesh model.

[0009] This disclosure provides a method for training a neural network. The neural network model includes a mapping module and a deformation module. The mapping module is equipped with a first feature extraction network, a second feature extraction network, a first adaptation network, and a second adaptation network. The method includes: determining preliminary features of a source two-dimensional image sample using the first feature extraction network based on the source two-dimensional image sample; determining embedding features of the source two-dimensional image sample using the first adaptation network based on the preliminary features of the source two-dimensional image sample; determining preliminary features of a target two-dimensional image sample using the second feature extraction network based on the target two-dimensional image sample; determining embedding features of the target two-dimensional image sample using the second adaptation network based on the preliminary features of the target two-dimensional image sample; and training the first adaptation network and the second adaptation network based on the embedding features of the source two-dimensional image sample and the embedding features of the target two-dimensional image sample.

[0010] This disclosure provides an apparatus for processing a mesh model, the apparatus comprising: a mapping module configured to: acquire a source two-dimensional image and determine embedding features of the source two-dimensional image based on the source two-dimensional image; acquire a target two-dimensional image and determine embedding features of the target two-dimensional image based on the target two-dimensional image; and determine semantic mapping features based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image, the semantic mapping features indicating the correspondence between pixels in the source two-dimensional image and pixels in the target two-dimensional image; and a deformation module configured to: iteratively update a deformation model based on the semantic mapping features, and deform and render a source mesh model based on the updated deformation model to obtain a target mesh model that incorporates the geometric information of the source mesh model.

[0011] This disclosure provides an electronic device, including: one or more processors; and one or more memories, wherein the memories store a computer-executable program, and when the processor executes the computer-executable program, the above-described method is performed.

[0012] This disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the above-described method.

[0013] According to another aspect of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable medium and executes the computer instructions, causing the electronic device to perform the methods provided in the foregoing aspects or various alternative implementations of the foregoing aspects.

[0014] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.

[0016] Figure 1 is an example schematic diagram illustrating a scenario according to an embodiment of the present disclosure.

[0017] Figure 2 shows a schematic diagram of an example shape corresponding to a task according to an embodiment of the present disclosure.

[0018] Figure 3 shows a flowchart of a method for processing a mesh model according to an embodiment of the present disclosure.

[0019] Figure 4 shows a schematic diagram of a neural network model for performing a shape correspondence task according to an embodiment of the present disclosure.

[0020] Figure 5 shows a schematic diagram of a training method for an improved neural network model according to an embodiment of the present disclosure.

[0021] Figure 6 shows a schematic diagram of an electronic device according to an embodiment of the present disclosure.

[0022] Figure 7 illustrates a schematic diagram of the architecture of an exemplary computing device according to an embodiment of the present disclosure.

[0023] Figure 8 shows a schematic diagram of a storage medium according to an embodiment of the present disclosure. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0025] In this specification and accompanying drawings, operations and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these operations and elements are omitted. Furthermore, in the description of this disclosure, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0026] To facilitate the description of this disclosure, the following concepts related to this disclosure are introduced.

[0027] Optionally, the models used in embodiments of this disclosure as described below can all be artificial intelligence models, especially artificial intelligence-based neural network models. Typically, artificial intelligence-based neural network models are implemented as acyclic graphs, where neurons are arranged in different layers. Generally, a neural network model includes an input layer and an output layer, separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating the output in the output layer. Nodes are fully connected to nodes in adjacent layers via edges, and there are no edges between nodes within each layer. Data received at nodes in the input layer of the neural network is propagated to nodes in the output layer via any of the hidden layers, activation layers, pooling layers, convolutional layers, etc. The input and output of the neural network model can take various forms, and this disclosure does not limit this.

[0028] The solutions provided in this disclosure involve technologies such as artificial intelligence and / or machine learning, which are specifically illustrated through the following embodiments.

[0029] First, with reference to FIG1, an application scenario of the method for processing a mesh model according to an embodiment of the present disclosure and a corresponding apparatus will be described. FIG1 shows a schematic diagram of an application scenario 100 according to an embodiment of the present disclosure, wherein a server 110 and a plurality of terminals 120 are schematically shown.

[0030] The neural network model of this disclosure can be integrated into various electronic devices, such as any electronic device among the server 110 and multiple terminals 120 in Figure 1. For example, the neural network model can be integrated into terminal 120. Terminal 120 can be a mobile phone, tablet computer, laptop computer, desktop computer, personal computer (PC), smart speaker, or smartwatch, but is not limited to these. Alternatively, the neural network model can also be integrated into server 110. Server 110 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be directly or indirectly connected via wired or wireless communication, and this disclosure does not impose any limitations.

[0031] It is understood that the apparatus for inference using the neural network model of the embodiments of this disclosure can be a terminal, a server, or a system composed of a terminal and a server. The method for processing the grid model of the embodiments of this disclosure can be executed on a terminal, on a server, or jointly by a terminal and a server.

[0032] The artificial intelligence models provided in this disclosure can also relate to artificial intelligence cloud services in the field of cloud technology. It is worth noting that both the server 110 and terminal 120 according to the embodiments of this disclosure adhere to data protection principles, respect users' data rights, and ensure user data security and privacy. The server 110 and terminal 120 according to the embodiments of this disclosure will clearly inform users of the purpose, method, and scope of collecting, using, storing, transmitting, and deleting user data, and obtain user consent. The server 110 and terminal 120 according to the embodiments of this disclosure will take reasonable technical and management measures to prevent user data from being leaked, tampered with, damaged, or lost. The providers of the server 110 and terminal 120 according to the embodiments of this disclosure will regularly review and update user data, and promptly delete expired or useless data. Furthermore, cloud service providers using the embodiments of this disclosure respect users' rights to data access, correction, deletion, withdrawal of consent, complaints, and claims, and provide convenient channels and procedures to enable users to effectively exercise these rights.

[0033] Furthermore, the process of data analysis using artificial intelligence technology in terminal 120 or server 110 is conducted based on the principles of legality, rationality, and transparency. The data collected and processed by the artificial intelligence model according to embodiments of this disclosure is relevant, necessary, and appropriate for the predictive purpose, and does not contain any personally identifiable or sensitive information. The neural network model according to embodiments of this disclosure employs appropriate techniques and organizational measures to protect the security and integrity of data, preventing unauthorized access, use, or disclosure.

[0034] The artificial intelligence-based neural network model according to embodiments of this disclosure will comply with relevant data protection regulations and ethical principles. This neural network model is trained on a large amount of anonymized and de-identified data, and does not infringe on the privacy rights of any individual or group. The artificial intelligence model has also undergone rigorous testing and evaluation to ensure that its output results are accurate and reliable, and will not cause any misleading or discriminatory behavior. The artificial intelligence model is designed solely to improve service quality and customer satisfaction and will not be used for any illegal or unethical purposes. Furthermore, the neural network model will be regularly reviewed and updated to adapt to changes in the data environment and legal regulations.

[0035] Shape correspondence is an important research topic in computer vision and graphics. Its main purpose is to construct a point-to-point mapping between a source mesh model and a target mesh model, so as to preserve the geometric features of the source mesh model as completely as possible when generating the target mesh model. Currently, there are two main traditional methods for implementing shape correspondence: functional map methods and registration-for-correspondence methods.

[0036] Function mapping has been widely used in recent years, transforming complex point mapping problems into function mapping problems. In "controlled" scenarios, such as simulations of industrial product design, where the topological differences between mesh models are small, function mapping can quickly and accurately find shape correspondences. However, because function mapping relies on strictly aligned low-rank bases, it performs poorly when dealing with non-equidistant correspondences.

[0037] Registration and mapping methods iteratively deform the source mesh model, gradually aligning it precisely with the target mesh model and generating a dense correspondence. However, these methods often lead to unstable deformation, resulting in distortion artifacts and making it difficult to find an ideal balance between smoothness and deformation accuracy. Furthermore, registration and mapping methods heavily rely on manually added initial alignment or sparse 3D correspondence annotations. When the mesh model undergoes significant non-equidistant deformation, manually adding such annotations is often difficult, leading to a significant degrade in the performance of registration and mapping methods.

[0038] Therefore, both traditional methods for achieving shape correspondence tasks face the problems of unstable deformation and high requirements for manual annotation, which need to be further improved.

[0039] This disclosure uses a registration and correspondence method as the basic paradigm. Given a source mesh model and a target mesh model, this disclosure uses a two-dimensional image-based mapping method to register the source mesh model to the target mesh model without manual annotation, thereby achieving accurate deformation from the source mesh model to the target mesh model.

[0040] In one aspect of the embodiments of this disclosure, the two-dimensional image-based mapping model according to this disclosure can generate stable semantic mapping features. Even in scenarios with significant changes in the topology, shape, and pose of the mesh model, these semantic mapping features exhibit strong robustness, making them more suitable for applications involving differentiable rendering and iterative optimization.

[0041] Differentiable rendering is a special rendering technique that supports backpropagation to calculate the loss gradient during the rendering process. Tools like NVDiff support differentiable rendering, efficiently rendering differentiable semantic flow maps and allowing the calculation of loss gradients during backpropagation, thus supporting model training and optimization. Iterative optimization is an optimization strategy used when dealing with mesh models. In this process, the parameters of the deformable model are continuously updated through multiple iterations, allowing the deformable model to gradually deform the source mesh model to align its shape with the target mesh model. The goal of iteration is usually to make the semantic flow loss converge to a minimum, ultimately achieving accurate deformation from the source mesh model to the target mesh model.

[0042] In one aspect of the embodiments of this disclosure, during the training process of the two-dimensional image-based mapping model according to this disclosure, the two-dimensional image-based mapping model can learn how to construct accurate pixel-to-pixel correspondences in two dimensions through negative loss, avoiding erroneous correspondences caused by the morphological symmetry between the source mesh model and the target network in traditional schemes.

[0043] This disclosure proposes an improved neural network model. The improved neural network model includes at least a mapping model and a deformation model. The mapping model generates semantic mapping features based on a two-dimensional image. The deformation model iteratively updates the semantic mapping features to obtain a target mesh model. However, this disclosure is not limited thereto.

[0044] Therefore, this disclosure utilizes semantic mapping features that indicate the correspondence between pixels in a source 2D image and pixels in a target 2D image to guide the deformation process performed by the deformation model, thereby achieving deformation from the source mesh model to the target mesh model. Through iterative optimization, the deformation model progressively deforms the source mesh model to align with the shape of the target mesh model. Thus, embodiments of this disclosure, through the collaborative work of the mapping model and the deformation model, can achieve accurate deformation from the source mesh model to the target mesh model even under non-equidistant conditions.

[0045] For an improved neural network model, embodiments of this disclosure provide a method for processing a mesh model to perform inference based on such an improved neural network model. The method includes: acquiring a source two-dimensional image and determining embedding features of the source two-dimensional image based on the source two-dimensional image; acquiring a target two-dimensional image and determining embedding features of the target two-dimensional image based on the target two-dimensional image; determining semantic mapping features based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image, the semantic mapping features indicating the correspondence between pixels in the source two-dimensional image and pixels in the target two-dimensional image; iteratively updating a deformation model based on the semantic mapping features; and deforming and rendering the source mesh model based on the updated deformation model to obtain a target mesh model that incorporates the geometric information of the source mesh model.

[0046] In the process of reasoning using an improved neural network, the embodiments of this disclosure register the source mesh model to the target mesh model without manual annotation through a mapping model based on a two-dimensional image, thereby achieving accurate deformation from the source mesh model to the target mesh model and significantly improving the performance of shape correspondence tasks.

[0047] Similarly, for the improved neural network model, this disclosure also provides a method for training the neural network to train the improved neural network model. The neural network model includes a mapping module and a deformation module. The mapping module is equipped with a first feature extraction network, a second feature extraction network, a first adaptation network, and a second adaptation network. The method includes: determining preliminary features of the source two-dimensional image samples using the first feature extraction network based on the source two-dimensional image samples; determining embedding features of the source two-dimensional image samples using the first adaptation network based on the preliminary features of the source two-dimensional image samples; determining preliminary features of the target two-dimensional image samples using the second feature extraction network based on the target two-dimensional image samples; determining embedding features of the target two-dimensional image samples using the second adaptation network based on the preliminary features of the target two-dimensional image samples; and training the first adaptation network and the second adaptation network based on the embedding features of the source two-dimensional image samples and the embedding features of the target two-dimensional image samples.

[0048] During the training phase, this embodiment employs symmetry loss to learn how to construct accurate pixel-to-pixel correspondences in two dimensions, avoiding erroneous correspondences caused by the morphological symmetry between the source mesh model and the target network, as seen in traditional methods. Symmetry loss is a loss mechanism used in the training of mapping models based on two-dimensional images. Its function is to enable the model to learn and construct accurate pixel-to-pixel correspondences in two-dimensional images, avoiding erroneous correspondences caused by the morphological symmetry between the source mesh model and the target mesh model.

[0049] The method for processing a mesh model, the method for training a neural network, and the apparatus according to embodiments of the present disclosure are described below with reference to Figures 2 to 8.

[0050] Figure 2 shows a schematic diagram of an example shape corresponding to a task according to an embodiment of the present disclosure.

[0051] Shape mapping tasks can be widely applied to various scenarios such as topology re-conversion, attribute transfer, and shape interpolation.

[0052] Retopology is an operation in computer vision and graphics based on shape correspondence tasks. It aims to determine the correspondence between mesh models with different topological structures, transforming a complex source topology into a target topology that better suits specific application needs, thereby improving the naturalness of 3D modeling. For example, when converting a source mesh model of a cartoon bear toy into a target mesh model of a cartoon fox toy, retopology can accurately map the surface pattern of the source mesh model onto the target mesh model.

[0053] For example, in retopology scenarios, shape mapping tasks can be used to determine the correspondence between mesh models with different topological structures. This transforms complex source topologies into target topologies more suitable for specific application needs, thereby improving the naturalness of 3D modeling. As shown in Figure 2, in a retopology scenario, the input source mesh model is a cartoon bear doll with a regular grid pattern on its surface. The target mesh model is a cartoon fox doll with different body shapes and ear shapes. Through the shape mapping task, the grid pattern of the source mesh model is accurately mapped to the target mesh model, ensuring a regular and natural grid arrangement on the fox doll's surface, adapting to its unique shape characteristics.

[0054] Attribute transfer is an operation in shape mapping applications. Once the correspondence between the source and target mesh models is clear, attributes such as color and texture from the source mesh model are transferred to the target mesh model, achieving the goal of quickly assigning attributes to the target mesh model. For example, the plaid pattern of a source clothing mesh model with a regular plaid pattern can be transferred to a target clothing mesh model with a different cut.

[0055] For example, in attribute transfer scenarios, once the correspondence between the source and target mesh models is determined, attributes such as color and texture from the source mesh model can be transferred to the target mesh model, thus achieving rapid attribute assignment. The input source mesh model is a 3D clothing mesh model with a regular checkered pattern, while the target mesh model is a 3D clothing mesh model with a different cut. Through shape mapping, the checkered pattern on the source mesh model can be correctly mapped to the target mesh model, ensuring its regular arrangement and natural deformation on the target clothing.

[0056] Shape interpolation is a processing method based on shape correspondence. It performs interpolation calculations based on the correspondence between multiple mesh models, generating a series of transition mesh models to support animation production and achieve special deformation effects. Typically, linear interpolation is used, linearly combining the coordinates of corresponding points in the source and target mesh models according to a certain interpolation ratio. Changing the interpolation ratio generates different transition mesh models. For example, it's used in processing models of facial expression changes in animated characters to achieve natural transitions in facial details.

[0057] In shape interpolation scenarios, interpolation calculations are performed based on the correspondence between multiple mesh models to generate a series of transition mesh models, supporting animation production and deformation effects. Specifically, a linear interpolation method is used. Based on the correspondence between multiple mesh models, for each corresponding point, the coordinates of corresponding points in the source and target mesh models are linearly combined according to a certain interpolation ratio. The calculation formula is P. interp =(1-α)P src +αPtgt , where P interp P represents the coordinates of the corresponding point in the transition mesh model. src P represents the coordinates of the corresponding point in the source mesh model. tgt Here, α represents the coordinates of the points corresponding to the target mesh model, and α is the interpolation ratio (0 ≤ α ≤ 1). By changing the value of α, a series of transition mesh models can be generated. For example, the source mesh model is a model of a smiling animated character, whose facial texture includes details such as beard and freckles. The target mesh model is a model of the same character crying. Through shape mapping, facial textures are accurately transferred, and freckles and beards can naturally adapt to different expression changes, providing stable facial detail transitions for animation production.

[0058] Figure 3 shows a flowchart of a method 30 for processing a mesh model according to an embodiment of the present disclosure.

[0059] Method 30 can be executed at a terminal device or a server (such as terminal 120 or server 110 as shown in FIG1). Method 30 includes the following operations S301 to S305. Of course, method 30 may also include more or fewer operations, and this disclosure is not limited thereto.

[0060] In operation S301, a source two-dimensional image is acquired, and the embedding features of the source two-dimensional image are determined based on the source two-dimensional image. In operation S302, a target two-dimensional image is acquired, and the embedding features of the target two-dimensional image are determined based on the target two-dimensional image.

[0061] Optionally, both the source 2D image and the target 2D image can be planar images composed of pixels arranged in rows and columns, with each pixel containing color or grayscale information. For example, the source 2D image can be a 2D image obtained by rendering the source mesh model based on its geometric information under a preset camera viewpoint, such as an RGB image rendered from different perspectives like the front view, side view, or top view of the source mesh model. The target 2D image is a 2D image obtained by rendering the target mesh model based on its geometric information under the same preset camera viewpoint as the source 2D image. The geometric information of the source mesh model differs from that of the target mesh model. For example, if the source 2D image is a 45-degree angle view of the source mesh model, then the corresponding target 2D image should also be a 45-degree angle view of the source mesh model. For example, given a series of fixed camera viewpoints C... i The data is set to i = 1, 2, ..., K, and a 2D image from the fixed camera viewpoint is rendered as the source 2D image. and target two-dimensional image Of course, this disclosure is not limited to this.

[0062] Optionally, for example, a source mesh model refers to a mesh model used to represent a source 3D object in a data-driven form. This polygonal mesh model includes the 3D coordinate information of multiple vertices and the triangular facet information formed by these vertices. For example, a source mesh model of a cube may contain the spatial coordinates of 8 vertices and the triangular facet information formed by these vertices. A target mesh model refers to a mesh model used to represent a target 3D object in a data-driven form. This mesh model also includes the vertex coordinate information and triangular facet information of the target 3D object. For example, a target mesh model of a sphere may contain the spatial coordinates of several vertices and the triangular facet information formed by these vertices. Of course, this disclosure is not limited thereto.

[0063] Optionally, the goal of the shape mapping task is to generate a target mesh model that retains the geometric information of the source mesh model as completely as possible. For example, accurately mapping the geometric information related to color, edges, topology, and texture in the source mesh model of a cube onto the target mesh model of a sphere.

[0064] Optionally, the geometric information of the source mesh model and the target mesh model may not be directly adapted, and there may be a potential non-equidistant correspondence. Specifically, the geometric information of a mesh model refers to the geometric shape information, texture information, and structural information used to characterize the 3D object corresponding to that mesh model. Geometric shape information includes: the 3D coordinate information of multiple vertices of the mesh model and the triangular facet information formed by connecting these vertices. A non-equidistant correspondence refers to the situation where the source mesh model and the target mesh model cannot achieve a complete match through simple rigid body transformations (such as translation, rotation, or scaling). For example, the source mesh model may represent a smiling expression on a face, while the target mesh model represents a neutral expression on the same face, and there are complex local differences in their geometric deformations. Of course, this disclosure is not limited to this.

[0065] Optionally, the embedding features of the source 2D image refer to the high-dimensional feature vectors that can represent the visual content of the image, extracted from the source 2D image through a feature extraction network and an adaptation network. These high-dimensional feature vectors reside in the embedding space and are therefore also called the embedding features of the source 2D image. Each feature point in the embedding features of the source 2D image can indicate information such as local texture features and global semantic features of the corresponding pixel in the source 2D image, or it can indicate low-level features such as edges, corners, textures, and shapes of the corresponding pixel in the source 2D image, or high-level features such as object categories and scene layouts of the corresponding pixel in the source 2D image. The embedding features of the target 2D image refer to the embedding features that can represent the visual content of the image, extracted from the target 2D image through the same or similar feature extraction network and adaptation network. These embedding features have the same dimension and similar semantic meaning as the features of the source 2D image. Of course, this disclosure is not limited to this.

[0066] For example, in operation S301, preliminary features of the source two-dimensional image can be determined using a first feature extraction network based on the source two-dimensional image; and embedding features of the source two-dimensional image can be determined using a first adaptation network based on the preliminary features of the source two-dimensional image. In operation S302, preliminary features of the target two-dimensional image can be determined using a second feature extraction network based on the target two-dimensional image; and embedding features of the target two-dimensional image can be determined using a second adaptation network based on the preliminary features of the target two-dimensional image. The embedding features of the source two-dimensional image and the embedding image of the target two-dimensional image reside in the same embedding space. Of course, this disclosure is not limited thereto.

[0067] Specifically, in operation S301, when the first feature extraction network is a pre-trained ResNet-50, the source 2D image is input into this network. ResNet-50 extracts features from the source 2D image through its multiple convolutional layers. Different convolutional layers extract features at different levels; for example, shallow convolutional layers extract low-level features such as edges and textures, while deep convolutional layers extract high-level features such as semantics. These extracted features are then concatenated to obtain the preliminary features of the source 2D image. The first adaptation network is a neural network containing three fully connected layers, each followed by a ReLU activation function. The preliminary features of the source 2D image are input into the first adaptation network. After linear transformation by the three fully connected layers and non-linear transformation by ReLU, the features are mapped to a fixed-dimensional embedding space, resulting in the embedded features of the source 2D image. Similarly, in operation S302, the target 2D image is input into the second feature extraction network (also a pre-trained ResNet-50) to obtain preliminary features, which are then input into the second adaptation network (with the same structure) to obtain the embedded features of the target 2D image.

[0068] For example, in some aspects of this disclosure, the first feature extraction network and the second feature extraction network may be the same feature extraction network, and this feature extraction network may be pre-trained. That is, the pre-trained first and second feature extraction networks can be used to extract preliminary features of the source 2D image and the target 2D image, and then the preliminary features are adapted by an adaptation network to map the preliminary features into the embedding space to obtain the embedding features X of the source 2D image. src Embedding features X of the target 2D image tgt .

[0069] For example, assuming the first and second feature extraction networks are stable diffusion models, the source and target 2D images can be input into the pre-trained stable diffusion model, respectively. Multi-scale features are extracted from multiple intermediate layers of its UNet encoder as preliminary features of the source 2D image. These preliminary features can contain representations of the image at different semantic levels. As another example, assuming the first and second feature extraction networks are DINO visual transformer models, the source and target 2D images can also be input into the pre-trained DINO visual transformer model, respectively. High-level semantic features are extracted from the output of its last transformation block. For example, for a source 2D image containing a cat, the extracted preliminary features can encode multiple levels of visual information such as the cat's fur texture, ear shape, and body posture. Of course, this disclosure is not limited to these limitations.

[0070] Optionally, after obtaining the features of the source 2D image and the preliminary features of the target 2D image, lightweight adaptation networks with fewer parameters (i.e., the first adaptation network and the second adaptation network) can be used to map these features into a common embedding space. The first and second adaptation networks can be neural networks composed of several fully connected layers and nonlinear activation functions, for example, they can contain two fully connected layers with ReLU activation functions. The first and second adaptation networks are trainable, and their training methods will be described in detail later; this disclosure will not elaborate further here. Through the first and second adaptation networks, the features of the source 2D image and the target 2D image are transformed into the same embedding space. Of course, this disclosure is not limited to this.

[0071] In operation S303, semantic mapping features are determined based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image. The semantic mapping features indicate the correspondence between pixels in the source two-dimensional image and pixels in the target two-dimensional image.

[0072] Optionally, in operation S303, a nearest neighbor search can be performed based on the embedding features of the source 2D image and the embedding features of the target 2D image to determine the semantic mapping features. Specifically, for each feature vector in the embedding features of the source 2D image, a KD-tree (K-Dimensional Tree) data structure is used to accelerate the search. First, a KD-tree of the embedding features of the target 2D image is constructed. Then, the target feature vector with the closest Euclidean distance to the source feature vector is searched in the KD-tree. The pixel position corresponding to this target feature vector is used as the mapping position of the pixel corresponding to the source feature vector, thereby determining the semantic mapping features. The semantic mapping features can be represented as a two-dimensional displacement relationship between pixels in the source image and pixels in the target image in the form of a semantic flow map. This semantic flow map can be viewed as a pixel-level correspondence matrix, where each element describes the corresponding position of a pixel in the source 2D image in the target 2D image. For each camera viewpoint C... i This can generate a corresponding semantic stream mapping S. i To reflect the source two-dimensional image and target two-dimensional image The pixel correspondence between them. Of course, this disclosure is not limited to this. Nearest neighbor search is a method for determining the correspondence between feature points in a source 2D image and a target 2D image. It calculates the distance between each feature point in the embedded features of the source 2D image and each feature point in the embedded features of the target 2D image. For each feature point in the source 2D image, it finds the feature point with the smallest distance to the target 2D image as its corresponding point, thus determining the pixel-to-pixel correspondence. KD-tree data structures can usually be used to accelerate the search process.

[0073] Optionally, nearest neighbor search can be achieved by calculating the distance between each feature point in the embedded features of the source 2D image and each feature point in the embedded features of the target 2D image. Specifically, for each feature point in the source 2D image, its distance to all feature points in the target 2D image is calculated, and the target feature point with the smallest distance is considered to be the corresponding point of the source feature point.

[0074] Optionally, the correspondence from feature points can further indicate pixel-to-pixel correspondences. Thus, the semantic mapping features can indicate the correspondence from pixels in the source 2D image to pixels in the target 2D image. For example, each feature point corresponds to a pixel block in the 2D image. Through mapping, the correspondence between feature points is extended to all pixels they cover. In this way, a global correspondence from each pixel in the source 2D image to the corresponding pixel in the target 2D image can be generated, thereby achieving finer-grained matching. By operating S301 to S303, non-equidistant pixel-to-pixel correspondences can be determined without manual matching. Of course, this disclosure is not limited thereto.

[0075] In operation S304, the deformable model is iteratively updated based on the semantic mapping features.

[0076] Specifically, the deformable model can be iteratively updated based at least in part on the semantic mapping features.

[0077] Optionally, with the registration-correspondence method as the basic paradigm, the deformation model can be used to describe the geometric transformation relationship between the source mesh model and the target mesh model. Operation S304, by iteratively updating the deformation model, can gradually obtain a deformation model that can transform the geometric information of the source mesh model into the target mesh model. In one aspect of this disclosure, the deformation model can be represented and learned using neural networks to capture local and global features in non-rigid deformation, for example, by modeling the local deformation gradient during the deformation process using Neural Jacobian Fields (NJF), or by using neural implicit functions to represent the continuous deformation field. Of course, this disclosure is not limited thereto. Neural Jacobian Fields are a technique for representing and learning deformation models. It can capture local and global features in non-rigid deformation by defining the Jacobian matrix and related gradient operators for each face, combined with mathematical methods such as the Poisson equation, to deform the source mesh model, producing smooth deformation results and minimizing distortion while preserving the details of the mesh model.

[0078] When using NJF as the deformation model, one or more embodiments of this disclosure can produce smooth deformation results while preserving details and minimizing distortion. Thereafter, although NJF is used as an example deformation model for description, operation S304 can also be implemented using other deformation models. Of course, this disclosure is not limited thereto.

[0079] In operation S305, based on the updated deformation model, the source mesh model is deformed and rendered to obtain a target mesh model that incorporates the geometric information of the source mesh model.

[0080] For example, a target mesh model that incorporates the geometric information of the source mesh model refers to a mesh model processed by a deformation model. This mesh model retains the topological structure of the target mesh model while inheriting some geometric features of the source mesh model. Specifically, this fusion can be reflected in geometric attributes such as vertex positions, normals, and curvature. For instance, in a task involving a human body model, the deformed model can maintain the topological structure of the target human body while incorporating geometric details such as body shape and pose information from the source model. Of course, this disclosure is not limited to this.

[0081] In summary, referring to one aspect of the present disclosure described in Figure 3, given a source mesh model and a target mesh model, stable semantic mapping features can be generated based on a two-dimensional image, and the source mesh model can be registered to the target mesh model without manual annotation, thereby achieving accurate deformation from the source mesh model to the target mesh model.

[0082] Specifically, in this aspect, stable semantic mapping features are generated based on operations S301 to S303. Even in scenarios with significant changes in the topology, shape, and orientation of the mesh model, these semantic mapping features exhibit strong robustness to adapt to the differentiable rendering and iterative optimization process in operation S304. In operation S304, the semantic mapping features guide the deformation process, and through iterative optimization, the deformation model progressively deforms the source mesh model to align with the shape of the target mesh model. Therefore, even under non-equidistant conditions, this disclosure can achieve accurate deformation from the source mesh model to the target mesh model.

[0083] Next, some optional details of embodiments of the present disclosure will be described with reference to FIG4. FIG4 shows a schematic diagram of a neural network model 40 for performing a shape correspondence task according to an embodiment of the present disclosure, which can be used to implement all or part of the operations according to the embodiments of the present disclosure, such as operations S301 to S304. Of course, the present disclosure is not limited thereto.

[0084] Specifically, the neural network model 40 for performing the shape correspondence task includes, but is not limited to, a mapping module 401 and a deformation module 402. The mapping module 401 carries a mapping model for performing operations S301 to S303. The deformation module 402 carries a deformation model for performing operation S304. Of course, the neural network model 40 may also include more or fewer models or modules, and this disclosure does not impose any limitations on this.

[0085] Optionally, the input to the neural network model 40 performing the shape correspondence task includes a source mesh model (shown as a 3D cartoon image of a red fox wearing a brown shirt in Figure 4) and a target mesh model (shown as a 3D cartoon image of a white rabbit wearing blue pants in Figure 4). The output objective of the neural network model 40 is to fuse the geometric information of the source mesh model into the target mesh model as much as possible, to generate a target mesh model that incorporates the geometric information of the source mesh model, such as the rendered target mesh model in Figure 4. It is located in the lower right corner of Figure 4 and features a colorful rabbit image. The colorful rabbit image in Figure 4 is for illustrative purposes only and does not represent the actual rendering result.

[0086] Optionally, in operation S301, the mapping module 401 can acquire a single source 2D image by rendering the source mesh model based on the geometric information of the source mesh model according to the camera viewpoint, to obtain a source 2D image corresponding to the camera viewpoint. Simultaneously, in operation S302, the mapping module 401 can also acquire a single target 2D image by rendering the target mesh model based on the geometric information of the target mesh model according to the camera viewpoint, to obtain a target 2D image corresponding to the camera viewpoint. Although this example only describes the acquisition method of a single 2D image, this example can also be extended to scenarios involving the acquisition of multiple 2D images. This disclosure is not limited thereto.

[0087] For example, we can assume there are K camera viewpoints, each with an index of i. In operations S301 and S302, the mapping module 401 obtains K source 2D images respectively. and K target two-dimensional images Source 2D images and target 2D images with the same index correspond to the same camera viewpoint. Then, the mapping module 401 uses these K source 2D images... and K target two-dimensional images To determine K latent features of the source 2D image and K latent features of the target 2D image.

[0088] Specifically, as described above, in operation S301, the mapping module 401 can be based on the source two-dimensional image. The source 2D image is determined using a feature extraction network 4011. Preliminary features, then based on the source two-dimensional image Preliminary features are obtained by using an adaptation network 4012 to determine the source 2D image. Embedding features In operation S302, the mapping module 401 can be based on the target two-dimensional image. The feature extraction network 4013 is used to determine the preliminary features of the target two-dimensional image, and the adaptation network is used to determine the target two-dimensional image. Embedding features

[0089] Next, in operation S303, the mapping module 401 can base its work on the embedding features of the source two-dimensional image. and the embedding features of the target two-dimensional image Perform a nearest neighbor search to determine the semantic mapping feature S i The semantic mapping feature represents the two-dimensional displacement relationship between source image pixels and target image pixels in the form of a semantic flow mapping graph.

[0090] In operation S304, the deformation module 402, in each iteration, can use the deformation model (NJF) to estimate the positions of vertices in the source mesh model after deformation to the target mesh model; based on the positions Φ of the vertices in the source mesh model after deformation to the target mesh model... * Determine the displacement corresponding to the vertex; based on the displacement F corresponding to the vertex i Determine the estimated semantic mapping features and the semantic mapping features based on the estimation. With the semantic mapping feature S i Difference L flow Update the parameters in the deformable model.

[0091] The following is a brief explanation of the NJF as a deformation model. Specifically, the per-face Jacobian matrix J of the NJF can be defined by formula (1). i .

[0092] Among them, v k v j v i These are the positions of the three original vertices of triangle i in the source 3D mesh model, φ k φ j φ i J represents the positions of three deformable vertices in the target 3D mesh model. i ∈R 3×2 It is a surface Jacobian matrix, B i ∈R 3×2 These are two column vectors that form an orthogonal basis of the tangent space of the surface. This formula also defines a gradient operator, as shown in formula (2).

[0093] Where Φ∈R n×3The position of the vertex after one or more iterations is the position of the deformed vertex. Solving for the position of the deformed vertex involves the Poisson equation. For example, the Poisson equation defines the position Φ of the deformed vertex using formula (3). * .

[0094] Among them, |t i | is the region where triangular facet i is located. This Poisson method can be solved using the least squares method according to formula (4).

[0095] Where L is the coche Laplacian operator of the mesh model, A is the mass matrix of the mesh model, and J is the estimated Jacobian matrix J0. i The stack. During registration, the transformation matrix of each face is... For optimization, these matrices are projected onto the tangent space B of each face. i Above, through Obtain the Jacobian matrix J i Finally, formula (4) is used to solve for the final vertex position Φ. * .

[0096] The specific process of solving the Poisson method using the least squares method is as follows:

[0097] Formula (3) This is transformed into a least squares problem in matrix form Ax = b, where A is the coefficient matrix and x is the transformed vertex position Φ to be solved. * b is a constant vector. By analyzing A... T Ax = A T b is used to solve for x, resulting in x = (A) T A) -1 A T b, that is, according to formula (4) Please provide a solution.

[0098] The core idea of ​​formulas (1) to (4) is to define the deformation in a compact, parameterized tangent space, which simplifies the optimization process compared to operating in the environment space. Of course, this disclosure is not limited thereto.

[0099] According to NJF, given the position Φ of the deformed vertex * The vertex can be projected according to the corresponding camera view and the displacement F of that vertex can be calculated. i This process can be described by formula (5). F i =Π(Φ * C i )-Π(V src C i(5)

[0100] Where, Φ * It refers to the position of the vertex of the triangular facet in the source mesh model after NJF deformation, while V src It refers to the position of the vertices of the triangular facets in the source mesh model, C. i Let F be the viewpoint of the i-th camera, and Π be the projection operator used to project the coordinates of the vertex onto a 2D coordinate system. Specifically, F... i This could be the displacement of the deformed vertex relative to the vertices of the triangular facet in the source mesh model. F i It is the spatial change of the vertices of the mesh model during the shape deformation process, which describes how the source vertex moves to the target position in the deformed mesh model.

[0101] Then, given the displacement F i and camera perspective C i Then, the source 3D mesh model M can be processed. src Shading is performed to obtain a differentiable semantic flow map. Shading is the process of assigning color to vertices based on certain properties (such as position, normal, texture, or displacement). For example, NVDiff can be used for shading and rendering. NVDiff is a tool that supports differentiable rendering, capable of efficiently rendering differentiable semantic flow maps. It also allows the calculation of the loss gradient during backpropagation. This process can be described by equation (6).

[0102] That is, formula (6) describes the use of the normalized two-dimensional displacement as the vertex color, combined with the differentiable rendering function R(·) to render the source mesh model M. src From camera perspective C i The captured 2D image is rendered (colored) to obtain a differentiable semantic flow map. The process.

[0103] Due to differentiable semantic flow map It is NJF based on the source 3D mesh model M src Camera perspective C i and displacement F i The estimated value is closer to the semantic mapping feature S determined in operation S303. i This indicates that the more fully the geometric information of the source mesh model is reflected in the deformed target mesh model, the better. Therefore, an optional objective of the iteration in operation S304 is to map the differentiable semantic flow graph. In operation S303, the semantic mapping feature S is determined.i The difference in the loss converges to a minimum, which is subsequently named the semantic flow loss. Specifically, this process can be illustrated by equation (7).

[0104] Therefore, in each iteration, the semantic mapping features obtained in operation S303 can be used to guide the iteration of parameters in the deformation model NJF, thereby enabling the deformation model NJF that finally performs deformation to fuse the geometric information of the source mesh model and the geometric information of the target mesh model, so as to obtain the target mesh model that fuses the geometric information of the source mesh model in operation S305.

[0105] Alternatively, in addition to using semantic mapping features to guide the iteration of parameters in the deformable model NJF, other information (such as geometric alignment loss, anti-shear loss, etc., detailed below) can be combined with semantic mapping features to guide the iteration of parameters in the deformable model NJF.

[0106] Specifically, in one aspect of this disclosure, in operation S304, in each iteration, the deformation model can be used to estimate the position of the vertices in the source mesh model after deformation to the target mesh model; based on the position of the vertices in the source mesh model after deformation to the target mesh model, the displacement corresponding to the vertex is determined; based on the displacement corresponding to the vertex, the mesh model after deformation by the deformation model is determined; and based on the difference between the deformed mesh model and the target mesh model, the parameters in the deformation model are updated.

[0107] Optionally, the difference between the deformed mesh model and the target mesh model is shown in Figure 4 as the CD loss L. cd First, based on the vertex displacements described above, the deformed mesh model M can be obtained. deformed Then calculate M. deformed Vertex and target mesh model M target Chamfer distance loss between vertices L cd The Chamfer distance loss measures the geometric difference between the two and can effectively align the mesh model shape. Therefore, one possible direction for iteratively updating the deformable model is to further minimize / converge the Chamfer distance loss. Alternatively, the Chamfer distance loss can be replaced by different distance metrics (such as Euclidean distance or Manhattan distance) to address different types of shape-corresponding tasks.

[0108] Specifically, in one aspect of this disclosure, in operation S304, in each iteration, the deformation model can be used to estimate the position of vertices in the source mesh model after deformation to the target mesh model; based on the position of the vertices in the source mesh model after deformation to the target mesh model, the displacement corresponding to the vertex is determined; based on the displacement corresponding to the vertex, the mesh model after deformation by the deformation model is determined; the deformed mesh model is rendered differently to determine the normal map corresponding to the deformed mesh model; the normal map corresponding to the target mesh model is determined; and the parameters in the deformation model are updated based on the difference between the normal map corresponding to the deformed mesh model and the normal map corresponding to the target mesh model.

[0109] Optionally, the difference between the normal map corresponding to the deformed mesh model and the normal map corresponding to the target mesh model is shown in Figure 4 as the normal loss L. normal The normal map represents the normal vector of each point on the surface of the mesh model, that is, the vector perpendicular to the surface at that point. Then, these normals are calculated to the target mesh model M. target The difference between the normals of corresponding points. Optionally, this difference can be measured using point-to-point angle difference or cosine similarity. In one aspect of this disclosure, differentiable rendering techniques can be used to render M. deformed The normal plot, and minimizing the normal loss L. normal Or cause a loss of normal L normal Convergence is achieved to further optimize the mesh model shape alignment. This is achieved through the normal loss L... normal This allows the deformed mesh model surface to be as close as possible to the target mesh model in the normal direction, thereby improving the matching degree of surface details.

[0110] CD loss L cd and normal loss L normal Both belong to the category of geometry alignment loss, and both aim to correct the deformation of the source mesh model into a new mesh model M. deformed Align the shape with the target mesh model. Of course, other geometric alignment losses can also be used, and this disclosure is not limited thereto.

[0111] Specifically, in one aspect of this disclosure, in operation S304, in each iteration, the Jacobian matrix of each face of the vertices in the source mesh model after deformation to the target mesh model can be estimated using the deformation model; the displacement corresponding to the vertex is determined based on the position of the vertex in the source mesh model after deformation to the target mesh model; and the mesh model M after deformation by the deformation model is determined based on the displacement corresponding to the vertex. deformed Determine the Jacobian matrix of each face in the deformed mesh model. And based on the difference between the Jacobian matrix and the identity matrix I3 of each face in the deformed mesh model, the parameters in the deformed model are updated.

[0112] Specifically, the difference between the Jacobian matrix and the identity matrix I3 of each face in the deformed mesh model is called the identity-preserving loss. This loss is used to constrain the transformation matrix of each face in the mesh model to further increase the stability of the shape during deformation. identity The expression is:

[0113] in, It is a mesh model M deformed The Jacobian matrix of the i-th face (or triangular facet). I3 is a 3×3 identity matrix representing the ideal case without deformation. F represents the number of faces in the mesh model. The deformation process applies a transformation to every face in the mesh model, therefore the identity preservation loss L... identity The transformation matrix of each face needs to be summed. By regularizing the transformation matrix of each face, the identity preservation term ensures that the mesh model does not exhibit excessive distortion during deformation, especially during large-scale deformations, effectively avoiding unnatural shape transformations. The optimization goal of this term is to make the transformation matrix of each face after deformation as close as possible to the identity matrix, thereby maintaining the smoothness and naturalness of the deformation.

[0114] However, using only the identity preservation loss is insufficient to solve all over-deformation problems. Applying a large weighting factor limits a wide range of pose deformations, while a small weighting factor may lead to uneven deformation. Therefore, in one aspect of this disclosure, in operation S304, in each iteration, the deformation model can be used to estimate the Jacobian matrix of each face of the source mesh model after the vertices are deformed into the target mesh model; the displacement corresponding to the vertex is determined based on the position of the vertex after the source mesh model is deformed into the target mesh model; the mesh model after deformation by the deformation model is determined based on the displacement corresponding to the vertex; the Jacobian matrix of each face in the deformed mesh model and the rotation component of the Jacobian matrix after polar decomposition are determined; and the parameters in the deformation model are updated based on the difference between the Jacobian matrix of each face in the deformed mesh model and the corresponding rotation component.

[0115] Specifically, the difference between the Jacobian matrix and the corresponding rotation component of each face in the deformed mesh model can be referred to as shear-resistant loss. Shear-resistant loss is an important term used in the regularization deformation process, aiming to suppress excessive shear deformation during deformation and ensure the smoothness and naturalness of the mesh model deformation. Shear deformation refers to unbalanced stretching or compression of the mesh model during deformation, leading to shape distortion, especially when the object undergoes non-uniform deformation or high-frequency detail processing.

[0116] Shear loss resistance L shear By constraining the rotational components of the deformation matrix, it is ensured that deformation does not occur solely through non-rigid shear deformation. Specifically, the shear loss L... shear The format is as follows:

[0117] in, It is a mesh model M deformed Jacobian matrix of the i-th face (or triangular facet). yes The rotation component is extracted through polar decomposition. Polar decomposition decomposes an arbitrary matrix into two components: a rotation component and a positive definite transformation component. The rotation component describes only pure rotation, without including shearing or scaling deformation. This part reflects local orientation changes on the surface, but does not include stretching or compression of the shape. Therefore, This value can be used to represent the loss of surface F due to shearing; the smaller the better. Through this regularization method, shear deformation during the deformation process is effectively suppressed, thereby ensuring a smoother and more stable mesh model deformation. Of course, this disclosure is not limited to this.

[0118] In one aspect of this disclosure, the aforementioned losses can be synthesized, with the final loss function serving as the update target for operation S304, to ensure that deformation and geometric alignment in the shape correspondence task can be performed smoothly and naturally. This final loss function is formulated as follows: L = λ flow L flow +λ cd L cd +λ normal L normal +λ identity L identity +λ shear L shear (10)

[0119] The various losses have been described above. This loss function balances and optimizes the constraints during the deformation process by minimizing the weighted sum of these losses. The transformation matrix on each surface is then iteratively optimized. After final optimization convergence, a deformed model with optimal parameters can be obtained. Optionally, the weights of each loss term can be adjusted according to task requirements to achieve the best results in different scenarios. Of course, this disclosure is not limited to this.

[0120] In one aspect of this disclosure, the application of the final loss function optimizes the shape deformation process by minimizing the weighted sum of each sub-loss term, particularly in non-equidistant correspondence tasks. Specifically, after optimization convergence, the registration result Φ can be obtained using the aforementioned formula (4). final To extract the 3D correspondence, it is first necessary to examine each deformed vertex. Find the target mesh model M tgt The closest face. Then, Project onto the nearest surface and calculate the centroid coordinates, thus projecting onto the source mesh model M. src and target mesh model M tgt A final mapping relationship is established between the source and target mesh models. This process ensures the accurate mapping and position adjustment of each vertex after deformation in non-equidistant correspondence tasks. Of course, this disclosure is not limited to this. Centroid coordinates are a coordinate representation used when establishing an accurate mapping relationship between the source and target mesh models. To extract the 3D correspondence, the closest face in the target mesh model must be found for each deformed vertex. This vertex is projected onto this closest face, and its centroid coordinates are calculated. This achieves accurate mapping and position adjustment of vertices between the source and target mesh models, ensuring the accurate position of vertices after deformation in non-equidistant correspondence tasks.

[0121] In one aspect of this disclosure, the design of the final loss function fully considers the non-isotropic deformation problem in shape-corresponding tasks. By combining multiple loss terms, various challenges such as semantic alignment, geometric alignment, normal consistency, and shearing suppression can be addressed while ensuring the smoothness and stability of the deformation process. For example, when performing facial expression transfer, the final loss function not only requires accurate matching of the vertex positions of the mesh model but also needs to ensure the natural deformation of the facial expression and the consistency of the normal direction, avoiding unnatural distortion of the facial expression. By optimizing this loss function, a high degree of consistency in shape and surface details can be ensured in complex deformation tasks. Of course, the loss terms can also be customized according to specific application requirements to handle different types of deformations and corresponding tasks.

[0122] Next, some optional details of embodiments of the present disclosure will be described with reference to FIG5. FIG5 shows a schematic diagram of a training method 5 for an improved neural network model according to an embodiment of the present disclosure.

[0123] The improved neural network model training method 50 according to embodiments of the present disclosure includes the following operations S501 to S505. Of course, method 50 may also include more or fewer operations, and the present disclosure is not limited thereto.

[0124] In operation S501, based on the source two-dimensional image sample, the preliminary features of the source two-dimensional image sample are determined using a first feature extraction network.

[0125] In operation S502, based on the preliminary features of the source two-dimensional image sample, the embedding features of the source two-dimensional image sample are determined by the first adaptation network.

[0126] In operation S503, based on the target two-dimensional image sample, the preliminary features of the target two-dimensional image sample are determined using a second feature extraction network.

[0127] In operation S504, based on the preliminary features of the target two-dimensional image sample, the embedding features of the target two-dimensional image sample are determined by the second adaptation network.

[0128] In operation S505, the first adaptation network and the second adaptation network are trained based on the embedding features of the source two-dimensional image sample and the embedding features of the target two-dimensional image sample.

[0129] As mentioned earlier, the first and second feature extraction networks are pre-trained, and this disclosure does not require further training of them. The training process only involves the first and second adaptation networks. Typically, the first and second adaptation networks are trained jointly.

[0130] Operation S505 may optionally include: sampling matching source 2D points and target 2D points from the embedding features of the source 2D image and the embedding features of the target 2D image; determining the distance between the source 2D points and the target 2D points based on the matching source 2D points and the target 2D points; and adjusting the parameters in the first adaptation network and the second adaptation network based on the distance between the source 2D points and the target 2D points. This disclosure is not limited thereto.

[0131] In specific sampling, a uniform sampling method is adopted, uniformly selecting N feature points from both the source and target 2D image embedding features as matching point pairs. Based on the matched source and target 2D points, the cosine similarity between them is calculated as a distance metric. The stochastic gradient descent (SGD) optimization algorithm is used to calculate the contrastive loss L based on the cosine similarity.con The algorithm calculates the gradient of the loss with respect to the parameters of the first and second adapted networks, updates the network parameters in the opposite direction of the gradient, and sets the learning rate to 0.001.

[0132] Specifically, in one aspect of this disclosure, a pair of mesh model samples from a set of 3D mesh model samples are randomly sampled as a source mesh model sample and a target mesh model sample, and the camera pose (C) is randomly sampled based on the prior camera distribution. s C t To render a pair of images (I) src ,I tgt () serves as the source 2D image sample and the target 2D image sample. This random sampling process has a 50% probability that the pose C of the source camera and the target camera will be different. r =C t Of course, other probabilities can be used to make the source camera and target camera poses equal, and this disclosure is not limited thereto. Furthermore, the rendered pair of images (I...) src ,I tgt This can be a normal image or an RGB image. By projecting the true 3D correspondence onto the image plane and utilizing camera pose information, a matching image from the I... src The point and from I tgt The points are monitored. Of course, this disclosure is not limited to this. In this embodiment, two different mesh model samples can be randomly selected from the three-dimensional mesh model sample set as the source mesh model sample and the target mesh model sample. That is, each mesh model in the sample set is assigned the same selection probability, and the corresponding index can be generated by a random number generator to select the sample.

[0133] Specifically, in another aspect of this disclosure, source two-dimensional image samples and target two-dimensional image samples can also be obtained from a two-dimensional image dataset. For example, source two-dimensional images and target two-dimensional image pairs can be uniformly sampled and fed into a first (second) feature extractor and a first (second) adaptation network, respectively, to generate corresponding embedding vectors (X). src ,X tgt Of course, this disclosure is not limited to this.

[0134] Regardless of whether the source and target 2D image samples are obtained by sampling a 3D mesh model sample set or a 2D image dataset, the first and second adaptation networks can be trained using contrastive loss. Contrastive loss encourages the embedding vectors of matching points in the source and target images to be close, thereby achieving accurate point-to-point matching. Specifically, this contrastive loss L... con It can be formalized as follows. L con =CL(X) src (P s),X tgt (P t (11)

[0135] Where CL is a contrast loss similar to CLIP, and P s and P t It is a matching pair of two-dimensional points, X src (·) and X tgt (·) indicates that from X src and X tgt The process of sampling feature points.

[0136] However, relying solely on contrastive loss may introduce a self-similarity problem, where highly similar regions (such as left and right hands) may be incorrectly mapped to the same features. To address this issue, one aspect of this disclosure also proposes that, in operation S505, a first vertex may be sampled from the source mesh model corresponding to the source 2D image sample; a second vertex may be sampled from the target mesh model corresponding to the target 2D image sample, wherein the second vertex has a pseudo-geodesic distance greater than a predetermined threshold with the first vertex; and the parameters in the first and second adaptation networks may be adjusted using the vertex pair consisting of the first and second vertices as negative samples.

[0137] Pseudo-geodetic distance is a metric used to measure the distance between vertices in a source and target mesh model. During the training of the first and second adaptation networks, a pseudo-geodetic distance matrix is ​​introduced to efficiently compute the geographic distance between the two mesh models on a shared mesh model template. Vertices with pseudo-geodetic distances greater than a predetermined threshold are selected as negative samples to avoid erroneous matching of geometrically similar but substantially dissimilar parts between the source and target mesh models.

[0138] If the P matched in formula (11) s and P t If a sample is considered a positive sample, then a pseudo-geodetic distance greater than a predetermined threshold can be considered a negative sample. The loss calculated based on this negative sample can be called the geometry-grounded negative loss, which helps to further distinguish geometrically similar but substantially dissimilar parts in the source and target mesh models.

[0139] For example, for each vertex in the source mesh model, the geometry-guided negative loss avoids mismatches by selecting vertices with greater geographic distance in the target mesh model as negative samples. Specifically, the negative loss, by introducing a pseudo-geodesic distance matrix G, enables efficient computation of the geographic distance between two mesh models on a shared mesh model template. A shared mesh model template is a standardized, unified representation of a mesh model, typically used to provide geometric alignment and mapping between different mesh models. As a "reference model," the shared mesh model template usually has a consistent geometry and can be used to transform multiple source or target mesh models to this standard mesh model through parameterization or some form of mapping, facilitating comparison, pairing, and alignment. The negative loss function L... neg This can be expressed as: L neg =∑(p,q),G(p,q)>th||X src (Π(p,C s ))·X tgt (Π(q,C t ))||2 (12)

[0140] Where G is the pseudo-geodetic distance matrix pre-computed in the shared mesh model template, th is the threshold distance, (p, q) are vertices randomly sampled from the source and target mesh models, respectively, and Π(·) represents the projection operation. The geometry-guided negative loss alleviates the self-similarity problem, ultimately achieving a more robust shape correspondence. Therefore, the total loss used to train the first and second fitting models is defined as: L 2D =L con +λ neg L neg , where λ neg It is a hyperparameter that controls the weight of the negative loss and is used to balance the relationship between the geometrically guided negative loss and the contrastive loss.

[0141] In one embodiment, an apparatus for processing a mesh model is provided, the apparatus comprising:

[0142] The mapping module is configured to: acquire a source two-dimensional image and determine the embedding features of the source two-dimensional image based on the source two-dimensional image; acquire a target two-dimensional image and determine the embedding features of the target two-dimensional image based on the target two-dimensional image; and determine semantic mapping features based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image, wherein the semantic mapping features indicate the correspondence between pixels in the source two-dimensional image and pixels in the target two-dimensional image.

[0143] The deformation module is configured to: iteratively update the deformation model based on the semantic mapping features, and deform and render the source mesh model based on the updated deformation model to obtain a target mesh model that integrates the geometric information of the source mesh model.

[0144] In one embodiment, the mapping module is further configured to: determine preliminary features of the source two-dimensional image using a first feature extraction network based on the source two-dimensional image; and determine embedding features of the source two-dimensional image using a first adaptation network based on the preliminary features of the source two-dimensional image; determine preliminary features of the target two-dimensional image using a second feature extraction network based on the target two-dimensional image; and determine embedding features of the target two-dimensional image using a second adaptation network based on the preliminary features of the target two-dimensional image, wherein the embedding features of the source two-dimensional image and the embedding image of the target two-dimensional image are located in the same embedding space.

[0145] In one embodiment, the mapping module is further configured to: render the source mesh model based on the geometric information of the source mesh model and the camera viewpoint to obtain a source 2D image corresponding to the camera viewpoint; and render the target mesh model based on the geometric information of the target mesh model and the camera viewpoint to obtain a target 2D image corresponding to the camera viewpoint; wherein the geometric information of the source mesh model is different from the geometric information of the target mesh model.

[0146] In one embodiment, the mapping module is further configured to: perform nearest neighbor search based on the embedding features of the source two-dimensional image and the embedding features of the target two-dimensional image to determine the semantic mapping features, wherein the semantic mapping features represent the two-dimensional displacement relationship between source image pixels and target image pixels in the form of a semantic flow mapping graph.

[0147] In one embodiment, the deformation module is further configured to: in each iteration, estimate the position of a vertex in the source mesh model after deformation to the target mesh model using the deformation model; determine the displacement corresponding to the vertex based on the position of the vertex in the source mesh model after deformation to the target mesh model; determine the estimated semantic mapping feature based on the displacement corresponding to the vertex; and update the parameters in the deformation model based on the difference between the estimated semantic mapping feature and the semantic mapping feature.

[0148] In one embodiment, the deformation module is further configured to: in each iteration, use the deformation model to estimate the position of vertices in the source mesh model after deformation to the target mesh model; determine the displacement corresponding to the vertex based on the position of the vertex in the source mesh model after deformation to the target mesh model; determine the mesh model after deformation by the deformation model based on the displacement corresponding to the vertex; and update the parameters in the deformation model based on the difference between the deformed mesh model and the target mesh model.

[0149] In one embodiment, the deformation module is further configured to: in each iteration, use the deformation model to estimate the position of vertices in the source mesh model after deformation to the target mesh model; determine the displacement corresponding to the vertex based on the position of the vertex in the source mesh model after deformation to the target mesh model; determine the mesh model after deformation by the deformation model based on the displacement corresponding to the vertex; perform differentiable rendering on the deformed mesh model to determine the normal map corresponding to the deformed mesh model; determine the normal map corresponding to the target mesh model; and update the parameters in the deformation model based on the difference between the normal map corresponding to the deformed mesh model and the normal map corresponding to the target mesh model.

[0150] In one embodiment, the deformation module is further configured to: in each iteration, estimate the Jacobian matrix of each face of the source mesh model after the vertices in the source mesh model are deformed into the target mesh model using the deformation model; determine the displacement corresponding to the vertex based on the position of the vertex in the source mesh model after being deformed into the target mesh model; determine the mesh model after deformation by the deformation model based on the displacement corresponding to the vertex; determine the Jacobian matrix of each face in the deformed mesh model; and update the parameters in the deformation model based on the difference between the Jacobian matrix of each face in the deformed mesh model and the identity matrix.

[0151] In one embodiment, the deformation module is further configured to: in each iteration, estimate the Jacobian matrix of each face of the source mesh model after the vertices in the source mesh model are deformed into the target mesh model using the deformation model; determine the displacement corresponding to the vertex based on the position of the vertex in the source mesh model after being deformed into the target mesh model; determine the mesh model after deformation by the deformation model based on the displacement corresponding to the vertex; determine the Jacobian matrix of each face in the deformed mesh model and the rotation component of the Jacobian matrix after polar decomposition; and update the parameters in the deformation model based on the difference between the Jacobian matrix of each face in the deformed mesh model and the corresponding rotation component.

[0152] In one embodiment, an apparatus for training a neural network model is provided, including the neural network model, which includes a mapping module and a deformation module. The mapping module is equipped with a first feature extraction network, a second feature extraction network, a first adaptation network, and a second adaptation network.

[0153] A first feature extraction network is configured to determine preliminary features of the source two-dimensional image samples based on the source two-dimensional image samples;

[0154] The first adaptation network is configured to determine the embedding features of the source two-dimensional image samples based on the preliminary features of the source two-dimensional image samples;

[0155] The second feature extraction network is configured to determine preliminary features of the target two-dimensional image sample based on the target two-dimensional image sample;

[0156] The second adaptation network is configured to determine the embedding features of the target two-dimensional image sample based on the preliminary features of the target two-dimensional image sample;

[0157] The device further includes:

[0158] The network training module is used to train the first adaptation network and the second adaptation network based on the embedding features of the source two-dimensional image samples and the embedding features of the target two-dimensional image samples.

[0159] In one embodiment, the network training module is further configured to: sample matching source 2D points and target 2D points from the embedding features of the source 2D image and the embedding features of the target 2D image; determine the distance between the source 2D points and the target 2D points based on the matching source 2D points and the target 2D points; and adjust the parameters in the first adaptation network and the second adaptation network based on the distance between the source 2D points and the target 2D points.

[0160] In one embodiment, the network training module is further configured to: sample a first vertex from a source mesh model corresponding to the source two-dimensional image sample; sample a second vertex from a target mesh model corresponding to the target two-dimensional image sample, wherein the second vertex and the first vertex have a pseudo-geodetic distance greater than a predetermined threshold; and adjust the parameters in the first adaptation network and the second adaptation network using the vertex pair formed by the first vertex and the second vertex as negative samples.

[0161] According to another aspect of this disclosure, an electronic device is also provided for implementing the method according to embodiments of this disclosure. FIG6 shows a schematic diagram of an electronic device 2000 according to an embodiment of this disclosure.

[0162] As shown in Figure 6, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-readable code, which, when executed by the one or more processors 2010, can perform the methods described above.

[0163] The processor in this disclosure embodiment can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, operations, and logic block diagrams disclosed in this disclosure embodiment. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 architecture or an ARM architecture.

[0164] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0165] For example, the methods or apparatus according to embodiments of this disclosure can also be implemented using the architecture of the computing device 3000 shown in FIG. 7. As shown in FIG. 7, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the methods provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, the architecture shown in FIG. 7 is merely exemplary, and one or more components in the computing device shown in FIG. 7 may be omitted as needed when implementing different devices.

[0166] According to another aspect of this disclosure, a computer-readable storage medium is also provided. Figure 8 shows a schematic diagram of the storage medium 4000 according to this disclosure.

[0167] As shown in FIG8, computer-readable instructions 4010 are stored on the computer storage medium 4020. When the computer-readable instructions 4010 are executed by a processor, the methods according to embodiments of the present disclosure described with reference to the above figures can be performed. The computer-readable storage medium in the embodiments of the present disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0168] This disclosure also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform a method according to an embodiment of this disclosure.

[0169] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0170] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0171] In summary, this application provides a method for processing mesh models, a method for training neural networks, an apparatus for processing mesh models, an electronic device, a computer-readable storage medium, and a computer program product. The method involves acquiring a source 2D image and determining its embedding features, acquiring a target 2D image and determining its embedding features, and then determining semantic mapping features based on these two embedding features. These semantic mapping features indicate the correspondence between pixels in the source 2D image and the target 2D image. Next, a deformation model is iteratively updated based on the semantic mapping features. Finally, the updated deformation model is used to deform and render the source mesh model, resulting in a target mesh model that incorporates the geometric information of the source mesh model. This process eliminates the reliance on manual annotation in traditional methods, utilizing 2D image mapping to construct correspondences, achieving accurate deformation from the source mesh model to the target mesh model without manual intervention. From a technical perspective, it avoids the subjectivity and inefficiency problems of manual annotation, improving the processing efficiency and automation of shape correspondence tasks. Furthermore, the semantic mapping features exhibit strong robustness in scenarios with significant changes in mesh model topology, shape, and pose, adapting to complex and varied inputs, providing a stable and reliable foundation for differentiable rendering and iterative optimization, and enhancing the stability and adaptability of the entire processing process.

[0172] Furthermore, when determining the embedding features of the source and target 2D images, based on the source 2D image, a first feature extraction network is used to determine its preliminary features, and then a first adaptation network is used to determine the embedding features of the source 2D image; based on the target 2D image, a second feature extraction network is used to determine its preliminary features, and then a second adaptation network is used to determine the embedding features of the target 2D image, with both located in the same embedding space. The first and second feature extraction networks typically employ pre-trained models, such as ResNet-50, stable diffusion models, or DINO visual converter models. These models have been trained on a large amount of data and possess powerful feature extraction capabilities, enabling them to extract multi-level, multi-scale features from the source and target 2D images. The adaptation network maps these preliminary features to the same embedding space, making the features of different images comparable and consistent. From a technical perspective, this approach reduces the dimensionality and redundancy of features, improves the compactness and expressiveness of features, helps to more accurately determine semantic mapping features subsequently, reduces matching errors caused by inconsistencies in feature spaces, improves the accuracy of feature processing and matching, and enhances the model's ability to capture and utilize features from different images.

[0173] When acquiring source and target 2D images, rendering is performed based on the geometric information of the source and target mesh models, respectively, according to the camera's viewpoint. This results in source and target 2D images corresponding to the camera's viewpoint, with the source and target mesh models having different geometric information. Acquiring images under the same camera viewpoint ensures the consistency of viewpoint between the source and target 2D images, allowing semantic mapping features to more accurately reflect the correspondence between the source and target mesh models. This consistency reduces geometric deformation and occlusion problems caused by viewpoint differences, lowers the complexity of image features, improves the model's ability to process different geometric information, enhances the model's robustness and adaptability in different scenarios, and enables the model to more accurately capture the spatial correspondence between the source and target mesh models.

[0174] When determining semantic mapping features, a nearest neighbor search is performed based on the embedding features of the source 2D image and the target 2D image, representing the 2D displacement relationship between pixels in the source image to pixels in the target image in the form of a semantic flow map. The nearest neighbor search utilizes a KD-tree data structure to accelerate the search process. A KD-tree is an efficient spatial indexing structure that can partition and organize high-dimensional data, significantly reducing the time complexity of the search. Through the KD-tree, the nearest neighbors of each feature vector in the source 2D image embedding features in the target 2D image embedding features can be found quickly and accurately, thus determining the correspondence between pixels. The semantic flow map intuitively displays this correspondence, making it clearer and easier to process. From a technical perspective, this method improves the efficiency and accuracy of correspondence determination, providing a reliable foundation for subsequent deformation model updates. It also facilitates the visualization and analysis of correspondences, contributing to further model optimization.

[0175] In iteratively updating the source mesh model based on semantic mapping features, each iteration uses the deformation model to estimate the position of the vertices of the source mesh model after deformation to the target mesh model. Based on this position, the displacement corresponding to the vertex is determined, thereby determining the estimated semantic mapping features. Then, the parameters in the deformation model are updated according to the difference between the estimated and actual semantic mapping features. Through continuous iteration, the deformation model can gradually adjust its parameters, making the estimated semantic mapping features closer to the actual semantic mapping features, thus enabling the deformation of the source mesh model to more accurately conform to the shape of the target mesh model. From a technical perspective, this iterative optimization method is based on a feedback mechanism, continuously correcting the parameters of the deformation model, improving the accuracy and stability of deformation, and reducing errors and distortions during the deformation process. Simultaneously, the iterative process can utilize differentiable rendering technology to calculate the loss gradient, achieving backpropagation and further optimizing the model, making the deformation process more efficient and accurate.

[0176] Furthermore, during the iterative update of the source mesh model, the deformed position of the vertices is estimated using the deformation model in each iteration. This determines the vertex displacement and the deformed mesh model, and the deformation model parameters are updated based on the differences between the deformed mesh model and the target mesh model. This approach not only considers semantic mapping features but also directly compares the differences between the deformed mesh model and the target mesh model, adjusting the deformation model holistically. From a technical perspective, this comprehensive adjustment method better aligns the deformed mesh model with the target mesh model, improving the quality and accuracy of shape correspondence tasks and ensuring that the geometric information of the source mesh model is more accurately integrated into the target mesh model. Simultaneously, this adjustment method based on overall differences avoids the accumulation of local errors, making the deformation process more stable and reliable.

[0177] During iterative updates of the source mesh model, the deformed mesh model undergoes differentiable rendering to determine its normal map, as well as the normal map of the target mesh model. The deformed model parameters are then updated based on the differences between the two normal maps. The normal map reflects the normal vector information of the mesh model's surface, and the consistency of these normal vectors is crucial for the surface detail and lighting effects of the mesh model. Differentiable rendering technology supports backpropagation to calculate the loss gradient during the rendering process. By comparing the differences in the normal maps and updating the parameters, the deformed mesh model's surface can be made closer to the target mesh model in the normal direction. This improves the matching degree of surface details, resulting in a more natural and accurate effect for the fused target mesh model under lighting, enhancing the model's realism and visual quality. Simultaneously, the combination of differentiable rendering and iterative optimization allows the model to achieve efficient deformation and optimization while preserving surface details.

[0178] During iterative updates of the source mesh model, the deformation model is used to estimate the Jacobian matrix of each face after vertex deformation, determining the Jacobian matrix of each face in the deformed mesh model. The deformation model parameters are then updated based on the difference between this matrix and the identity matrix. The Jacobian matrix describes the local deformation of each face in the mesh model, while the identity matrix represents the ideal situation without deformation. By constraining the difference between the Jacobian matrix and the identity matrix, the transformation matrix of each face in the mesh model can be regularized, avoiding excessive distortion and deformation. From a technical perspective, this increases the stability of the shape during deformation, making the deformation smoother and more natural. Especially during large-scale deformations, it effectively avoids unnatural shape transformations, improving the quality and reliability of the deformation. Simultaneously, this constraint method based on the Jacobian matrix allows for precise control of deformation at the local level, ensuring that the deformation of each face remains within a reasonable range.

[0179] During iterative updates of the source mesh model, the Jacobian matrix of each face in the deformed mesh model and its rotational components after polar decomposition are also determined. The deformed model parameters are updated based on the difference between the two. Polar decomposition decomposes the Jacobian matrix into rotational components and positive definite transformation components. The rotational components describe only pure rotation and do not include shearing or scaling deformation. By comparing the difference between the Jacobian matrix and the rotational components, excessive shearing deformation generated during deformation can be effectively suppressed. From a technical perspective, shearing deformation can lead to distortion of the mesh model shape, especially in non-uniform deformation or high-frequency detail processing. Suppressing shearing deformation ensures the smoothness and naturalness of the mesh model deformation, improves the stability and accuracy of deformation, and makes the fused target mesh model more consistent with expectations. At the same time, the application of polar decomposition allows for more precise analysis and control of deformation, enabling targeted adjustments to different deformation components.

[0180] In training the neural network, based on source 2D image samples, a first feature extraction network is used to determine their preliminary features, and a first adaptation network is used to determine the embedding features of the source 2D image samples. Based on target 2D image samples, a second feature extraction network is used to determine their preliminary features, and a second adaptation network is used to determine the embedding features of the target 2D image samples. Then, the first and second adaptation networks are trained based on the embedding features of the source and target 2D images. The first and second feature extraction networks can extract effective features from the source and target 2D image samples, and the adaptation network maps these features to the same embedding space. From a technical perspective, by training the adaptation network, the network can learn the accurate pixel-to-pixel correspondence between the source and target 2D images, improving the accuracy and generalization ability of the neural network in shape correspondence tasks, enabling the network to better adapt to different image and mesh models. Simultaneously, this training method based on feature extraction and adaptation can fully utilize the knowledge of pre-trained models, accelerate the network's convergence speed, and improve training efficiency.

[0181] During the training of the first and second adaptation networks, matching source and target 2D points are sampled from the embedding features of the source and target 2D images to determine their distance, and the network parameters are adjusted accordingly. By uniformly sampling matching point pairs and calculating their cosine similarity as a distance metric, the network learns the similarity features between matching points. Using stochastic gradient descent to optimize and update the network parameters based on contrastive loss makes the embedding vectors of matching points in the source and target images closer, thus achieving accurate point-to-point matching. From a technical perspective, this training method utilizes the idea of ​​contrastive learning, enhancing the network's ability to judge feature similarity by comparing the features of matching and non-matching points, thereby improving the network's matching accuracy and learning ability. Simultaneously, the use of stochastic gradient descent allows the network to be trained efficiently on large-scale data, avoiding the problem of local optima.

[0182] During the training of the first and second adaptation networks, first vertices are sampled from the source mesh model corresponding to the source 2D image samples, and second vertices are sampled from the target mesh model corresponding to the target 2D image samples. Vertex pairs with pseudo-geodesional distances greater than a predetermined threshold are used as negative samples to adjust the network parameters. Pseudo-geodesional distance effectively measures the distance between vertices in the source and target mesh models. Selecting vertices with larger pseudo-geodesional distances as negative samples avoids erroneous matching of geometrically similar but substantially dissimilar parts between the source and target mesh models. From a technical perspective, by introducing geometrically guided negative loss, the different parts of the source and target mesh models are further distinguished, alleviating the self-similarity problem and improving the robustness and accuracy of the network. This allows the network to more accurately establish correspondences when dealing with complex mesh models. Simultaneously, the use of pseudo-geodesional distance provides an effective metric for negative sample selection, making the selection more reasonable and targeted.

[0183] Furthermore, in retopology scenarios, shape correspondence tasks can determine the correspondence between mesh models with different topological structures, transforming complex source topologies into target topologies more suitable for specific application needs. This improves the naturalness and flexibility of 3D modeling, enabling effective conversion and application of mesh models with different topologies, thus expanding the application scope of 3D modeling. In attribute transfer scenarios, once the correspondence between the source and target mesh models is determined, attributes such as color and texture from the source mesh model can be transferred to the target mesh model. This attribute transfer method, based on accurate shape correspondence, enables fast and accurate attribute assignment, improving the efficiency and quality of attribute transfer, allowing the target mesh model to quickly obtain the required attribute features. In shape interpolation scenarios, interpolation calculations are performed based on the correspondence between multiple mesh models, generating a series of transitional mesh models. From a technical perspective, this provides strong support for animation production and the realization of special deformation effects. Through methods such as linear interpolation, natural and smooth transition effects can be generated, enhancing the realism and visual impact of animation and deformation effects.

[0184] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations of these embodiments or their features can be made without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure. For example, the following provides an overview of some aspects of this disclosure, which can be combined with any other aspects.

Claims

1. A method for processing a mesh model, performed by an electronic device, the method comprising: Acquire a source two-dimensional image, and determine the embedding features of the source two-dimensional image based on the source two-dimensional image; Acquire a target two-dimensional image, and determine the embedding features of the target two-dimensional image based on the target two-dimensional image; Based on the embedding features of the source 2D image and the embedding features of the target 2D image, semantic mapping features are determined, which indicate the correspondence between pixels in the source 2D image and pixels in the target 2D image. Based on the semantic mapping features, the deformation model is iteratively updated, and Based on the updated deformation model, the source mesh model is deformed and rendered to obtain a target mesh model that incorporates the geometric information of the source mesh model.

2. The method as described in claim 1, wherein, Based on the source two-dimensional image, determining the embedding features of the source two-dimensional image includes: Based on the source two-dimensional image, preliminary features of the source two-dimensional image are determined using a first feature extraction network; and Based on the preliminary features of the source two-dimensional image, the embedding features of the source two-dimensional image are determined through a first adaptation network; Based on the target two-dimensional image, determining the embedding features of the target two-dimensional image includes: Based on the target two-dimensional image, a second feature extraction network is used to determine the preliminary features of the target two-dimensional image; and Based on the preliminary features of the target two-dimensional image, the embedding features of the target two-dimensional image are determined through a second adaptation network. The embedding features of the source 2D image and the embedding image of the target 2D image are located in the same embedding space.

3. The method as described in claim 1 or 2, wherein, The acquisition of the source 2D image includes: rendering the source mesh model based on the geometric information of the source mesh model from the camera's perspective to obtain a source 2D image corresponding to the camera's perspective; and The acquisition of the target two-dimensional image includes: rendering the target mesh model based on the geometric information of the target mesh model based on the camera viewpoint, so as to obtain the target two-dimensional image corresponding to the camera viewpoint; The geometric information of the source mesh model differs from that of the target mesh model.

4. The method as described in claim 2 or 3, wherein, The determination of the semantic mapping features based on the embedding features of the source 2D image and the embedding features of the target 2D image includes: Based on the embedding features of the source 2D image and the target 2D image, a nearest neighbor search is performed to determine the semantic mapping features, which represent the 2D displacement relationship between source image pixels and target image pixels in the form of a semantic flow mapping graph.

5. The method according to any one of claims 1 to 4, wherein, The iterative update of the source mesh model based on the semantic mapping features includes: In each iteration, The deformation model is used to estimate the positions of vertices in the source mesh model after deformation to the target mesh model; Based on the position of the vertex in the source mesh model after deformation to the target mesh model, determine the displacement corresponding to the vertex; Based on the displacement corresponding to the vertex, the estimated semantic mapping features are determined; and The parameters in the deformable model are updated based on the difference between the estimated semantic mapping features and the semantic mapping features.

6. The method of claim 5, wherein, The iterative updating of the source mesh model based on the semantic mapping features further includes: In each iteration, The deformation model is used to estimate the positions of vertices in the source mesh model after deformation to the target mesh model; Based on the position of the vertex in the source mesh model after deformation to the target mesh model, determine the displacement corresponding to the vertex; Based on the displacement corresponding to the vertex, determine the mesh model after deformation by the deformation model; and Based on the differences between the deformed mesh model and the target mesh model, the parameters in the deformed model are updated.

7. The method of claim 5 or 6, wherein, The iterative updating of the source mesh model based on the semantic mapping features further includes: In each iteration, The deformation model is used to estimate the positions of vertices in the source mesh model after deformation to the target mesh model; Based on the position of the vertex in the source mesh model after deformation to the target mesh model, determine the displacement corresponding to the vertex; Based on the displacement corresponding to the vertex, determine the mesh model after deformation by the deformation model; The deformed mesh model is rendered differently to determine the normal map corresponding to the deformed mesh model. Determine the normal map corresponding to the target mesh model; and Based on the difference between the normal map corresponding to the deformed mesh model and the normal map corresponding to the target mesh model, the parameters in the deformed model are updated.

8. The method as described in any one of claims 5 to 7, wherein, The iterative updating of the source mesh model based on the semantic mapping features further includes: In each iteration, The Jacobian matrix of each face after the vertices in the source mesh model are deformed into the target mesh model is estimated using the deformation model. Based on the position of the vertex in the source mesh model after deformation to the target mesh model, determine the displacement corresponding to the vertex; Based on the displacement corresponding to the vertex, determine the mesh model after deformation by the deformation model; Determine the Jacobian matrix of each face in the deformed mesh model; and The parameters in the deformed mesh model are updated based on the difference between the Jacobian matrix and the identity matrix of each face in the deformed mesh model.

9. The method according to any one of claims 6 to 8, wherein, The iterative updating of the source mesh model based on the semantic mapping features further includes: In each iteration, The Jacobian matrix of each face after the vertices in the source mesh model are deformed into the target mesh model is estimated using the deformation model. Based on the position of the vertex in the source mesh model after deformation to the target mesh model, determine the displacement corresponding to the vertex; Based on the displacement corresponding to the vertex, determine the mesh model after deformation by the deformation model; Determine the Jacobian matrix of each face in the deformed mesh model and the rotational components of the Jacobian matrix after polar decomposition; and Based on the difference between the Jacobian matrix and the corresponding rotation component of each face in the deformed mesh model, the parameters in the deformed model are updated.

10. A method for training a neural network model, executed by an electronic device, the neural network model comprising a mapping module and a deformation module, the mapping module carrying a first feature extraction network, a second feature extraction network, a first adaptation network, and a second adaptation network, wherein... The method includes: Based on the source two-dimensional image samples, the preliminary features of the source two-dimensional image samples are determined using a first feature extraction network; Based on the preliminary features of the source two-dimensional image samples, the embedding features of the source two-dimensional image samples are determined through a first adaptation network; Based on the target two-dimensional image sample, the preliminary features of the target two-dimensional image sample are determined using a second feature extraction network; Based on the preliminary features of the target two-dimensional image sample, the embedding features of the target two-dimensional image sample are determined through a second adaptation network, and The first adaptation network and the second adaptation network are trained based on the embedding features of the source two-dimensional image samples and the embedding features of the target two-dimensional image samples.

11. The method of claim 10, wherein, The training of the first adaptation network and the second adaptation network includes: Sample matching source 2D points and target 2D points from the embedding features of the source 2D image and the embedding features of the target 2D image; Based on the matched source 2D point and target 2D point, determine the distance between the source 2D point and the target 2D point; and Based on the distance between the source 2D point and the target 2D point, the parameters in the first adaptation network and the second adaptation network are adjusted.

12. The method of claim 10, wherein, The training of the first adaptation network and the second adaptation network further includes: The first vertex is sampled from the source mesh model corresponding to the source two-dimensional image sample; A second vertex is sampled from the target mesh model corresponding to the target two-dimensional image sample, wherein the second vertex has a pseudo-geodetic distance greater than a predetermined threshold from the first vertex; and Using the vertex pair formed by the first vertex and the second vertex as negative samples, the parameters in the first adaptation network and the second adaptation network are adjusted.

13. An apparatus for processing a mesh model, the apparatus comprising: The mapping module is configured as follows: Acquire a source two-dimensional image, and determine the embedding features of the source two-dimensional image based on the source two-dimensional image; Acquire a target two-dimensional image, and determine the embedding features of the target two-dimensional image based on the target two-dimensional image; as well as Based on the embedding features of the source 2D image and the embedding features of the target 2D image, semantic mapping features are determined, which indicate the correspondence between pixels in the source 2D image and pixels in the target 2D image. The transformation module is configured as follows: Based on the semantic mapping features, the deformation model is iteratively updated, and Based on the updated deformation model, the source mesh model is deformed and rendered to obtain a target mesh model that incorporates the geometric information of the source mesh model.

14. An apparatus for training a neural network model, comprising a neural network model, the neural network model including a mapping module and a deformation module, the mapping module carrying a first feature extraction network, a second feature extraction network, a first adaptation network, and a second adaptation network: A first feature extraction network is configured to determine preliminary features of the source two-dimensional image samples based on the source two-dimensional image samples; The first adaptation network is configured to determine the embedding features of the source two-dimensional image samples based on the preliminary features of the source two-dimensional image samples; The second feature extraction network is configured to determine preliminary features of the target two-dimensional image sample based on the target two-dimensional image sample; The second adaptation network is configured to determine the embedding features of the target two-dimensional image sample based on the preliminary features of the target two-dimensional image sample; The device further includes: The network training module is used to train the first adaptation network and the second adaptation network based on the embedding features of the source two-dimensional image samples and the embedding features of the target two-dimensional image samples.

15. An electronic device comprising: One or more processors; and One or more memories, wherein the memories store a computer-executable program that, when executed by the processor, performs the method of any one of claims 1-12.

16. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as claimed in any one of claims 1-12.

17. A computer program product comprising a computer program that, when executed by a processor of an electronic device, implements the method as described in any one of claims 1-12.

Images