Method for disentangled reconstruction of dynamic digital human, and electronic device and storage medium

Abstract

The present invention can be applied to the technical field of computer vision and graphics. Provided are a method for disentangled reconstruction of a dynamic digital human, and an electronic device and a storage medium. The method comprises: using a technique for reconstructing three-dimensional shapes of a human body and garments from a monocular human body video, and using a representation method in which explicit geometry is combined with an implicit signed distance field (hmSDF), such that high-quality reconstruction of a dynamic disentangled digital human from a monocular video can be achieved. In the method, garments and a human body are separated and separately modeled, optimized hmSDF is used to achieve accurate segmentation of visible regions, and an SMPL model is also used to complete occluded human body regions, thereby ensuring the consistency and fidelity of the overall geometry. A linear blend skinning (LBS) deformation field and a non-rigid deformation field are used to capture human body motion and detailed variations, such that high-fidelity and continuous spatio-temporally disentangled human body and garment geometry is ultimately generated.

Description

A dynamic digital human decoupling and reconstruction method, electronic device, and storage medium Technical Field

[0001] This invention relates to the fields of computer vision and graphics technology, and more specifically to a dynamic digital human decoupling and reconstruction method, electronic device, and storage medium. Background Technology

[0002] With the development of computer vision and graphics technologies in recent years, there is a wide demand for AI-based virtual digital humans in applications such as virtual try-on, motion-driven systems, and film and television production. Especially in the recently popular metaverse, similar to how humans are a core component of real society, virtual digital human technology is one of the core technologies in the metaverse, and how to efficiently and effectively represent the 3D human body is a widely discussed issue. Among these, using low-dimensional vector parameterization to represent the 3D human body is one of the core technologies in virtual digital humans.

[0003] However, existing virtual digital human technologies suffer from technical problems such as a lack of efficient decoupling modeling methods, difficulty in reconstructing occluded areas, and low modeling accuracy and efficiency. Summary of the Invention

[0004] In view of the above problems, the present invention provides a dynamic digital human decoupling and reconstruction method, electronic device and storage medium to improve the efficiency and accuracy of virtual digital human decoupling and reconstruction.

[0005] According to a first aspect of the present invention, a dynamic digital human decoupling reconstruction method is provided, comprising:

[0006] Image segmentation is performed on the current frame image extracted from the monocular video of the target person to obtain the current frame image segmentation result, and a human-clothing hybrid symbolic distance field is generated by optimizing the initial human geometric template representing the digital human.

[0007] The human-clothing hybrid symbolic distance field is used to separate the human body and clothing in the visible area and complete the human body in the invisible area of ​​the initial human body geometric template, resulting in the completed human body geometric template.

[0008] A decoupled human geometry template is generated using the completed human geometry template, and then deformed using a non-rigid deformation field and a linear hybrid skin deformation field to obtain the deformed human geometry template.

[0009] The deformed human geometric template is optimized by differentiable rendering using the segmentation results of the current frame image to obtain the decoupled reconstruction result of the digital human corresponding to the current frame image. The optimization process of the initial human geometric template, the generation process of the decoupled human geometric template, and the differentiable rendering optimization process are supervised by a predefined loss function.

[0010] The parameters of the non-rigid deformation field are optimized, and digital human decoupling reconstruction is performed on each frame of the monocular video to obtain the dynamic and continuous digital human decoupling reconstruction result of the target person.

[0011] According to an embodiment of the present invention, the above-described image segmentation of the current frame image extracted from the monocular video of the target person to obtain the current frame image segmentation result includes:

[0012] Extract the current frame image from the monocular video of the target person, and use a visual image segmentation algorithm to segment the human body and clothing of the target person in the current frame image to obtain the human body segmentation region and clothing segmentation region of the current frame image.

[0013] According to an embodiment of the present invention, the above-mentioned generation of a human-clothing hybrid symbolic distance field by optimizing an initial human geometry template characterizing a digital human includes:

[0014] Construct a hybrid 3D tetrahedron based on implicit and explicit representation;

[0015] The initial human geometric template representing the digital human is obtained by initializing a hybrid 3D tetrahedron based on implicit and explicit representations.

[0016] The initial human body geometry template is optimized to generate a human body-clothing hybrid symbolic distance field.

[0017] According to an embodiment of the present invention, the above-described method of separating the human body and clothing in the visible region and completing the human body in the invisible region using a human-clothing hybrid symbolic distance field to obtain a completed human body geometric template includes:

[0018] The human body and clothing in the visible area of ​​the initial human body geometric template are separated by using the human body-clothing hybrid symbolic distance field to obtain the separated visible human body area;

[0019] Based on the separated visible human body region, the invisible human body region in the initial human body geometry is completed using a statistically based 3D human body model, resulting in a completed human body geometry template.

[0020] According to an embodiment of the present invention, the above-mentioned method of generating a decoupled human body geometric template using the completed human body geometric template, and deforming the decoupled human body geometric template using a non-rigid deformation field and a linear hybrid skin deformation field to obtain a deformed human body geometric template includes:

[0021] A decoupled human geometry template is generated using the completed human geometry template, wherein the decoupled human geometry template includes a decoupled human template and a decoupled clothing template.

[0022] The decoupled human body template and the decoupled clothing template are subjected to non-rigid deformation using a non-rigid deformation field to obtain a non-rigid deformed human body geometric template.

[0023] The non-rigidly deformed human body geometric template is deformed again using a linear hybrid skin deformation field to achieve dynamic posture changes of the human body and deformation of clothing details, resulting in a deformed human body geometric template.

[0024] According to an embodiment of the present invention, the above-mentioned optimization of the deformed human geometric template by differentiable rendering using the segmentation result of the current frame image to obtain the digital human decoupled reconstruction result corresponding to the current frame image includes:

[0025] By parsing the segmentation results of the current frame image, we can obtain the RGB, normal map, 2D human body mask, and 2D clothing mask of the current frame image.

[0026] By using the RGB, normal map, 2D human body mask, and 2D clothing mask of the current frame image, the deformed human geometric template is rendered using RGB differentiable rendering, normal map rendering, 2D human body mask differentiable rendering, and 2D clothing differentiable rendering, to obtain the decoupled reconstruction result of the target person in the current frame image.

[0027] According to embodiments of the present invention, the predefined loss function includes a color error loss function, a mask matching error loss function, a normal-aware matching error loss function, and a regularization loss function.

[0028] According to an embodiment of the present invention, the above-mentioned regularization loss function includes a programmatic regularization term, a regularization term that encourages hole opening, a hole regularization term, a collision penalty regularization term, and a geometric regularization term.

[0029] A second aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0030] A third aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0031] The dynamic digital human decoupled reconstruction method provided by this invention can decouple and model the geometry and clothing of a 3D dynamic human body separately, rather than simply generating an indivisible overall geometric model. By combining the advantages of explicit and implicit geometry through a hybrid symbolic distance field (hmSDF), it achieves independent representation of clothing and the human body, thereby supporting clothing replacement and personalized editing in various application scenarios. Simultaneously, the method provided by this invention can effectively fill in the geometrically missing areas of the human body caused by clothing occlusion, ensuring the continuity and realism of the shape. Compared to the coarse results of traditional methods that rely solely on simple speculation for completion, this invention significantly improves the geometric accuracy of invisible areas. Moreover, by combining linear blend skinning (LBS) with a non-rigid deformation field, this invention can accurately capture the detailed changes of dynamic clothing under complex human postures, including wrinkles and stretching effects. Compared to the shortcomings of existing methods that can only capture coarse dynamic clothing changes, this invention can generate more realistic dynamic clothing effects. Furthermore, this invention, through differentiable rendering optimization combined with monocular video frame supervision, can quickly generate dynamic human body models under limited hardware conditions. Compared to the inefficiency of traditional methods that rely on multiple cameras and complex templates for modeling, this invention can complete the dynamic modeling of an entire video in just a few hours using only a single-view video, making it highly efficient and low-cost. Attached Figure Description

[0032] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0033] Figure 1 is an application scenario diagram of the dynamic digital human decoupling and reconstruction method according to an embodiment of the present invention;

[0034] Figure 2 is a flowchart of a dynamic digital human decoupling and reconstruction method according to an embodiment of the present invention;

[0035] Figure 3 is a schematic diagram of the process of the dynamic digital human decoupling reconstruction method based on monocular video according to an embodiment of the present invention;

[0036] Figure 4 is a schematic diagram of clothing mask extraction according to an embodiment of the present invention;

[0037] Figure 5 is a schematic diagram of clothing editing according to an embodiment of the present invention;

[0038] Figure 6 is a schematic diagram of human posture editing according to an embodiment of the present invention;

[0039] Figure 7 is a schematic diagram of the structure of the dynamic digital human decoupling and reconstruction device according to an embodiment of the present invention;

[0040] Figure 8 is a block diagram of an electronic device suitable for implementing a dynamic digital human decoupling and reconstruction method according to an embodiment of the present invention. Detailed Implementation

[0041] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0042] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0043] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0044] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0045] Virtual digital humans are widely used in fields such as virtual reality, film and television production, animation generation, virtual try-on, and the metaverse. With the explosive development of the metaverse, research on virtual digital humans has become a hot topic in related technological fields.

[0046] The existing virtual digital human technology still has the following technical problems: (1) Lack of efficient decoupling modeling methods: Most existing monocular video human body reconstruction methods can only generate non-decoupled overall geometry, usually treating the human body and clothing as a whole for modeling. This method cannot achieve independent representation of the human body and clothing, limiting the flexibility of the model in multiple application scenarios. Especially in scenarios such as clothing replacement, virtual try-on, and animation production that require high-precision and editable human body models, existing methods cannot cope with the dynamic changes of different clothing types, resulting in failure to meet customization requirements or unrealistic performance. In addition, the inability to decouple clothing and human body means that the overall model must be remodeled every time the clothing or human body pose is modified, increasing development costs and computation time. (2) Problem of occluded area reconstruction: In monocular video, the parts covered by clothing will cause local occlusion of the human body, making the geometric information of these areas invisible. Existing methods often rely on speculation or simplified model completion when dealing with occluded areas, resulting in a lack of accuracy and coherence in the reconstructed shape of these occluded areas. Especially for clothing in dynamic videos, the shape of the occluded area changes with the change of human posture. Existing methods have difficulty ensuring the detail accuracy and geometric coherence of the model in this case, thus affecting the realism of the overall reconstruction effect. (3) Limited modeling accuracy and efficiency: Traditional human body reconstruction methods usually rely on multiple cameras and complex scanning templates to obtain higher modeling accuracy through multi-view collaboration. However, this method requires a lot of equipment support and computing resources, and is inefficient, which cannot meet the real-time requirements of practical applications. On the other hand, some simplified methods based on monocular video have advantages in modeling speed, but often ignore the capture of details, especially in the representation of clothing folds, texture details and human body surface morphology, which often have large errors, resulting in reconstruction results that are too rough or not accurate enough, and cannot be used in high-quality animation production or virtual reality applications.

[0047] Embodiments of the present invention provide a dynamic digital human decoupling reconstruction method, which is used to at least solve one of the problems in the prior art.

[0048] Figure 1 is an application scenario diagram of the dynamic digital human decoupling and reconstruction method according to an embodiment of the present invention.

[0049] As shown in Figure 1, the application scenario 100 according to this embodiment may include virtual reality, film and television production, animation generation, virtual try-on, and metaverse, etc. Network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0050] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0051] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0052] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0053] It should be noted that the dynamic digital human decoupling and reconstruction method provided in this embodiment of the invention can generally be executed by server 105. Correspondingly, the dynamic digital human decoupling and reconstruction device provided in this embodiment of the invention can generally be located in server 105. The dynamic digital human decoupling and reconstruction method provided in this embodiment of the invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the dynamic digital human decoupling and reconstruction device provided in this embodiment of the invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0054] It should be understood that the number of terminal devices, networks, and servers shown in Figure 1 is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0055] The following will describe in detail the dynamic digital human decoupling and reconstruction method of the disclosed embodiment based on the scenario described in Figure 1, with reference to Figures 2 to 6.

[0056] In the field of dynamic human and clothing modeling, traditional methods often struggle to accurately represent both the human body and complex clothing dynamics simultaneously. Existing methods rely on pre-built templates, such as human body templates or templates for fixed clothing types. These methods have limited expressive power for dynamic clothing, often constrained by the template's resolution and topology, making it difficult to capture the rich details of clothing folds and their dynamic changes. To address this, this invention proposes a dynamic human body modeling method based on a combination of explicit and implicit geometry. By combining the clear boundaries of explicit geometry with the high expressive power of implicit functions, it learns prior knowledge of human and clothing dynamics from monocular video and decouples the human and clothing representations using a hybrid symbolic distance field (hmSDF). This method requires no predefined clothing templates and can generate high-precision, richly detailed dynamic clothing shapes with only a few network parameters. Simultaneously, it enables independent editing and optimization of both the human body and clothing, significantly improving modeling flexibility and realism.

[0057] It should be specifically noted that the information (including but not limited to the target person's image information, posture information, action information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the target person or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, they do not violate public order and good morals, and corresponding operation entry points are provided for the target person to choose to authorize or refuse.

[0058] In scenarios involving automated decision-making using target information, the methods, devices, and systems provided in this invention offer users corresponding entry points for them to choose whether to agree to or reject the automated decision result. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0059] Figure 2 is a flowchart of the dynamic digital human decoupling and reconstruction method according to an embodiment of the present invention.

[0060] As shown in Figure 2, the above-mentioned dynamic digital human decoupling reconstruction method includes operations S210 to S250.

[0061] In operation S210, the current frame image extracted from the monocular video of the target person is segmented to obtain the current frame image segmentation result, and a human-clothing hybrid symbolic distance field is generated by optimizing the initial human body geometric template representing the digital human.

[0062] Frame images are extracted from the monocular video of the target person to obtain a set of frame images, i.e., a frame sequence. Each frame image is then segmented to obtain the human body region and clothing region of the target person. Note that the image segmentation results are used as "ground truth labels" in subsequent operations for rendering and optimizing the digital human's geometric template. The image segmentation results include the image's RGB information, normal map information, 2D human body mask, and 2D clothing mask.

[0063] The aforementioned initial human geometry template is constructed based on DMTet, combining implicit and explicit representations.

[0064] In operation S220, the human-clothing hybrid symbolic distance field is used to separate the human body and clothing in the visible area and complete the human body in the invisible area of ​​the initial human body geometric template, resulting in the completed human body geometric template.

[0065] In the above operation S220, the human body and clothing are first separated. Due to the presence of clothing, only part of the human body will remain in the visible area after separation. Therefore, it is necessary to complete the human body in the invisible area.

[0066] In operation S230, a decoupled human geometry template is generated using the completed human geometry template, and the decoupled human geometry template is deformed using a non-rigid deformation field and a linear hybrid skin deformation field to obtain the deformed human geometry template.

[0067] Through multiple deformations using non-rigid deformation fields and linear hybrid skin deformation fields, the details of the human body and clothing in the human body geometry template are enriched.

[0068] The aforementioned non-rigid deformation field can optionally be a multilayer perceptron (MLP) employing a fully connected neural network, including an input layer, a fully connected layer, an activation layer, and skip connections.

[0069] In operation S240, the segmentation results of the current frame image are used to perform differentiable rendering optimization on the deformed human geometric template to obtain the decoupled reconstruction result of the digital human corresponding to the current frame image. The optimization process of the initial human geometric template, the generation process of the decoupled human geometric template, and the differentiable rendering optimization process are supervised by a predefined loss function.

[0070] Throughout the process, a predefined loss function is used to supervise the decoupling and reconstruction process, thereby optimizing the obtained digital human decoupling and reconstruction results.

[0071] By operating the S250, optimizing the parameters of the non-rigid deformation field, and performing digital human decoupling reconstruction on each frame of the monocular video, the dynamic and continuous digital human decoupling reconstruction result of the target person is obtained.

[0072] For each frame of the monocular video of the target person, operations S210 to S240 are performed to obtain the decoupled reconstruction result of the digital human in each frame. The decoupled reconstruction result of the digital human in each frame is output in the frame image sequence to obtain the dynamic and continuous decoupled reconstruction result of the target person, that is, the dynamic and continuous digital human model of the target person.

[0073] The dynamic digital human decoupled reconstruction method provided by this invention can decouple and model the geometry and clothing of a 3D dynamic human body separately, rather than simply generating an indivisible overall geometric model. By combining the advantages of explicit and implicit geometry through a hybrid symbolic distance field (hmSDF), it achieves independent representation of clothing and the human body, thereby supporting clothing replacement and personalized editing in various application scenarios. Simultaneously, the method provided by this invention can effectively fill in the geometrically missing areas of the human body caused by clothing occlusion, ensuring the continuity and realism of the shape. Compared to the coarse results of traditional methods that rely solely on simple speculation for completion, this invention significantly improves the geometric accuracy of invisible areas. Moreover, by combining linear hybrid skinning (LBS) with a non-rigid deformation field, this invention can accurately capture the detailed changes of dynamic clothing under complex human postures, including wrinkles and stretching effects. Compared to the shortcomings of existing methods that can only capture coarse dynamic clothing changes, this invention can generate more realistic dynamic clothing effects. Furthermore, this invention, through differentiable rendering optimization combined with monocular video frame supervision, can quickly generate dynamic human body models under limited hardware conditions. Compared to the inefficiency of traditional methods that rely on multiple cameras and complex templates for modeling, this invention can complete the dynamic modeling of an entire video in just a few hours using only a single-view video, making it highly efficient and low-cost.

[0074] The dynamic digital human decoupling and reconstruction method provided by the present invention will be further described in detail below through specific embodiments and in conjunction with Figure 3.

[0075] Figure 3 is a schematic diagram of the process of the dynamic digital human decoupling reconstruction method based on monocular video according to an embodiment of the present invention.

[0076] Frame sequences are extracted from monocular video, and 2D masks of clothing and human body are obtained through image segmentation. As shown in Figure 3, a human-clothing hybrid symbolic distance field (hmSDF) is constructed to separate the visible areas of the human body and clothing and fill in the invisible areas. Static templates are generated based on hmSDF, including decoupled clothing and human body templates. A linear hybrid skinning (LBS) deformation field is constructed and combined with a non-rigid deformation field to realize dynamic pose changes of the human body and deformation of clothing details. Through differentiable rendering optimization, a high-fidelity decoupled geometric model is generated under supervision based on RGB, normal maps, and 2D masks. The parameters of the non-rigid deformation field are optimized to output a dynamically continuous decoupled reconstruction result of the human body and clothing.

[0077] According to an embodiment of the present invention, the above-mentioned image segmentation of the current frame image extracted from the monocular video of the target person to obtain the current frame image segmentation result includes: extracting the current frame image from the monocular video of the target person, and using a visual image segmentation algorithm to segment the human body and clothing of the target person in the current frame image to obtain the human body segmentation region and clothing segmentation region of the current frame image.

[0078] The following detailed description, using specific embodiments and in conjunction with Figure 4, further illustrates the input and segmentation of the monocular video provided by the present invention.

[0079] Figure 4 is a schematic diagram of clothing mask extraction according to an embodiment of the present invention.

[0080] Extract frame sequences {I} from the input monocular video. t |t=1,...,N}, and generate segmentation masks for clothing and the human body using existing 2D image segmentation methods.

[0081] Specifically, using open-source image segmentation tools such as SAM2 (Segment Anything 2), each frame of the image is parsed into independent regions of the human body and clothing. The segmentation results are shown in Figure 4, from left to right: the captured color image of the person in clothing, the complete clothing mask obtained from SAM2, the clothing mask obtained from SAM2, the mask obtained by only presenting the clothing geometry, and the mask of the effective clothing region after presenting the complete clothing geometry.

[0082] According to an embodiment of the present invention, the above-mentioned generation of a human-clothing hybrid symbolic distance field by optimizing the initial human body geometric template representing a digital human includes: constructing a hybrid 3D tetrahedron based on implicit and explicit representations; initializing the hybrid 3D tetrahedron based on implicit and explicit representations to obtain an initial human body geometric template representing a digital human; and optimizing the initial human body geometric template to generate a human-clothing hybrid symbolic distance field.

[0083] The initial human geometry template described above is generated based on DMTet, a deep learning version of the Marching Tetrahedra algorithm for high-resolution 3D shape synthesis. It combines implicit and explicit 3D representations to optimize surface reconstruction for fine geometric details. Through an end-to-end differentiable process, DMTet can generate 3D models from point clouds or coarse voxel inputs.

[0084] According to an embodiment of the present invention, the above-mentioned separation of the human body and clothing in the visible area and completion of the human body in the invisible area of ​​the initial human body geometric template using a human-clothing hybrid symbolic distance field to obtain a completed human body geometric template includes: separating the human body and clothing in the visible area of ​​the initial human body geometric template using a human-clothing hybrid symbolic distance field to obtain a separated visible human body area; and completing the invisible human body area in the initial human body geometry using a statistically based three-dimensional human body model based on the separated visible human body area to obtain a completed human body geometric template.

[0085] Human body S through hmSDF b Clothing S c The segmentation of the visible region is defined as shown in formula (1):

[0086] Where λ is the segmentation boundary. For the occluded region S b The geometric parameters of the SMPL model are used to complete the model.

[0087] According to an embodiment of the present invention, the above-mentioned method of generating a decoupled human geometry template using a completed human geometry template, and deforming the decoupled human geometry template using a non-rigid deformation field and a linear hybrid skin deformation field to obtain a deformed human geometry template includes: generating a decoupled human geometry template using a completed human geometry template, wherein the decoupled human geometry template includes a decoupled human geometry template and a decoupled clothing template; performing non-rigid deformation on the decoupled human geometry template and the decoupled clothing template respectively using a non-rigid deformation field to obtain a non-rigid deformed human geometry template; and further deforming the non-rigid deformed human geometry template using a linear hybrid skin deformation field to achieve dynamic posture changes of the human body and deformation of clothing details to obtain a deformed human geometry template.

[0088] The LBS deformation of the human body and the non-rigid deformation field of the clothing are modeled independently, and their dynamic consistency is ensured during the fusion process. The human body deformation is described by linear hybrid skin (LBS), as shown in Equation (2):

[0089] Where w j (x) is the skeleton weight, G j It is the rigid transformation matrix of the skeleton.

[0090] The non-rigid deformation field of the garment is modeled by MLP, as shown in Equation (3):

[0091] Where x is a point in the standard pose space, h t It is a latent variable in clothing, φ c These are network parameters.

[0092] According to an embodiment of the present invention, the above-mentioned optimization of the deformed human geometric template by differentiable rendering using the segmentation result of the current frame image to obtain the digital human decoupled reconstruction result corresponding to the current frame image includes: parsing the segmentation result of the current frame image to obtain the RGB, normal map, 2D human mask and 2D clothing mask of the current frame image; using the RGB, normal map, 2D human mask and 2D clothing mask of the current frame image to perform RGB differentiable rendering, normal map rendering, 2D human mask differentiable rendering and 2D clothing differentiable rendering of the deformed human geometric template to obtain the digital human decoupled reconstruction result of the target person under the current frame image.

[0093] The purpose of differentiable rendering optimization is to combine the human geometric template with the target person to generate a digital human model that is similar to the target person.

[0094] According to embodiments of the present invention, the predefined loss function includes a color error loss function, a mask matching error loss function, a normal-aware matching error loss function, and a regularization loss function.

[0095] According to an embodiment of the present invention, the above-mentioned regularization loss function includes a programmatic regularization term, a regularization term that encourages hole opening, a hole regularization term, a collision penalty regularization term, and a geometric regularization term.

[0096] The loss function for the optimization process includes color error, mask matching error, normal-aware matching error, and geometric regularization.

[0097] As shown in formula (4): L total =λ mask L mask +λ color L color +λ per L per +λ reg L reg (4).

[0098] During training, the model is optimized by minimizing the difference between the rendered result and the input image. This specifically includes the following loss mechanisms:

[0099] (1) Color Loss

[0100] This loss term measures the reconstruction accuracy by calculating the L1 error between the rendered RGB image and the supervised image (the real image). As shown in Equation (5):

[0101] Where 1(p) represents the category to which the current pixel p belongs. b When (p) is true, pixel p belongs to the human body; when 1 cWhen (p) is true, pixel p belongs to the clothing category. 1 b (p) and 1 c (p) can be true at all times, or only one of them can be true, depending on the mask used for supervision. R C-b It is a rendered RGB image of the human body, R C-c It is a rendered RGB image of the clothing, I C-b It is a true RGB image of the human body, I C-c It is a true RGB image of the clothing.

[0102] This loss term calculates pixel-level errors by comparing rendered images of the human body and clothing with real images, ensuring that the colors of the reconstructed image are as similar as possible to the real image.

[0103] (2) Mask Loss

[0104] Although irrelevant backgrounds have already been removed from the RGB image, mask loss can further constrain the accuracy of the boundaries. As shown in Equation (6):

[0105] Among them, I M-b It is the true mask of the human body, I M-c It is the true mask of the clothing. Mask loss further helps to optimize the accuracy of the boundary areas, ensuring the realistic reproduction of the boundaries between clothing and the human body.

[0106] (3) Perceptual Normal Loss

[0107] By using the Sapiens dataset to obtain the image's normal map as supervision, the effect of normal reconstruction is further enhanced. As shown in Equation (7):

[0108] Among them, R N It's rendering normals, I N It is the true normal, φ i (*) denotes the activation function of the i-th layer in the MobileNetV2 network. This loss term aims to improve the reconstruction quality of the normal map by using perceptual loss to enhance the matching between the rendered normals and the real normals, thereby improving the perceptual quality of the normals.

[0109] (4) Regularization Loss

[0110] Regularization terms help control the model's behavior during optimization, avoiding overfitting or unreasonable results. This section includes the following regularization losses:

[0111] (4.1) Eikonal Loss

[0112] To ensure the validity of the Signed Distance Field (SDF), an Eikonal term is added to the gradient g of the SDF value of each tetrahedron vertex when optimizing the SDF field, as shown in Equation (8):

[0113] The Eikonal loss term ensures that the gradient of the symbolic distance field meets expectations during the optimization process, thereby guaranteeing the correctness of the SDF value.

[0114] (4.2) Encourage Hole Opening

[0115] To identify opening locations using only image information (especially when the viewing angle is limited), a regularization term that encourages hole opening is introduced, as shown in Equation (9):

[0116] Where v(u) represents the signed distance of vertex u; L huber It is the Huber loss, used to apply smooth constraints to the opening.

[0117] This feature helps the model identify reasonable opening locations using image information, preventing incorrect opening recognition due to viewing angle limitations.

[0118] (4.3)Regularize Holes

[0119] To avoid an excessively large opening, constraints are imposed on all points visible from the current viewpoint, as shown in equation (10):

[0120] Here, ∈1 is a positive number used to control the maximum range of the opening.

[0121] This regularization term ensures that the opening of the sign distance field is not too large during the optimization process, thereby preventing unreasonable geometric deformation.

[0122] (4.4) Collision Penalty

[0123] To ensure that the clothing does not penetrate the surface of the human body, a collision penalty is introduced, as shown in formula (11):

[0124] Where d(x) represents the minimum distance from point x on the clothing surface to the human body surface; ∈2 is the distance threshold, which is usually set to 0.005 to prevent calculation errors; k collision It is the weight of the collision penalty.

[0125] (4.5) Geometry Regularization

[0126] To ensure the optimization process is constrained and produces smooth deformation results, this invention introduces geometric regularization terms. This invention adds the following two regularization terms: Normal Consistency Term: denoted as L n_consist Used to maintain the consistency of surface normals; Laplacian term: denoted as L laplacian It is used to smooth geometric deformations.

[0127] Figure 5 is a schematic diagram of clothing editing according to an embodiment of the present invention.

[0128] Figure 6 is a schematic diagram of human posture editing according to an embodiment of the present invention.

[0129] After obtaining the decoupled reconstruction model of the target person using the dynamic digital human decoupling reconstruction method provided by this invention, the digital human decoupling reconstruction model can be edited, including the editing of clothing and human posture.

[0130] As shown in Figure 5, Figure 5(a) shows a person about to undergo clothing editing, the top three pictures in Figure 5(b) show the clothes being changed, and the bottom three pictures in Figure 5(b) show the result after changing the clothes on the person in Figure 5(a).

[0131] Figure 6 is a schematic diagram of human posture editing. Figure 6(a) shows a clothed human body in a standard posture, and the three figures in Figure 6(b) show the editing results of three different postures.

[0132] The clothing editing and human posture editing shown in Figures 5 and 6 demonstrate that the dynamic digital human decoupling reconstruction method provided by this invention can achieve efficient editing of the target digital human model.

[0133] Compared with traditional three-dimensional human body parametric representation methods, the above-described solution of the present invention has the following advantages:

[0134] (1) Decoupled Reconstruction of Dynamic Human Body and Clothing: This invention can decouple the geometry of the three-dimensional dynamic human body and clothing separately, instead of simply generating an indivisible overall geometric model. By combining the advantages of explicit and implicit geometry through a hybrid symbolic distance field (hmSDF), independent representation of clothing and human body is achieved, thereby supporting clothing replacement and personalized editing in various application scenarios.

[0135] (2) Precise completion of occluded areas: With the help of the SMPL model and geometric optimization technology, this invention can effectively complete the geometrically missing areas of the human body caused by clothing occlusion, ensuring the continuity and realism of the shape. Compared with the rough effect of traditional methods that rely on simple speculation for completion, this invention significantly improves the geometric accuracy of invisible areas.

[0136] (3) Efficient capture of dynamic clothing details: By combining linear hybrid skin (LBS) with a non-rigid deformation field, this invention can accurately capture the detailed changes of dynamic clothing under complex human postures, including wrinkles and stretching effects. Compared with existing methods that can only capture rough dynamic clothing changes, this invention can generate more realistic dynamic clothing effects.

[0137] (4) High efficiency and high quality: This invention combines differentiable rendering optimization with supervision of monocular video frames, enabling the rapid generation of dynamic human models under limited hardware conditions. Compared to the low efficiency of traditional methods that rely on multiple cameras and complex templates for modeling, this invention only requires monocular video to complete the dynamic modeling of the entire video in a few hours, exhibiting high efficiency and low cost.

[0138] Corresponding to the aforementioned embodiments of the three-dimensional clothing human body parameterization representation method based on explicit-implicit combination, the present invention also provides embodiments of a three-dimensional clothing human body parameterization representation device based on explicit-implicit combination.

[0139] Figure 7 is a schematic diagram of the structure of the dynamic digital human decoupling and reconstruction device according to an embodiment of the present invention.

[0140] As shown in Figure 7, the above-mentioned dynamic digital human decoupling reconstruction device 700 includes an image segmentation and template initial optimization module 710, a separation and completion module 720, a multiple deformation module 730, a differentiable rendering and supervised training module 740, and a dynamic continuous decoupling reconstruction module 750.

[0141] The image segmentation and template initial optimization module 710 is used to perform image segmentation on the current frame image extracted from the monocular video of the target person, obtain the current frame image segmentation result, and generate a human-clothing hybrid symbolic distance field by optimizing the initial human body geometric template representing the digital human. In one embodiment, the image segmentation and template initial optimization module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0142] The separation and completion module 720 is used to separate the human body and clothing in the visible area and complete the human body in the invisible area of ​​the initial human body geometric template using a human body-clothing hybrid symbolic distance field, to obtain a completed human body geometric template. In one embodiment, the separation and completion module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0143] The multiple deformation module 730 is used to generate a decoupled human geometry template using the completed human geometry template, and to deform the decoupled human geometry template using a non-rigid deformation field and a linear hybrid skin deformation field to obtain the deformed human geometry template. In one embodiment, the multiple deformation module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0144] The differentiable rendering and supervised training module 740 is used to perform differentiable rendering optimization on the deformed human geometric template using the segmentation results of the current frame image, to obtain the decoupled reconstruction result of the digital human corresponding to the current frame image, and to supervise the optimization process of the initial human geometric template, the generation process of the decoupled human geometric template, and the differentiable rendering optimization process using a predefined loss function. In one embodiment, the differentiable rendering and supervised training module 740 can be used to execute the operation S240 described above, which will not be repeated here.

[0145] The dynamic continuous decoupling reconstruction module 750 is used to optimize the parameters of the non-rigid deformation field and perform digital human decoupling reconstruction on each frame of the monocular video to obtain a dynamic continuous digital human decoupling reconstruction result of the target person. In one embodiment, the dynamic continuous decoupling reconstruction module 750 can be used to perform the operation S250 described above, which will not be repeated here.

[0146] According to embodiments of this disclosure, any multiple modules among the image segmentation and template initial optimization module 710, separation and completion module 720, multiple deformation module 730, differentiable rendering and supervised training module 740, and dynamic continuous decoupling reconstruction module 750 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the image segmentation and template initial optimization module 710, separation and completion module 720, multiple deformation module 730, differentiable rendering and supervised training module 740, and dynamic continuous decoupling reconstruction module 750 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), programmable logic array (PLA), system-on-a-chip, system-on-a-substrate, system-on-package, application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, and firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the image segmentation and template initial optimization module 710, separation and completion module 720, multiple deformation module 730, differentiable rendering and supervised training module 740, and dynamic continuous decoupled reconstruction module 750 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0147] Figure 8 is a block diagram of an electronic device suitable for implementing a dynamic digital human decoupling and reconstruction method according to an embodiment of the present invention.

[0148] As shown in FIG8, an electronic device 800 according to an embodiment of the present invention includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0149] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0150] According to an embodiment of the present invention, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0151] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0152] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0154] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0155] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A dynamic digital human decoupling and reconstruction method, characterized in that, The method includes: Image segmentation is performed on the current frame image extracted from the monocular video of the target person to obtain the current frame image segmentation result, and a human-clothing hybrid symbolic distance field is generated by optimizing the initial human geometric template representing the digital human. The human-clothing hybrid symbolic distance field is used to separate the human body and clothing in the visible area and complete the human body in the invisible area of ​​the initial human body geometric template to obtain the completed human body geometric template. A decoupled human geometry template is generated using the completed human geometry template, and the decoupled human geometry template is deformed using a non-rigid deformation field and a linear hybrid skin deformation field to obtain a deformed human geometry template. The deformed human geometric template is optimized by differentiable rendering using the segmentation result of the current frame image to obtain the decoupled reconstruction result of the digital human corresponding to the current frame image. The optimization process of the initial human geometric template, the generation process of the decoupled human geometric template, and the differentiable rendering optimization process are supervised by a predefined loss function. The parameters of the non-rigid deformation field are optimized, and digital human decoupling reconstruction is performed on each frame of the monocular video to obtain the dynamic and continuous digital human decoupling reconstruction result of the target person.

2. The method according to claim 1, characterized in that, Image segmentation is performed on the current frame image extracted from the monocular video of the target person, and the current frame image segmentation results include: The current frame image is extracted from the monocular video of the target person, and the human body and clothing of the target person in the current frame image are segmented using a visual image segmentation algorithm to obtain the human body segmentation region and clothing segmentation region of the current frame image.

3. The method according to claim 1, characterized in that, The generation of a human-clothing hybrid symbolic distance field by optimizing the initial human geometry template representing the digital human includes: Construct a hybrid 3D tetrahedron based on implicit and explicit representation; The hybrid 3D tetrahedron based on implicit and explicit representation is initialized to obtain the initial human geometric template representing the digital human. The initial human body geometry template is optimized to generate the human body-clothing hybrid symbolic distance field.

4. The method according to claim 1, characterized in that, Using the human-clothing hybrid symbolic distance field, the initial human geometry template is subjected to separation of the human body and clothing in the visible region and completion of the human body in the invisible region, resulting in a completed human geometry template including: The human body and clothing in the visible area of ​​the initial human body geometric template are separated using the human-clothing hybrid symbolic distance field to obtain the separated visible human body area; Based on the separated visible human body region, the invisible human body region in the initial human body geometry is completed using a statistically based 3D human body model to obtain the completed human body geometry template.

5. The method according to claim 1, characterized in that, A decoupled human geometry template is generated using the completed human geometry template, and then deformed using a non-rigid deformation field and a linear hybrid skin deformation field to obtain a deformed human geometry template, including: A decoupled human geometry template is generated using the completed human geometry template, wherein the decoupled human geometry template includes a decoupled human body template and a decoupled clothing template. The decoupled human body template and the decoupled clothing template are subjected to non-rigid deformation using the non-rigid deformation field to obtain a non-rigid deformed human body geometric template. The non-rigidly deformed human geometric template is deformed again using the linear hybrid skin deformation field to achieve dynamic posture changes of the human body and deformation of clothing details, thus obtaining the deformed human geometric template.

6. The method according to claim 1, characterized in that, The deformed human geometric template is optimized by differentiable rendering using the segmentation results of the current frame image to obtain the decoupled reconstruction result of the digital human corresponding to the current frame image, including: By parsing the segmentation results of the current frame image, the RGB, normal map, 2D human body mask, and 2D clothing mask of the current frame image are obtained. Using the RGB, normal map, 2D human body mask, and 2D clothing mask of the current frame image, the deformed human geometric template is subjected to RGB differentiable rendering, normal map rendering, 2D human body mask differentiable rendering, and 2D clothing differentiable rendering to obtain the digital human decoupled reconstruction result of the target person under the current frame image.

7. The method according to claim 1, characterized in that, The predefined loss functions include color error loss function, mask matching error loss function, normal-aware matching error loss function, and regularization loss function.

8. The method according to claim 7, characterized in that, The regularization loss function includes a procedural regularization term, a regularization term that encourages hole opening, a hole regularization term, a collision penalty regularization term, and a geometric regularization term.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images