Image inpainting method and apparatus, and image inpainting model training method

Abstract

An image inpainting method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring a target bounding box and an inpainting mask of an image to be inpainted (S101); on the basis of the target bounding box, cropping the image to be inpainted to obtain a local image (S102); generating a guide vector on the basis of the local image (S103); inputting the guide vector and the inpainting mask into a diffusion model to obtain a target feature (S104); and inputting the target feature into an image decoder for decoding to obtain an inpainted image (S105).

Description

Image restoration methods, image restoration model training methods and devices

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on December 30, 2024, application number 202411975627.9, entitled "Image Restoration Method, Image Restoration Model Training Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer vision technology, and in particular to image restoration methods, image restoration model training methods and apparatus. Background Technology

[0004] Image generation technology is an important research direction in the field of computer vision, with broad application prospects. However, when the generated image involves a hand, current mainstream text-based image models often exhibit distortion in this area, leading to a decrease in image quality and severely restricting the promotion of practical applications.

[0005] In related technologies, common image restoration algorithms are mostly used for quality improvement tasks such as image denoising and old photo restoration. However, they have limited effect on the restoration of hand distortions (such as finger twisting, incorrect number of fingers, etc.) in text-based image models, and it is difficult to generate restoration results with reasonable hand structure and inconsistent with the background style.

[0006] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0007] This application proposes an image restoration method, an image restoration model training method, and an apparatus.

[0008] One aspect of this application provides an image restoration method, the method comprising the following steps:

[0009] Obtain the target detection bounding box and redraw mask of the image to be repaired;

[0010] A local image is obtained by cropping the image to be repaired based on the target detection box;

[0011] Generate a guiding vector based on the local image;

[0012] The guiding vector and the redrawn mask are input into the diffusion model to obtain the target features;

[0013] The target features are input into an image decoder for decoding to obtain a redrawn image.

[0014] In some embodiments, obtaining the target detection bounding box and redraw mask of the image to be repaired includes:

[0015] Input the image to be repaired into the hand detection model to obtain the initial detection box;

[0016] The initial detection box is expanded outward according to a preset multiple to obtain the target detection box;

[0017] An initial mask is generated based on the target detection bounding box, and edge feathering is performed on the initial mask to obtain a redrawn mask.

[0018] In some embodiments, generating the guiding vector based on the local image includes:

[0019] A three-dimensional model of a human hand is obtained by performing three-dimensional reconstruction based on the local image, and the three-dimensional model is rendered as a depth map;

[0020] The local image is input into the human skeleton extractor to obtain the skeleton image of the hand;

[0021] The local image is input into a pre-trained style encoder to obtain a style feature vector;

[0022] The depth map is input into the depth control network to obtain the depth feature vector;

[0023] The skeleton diagram is input into the skeleton control network to obtain the skeleton feature vector;

[0024] The deep feature vector, the skeleton feature vector, and the style feature vector are input into a feature fusion network and fused to obtain a guiding vector.

[0025] In some embodiments, inputting the guiding vector and the redrawn mask into the diffusion model to obtain target features includes:

[0026] Get the text prompt words;

[0027] The text prompt words are input into a text encoder to obtain a text vector;

[0028] The local image is input into an image encoder to obtain the original image features;

[0029] The text vector, the guiding vector, the original image features, and the redraw mask are input into the diffusion model to obtain the denoising features;

[0030] The target features are obtained by weighting the denoised features and the original image features according to the redraw mask.

[0031] In some embodiments, the method further includes:

[0032] The redrawn image replaces the local image in the image to be repaired to obtain the target repaired image.

[0033] One aspect of this application proposes a method for training an image inpainting model. The image inpainting model includes a diffusion model, a style encoder, a deep control network, a skeleton control network, a feature fusion network, a text encoder, an image encoder, and an image decoder. The method includes the following steps:

[0034] Obtain a training dataset, which includes left-handed-right-handed paired datasets and normal-distorted paired datasets;

[0035] The image restoration model is trained using the training dataset.

[0036] Determine the denoising loss of the diffusion model and obtain the loss threshold;

[0037] The image restoration model is adjusted by the denoising loss, and the step of training the image restoration model with the training dataset is returned until the denoising loss is less than or equal to the loss threshold, thus obtaining the trained image restoration model, which is used to perform image restoration using the above image restoration method.

[0038] In some embodiments, the left-handed-right-handed paired dataset includes left-handed-right-handed image pairs, and the normal-distortion paired dataset includes normal-distortion image pairs. Training the image inpainting model using the training dataset includes:

[0039] The first image from the left-hand-right image pair is input into the depth control network, the skeleton control network, and the image encoder; the second image from the left-hand-right image pair is input into the style encoder.

[0040] The normal image from the normal-distorted image pair is input into the depth control network, the skeleton control network, and the image encoder, while the distorted image from the left-handed-right-handed image pair is input into the style encoder.

[0041] In some embodiments, the target detection box includes a square.

[0042] In some embodiments, the target features include at least one of the following: depth information, skeleton information, style information, skin color information, and texture information of the local image.

[0043] Another aspect of this application provides an image restoration apparatus, the apparatus comprising:

[0044] The detection box acquisition module is used to acquire the target detection box and redraw mask of the image to be repaired;

[0045] The image cropping module is used to crop the image to be repaired according to the target detection box to obtain a local image;

[0046] The vector generation module is used to generate guiding vectors based on the local image;

[0047] The feature processing module is used to input the guiding vector and the redraw mask into the diffusion model to obtain the target features;

[0048] The decoding module is used to input the target features into the image decoder for decoding to obtain the redrawn image.

[0049] Another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0050] Another aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the disclosed drawings without creative effort.

[0052] Figure 1 is a flowchart of the image restoration method provided in an embodiment of this application;

[0053] Figure 2 is a flowchart of step S101 in Figure 1;

[0054] Figure 3 is a flowchart of step S103 in Figure 1;

[0055] Figure 4 is a flowchart of step S104 in Figure 1;

[0056] Figure 5 is a flowchart of the image restoration model training method provided in an embodiment of this application;

[0057] Figure 6 is a flowchart of preparing the training dataset provided in an embodiment of this application;

[0058] Figure 7 is a flowchart of the image restoration method provided in the embodiment of this application applied to an image restoration system;

[0059] Figure 8 is a flowchart of the redrawing process provided in an embodiment of this application;

[0060] Figure 9 is a schematic diagram of the image restoration device provided in an embodiment of this application;

[0061] Figure 10 is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0064] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0066] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0067] 1) DWPose, a human skeleton extractor, is used to detect and extract key skeletal points of the human body from images or videos, thereby enabling the recognition and analysis of human posture.

[0068] 2) CLIP (Contrastive Language-Image Pre-training) is a multimodal model developed by OpenAI that can understand and associate images with natural language descriptions.

[0069] 3) Diffusion model, a deep learning model based on probability generation, is inspired by the molecular diffusion process in physics. The diffusion model generates high-quality data by simulating the forward diffusion process of gradually adding noise to the data and the reverse diffusion process of gradually removing noise using a deep learning model.

[0070] 4) Generative Adversarial Network (GAN) is a deep learning framework. It consists of two main parts: a generator and a discriminator. They compete with each other through an adversarial training process, which enables the generator to generate data that closely approximates the real data distribution.

[0071] In recent years, image generation technology has become an important research direction in the field of computer vision, with broad application prospects. However, when the generated image involves the hand, current mainstream text-based image models often exhibit distortion in this area, leading to a decrease in image quality and severely restricting the promotion of practical applications.

[0072] In related technologies, common image restoration algorithms are mostly used for quality improvement tasks such as image denoising and old photo restoration. However, they have limited effect on the restoration of hand distortions (such as finger twisting, incorrect number of fingers, etc.) in text-based image models, and it is difficult to generate restoration results with reasonable hand structure and inconsistent with the background style.

[0073] In summary, the technical problems existing in the relevant technologies need to be improved.

[0074] In view of this, this application provides an image restoration method, apparatus, device, and medium. This scheme obtains the target detection box and redraw mask of the image to be restored; crops the image to be restored according to the target detection box to obtain a local image; generates a guiding vector according to the local image; inputs the guiding vector and the redraw mask into a diffusion model to obtain target features, maintaining the style consistency of the restoration result and improving the restoration effect; inputs the target features into an image decoder for decoding to obtain a redrawn image, effectively restoring the distorted hand area in the portrait and improving image quality.

[0075] This application also provides a method for training an image restoration model. This method involves acquiring a training dataset, including left-hand-right-hand paired datasets and normal-distorted paired datasets; training an image restoration model using the training dataset; determining the denoising loss of the diffusion model and obtaining a loss threshold; adjusting the image restoration model using the denoising loss; returning to the step of training the image restoration model using the training dataset until the denoising loss is less than or equal to the loss threshold, thus obtaining a trained image restoration model. This image restoration model is used for image restoration using the aforementioned image restoration method, improving the model's generalization ability. The image restoration model can effectively restore distorted hand regions in portraits, improving image quality.

[0076] The image restoration method and image restoration model training method provided in this application relate to the field of computer vision technology. The image restoration method and image restoration model training method provided in this application can be applied to a terminal, a server, or can be software running on a terminal or server.

[0077] In some embodiments, the terminal may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited thereto; the server may be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server may also be a node server in a blockchain network; the software may be an application that implements image restoration methods, but is not limited to the above forms.

[0078] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0079] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0080] Figure 1 is an optional flowchart of an image restoration method provided in an embodiment of this application. The method in Figure 1 may include, but is not limited to, steps S101 to S105.

[0081] Step S101: Obtain the target detection bounding box and redraw mask of the image to be repaired.

[0082] In some embodiments, the image to be repaired is input into a hand detection model to obtain an initial detection box; the initial detection box is expanded outward according to a preset multiple to obtain a target detection box; an initial mask is generated based on the target detection box, and edge feathering operation is performed on the initial mask to obtain a redraw mask.

[0083] In this design, the center coordinates of the detection box remain unchanged before and after expansion. If the expanded detection box extends beyond the image boundary, the pixels exceeding the boundary are filled with black. In some examples, the target detection box is a square.

[0084] Specifically, a mask with all zeros and the same size as the expanded detection bounding box is initialized, and the area of ​​the unexpanded detection bounding box inside the mask is assigned a value of 1. Then, the mask is feathered at the edges to facilitate a harmonious blending of the redrawn content and the background content.

[0085] In this embodiment, obtaining the target detection bounding box and redraw mask of the image to be repaired prepares for subsequent image cropping.

[0086] Step S102: Cropping the image to be repaired based on the target detection box to obtain a local image.

[0087] In some embodiments, a local image is obtained by cropping the image to be repaired based on the target detection bounding box. The location information of the local image is saved to prepare for subsequent image repair.

[0088] Step S103: Generate a guiding vector based on the local image.

[0089] Specifically, the guiding vector is used to guide the model to redraw a reasonable hand structure and a consistent image style.

[0090] In some embodiments, a guiding vector is generated based on feature information of a local image.

[0091] In some examples, features of a local image are extracted and mapped to a vector space.

[0092] In some embodiments, a 3D model of a human hand is obtained by 3D reconstruction based on local images, and the 3D model is rendered as a depth map; the local images are input into a human skeleton extractor to obtain a skeleton map of the hand; the local images are input into a pre-trained style encoder to obtain style feature vectors; the depth map is input into a depth control network to obtain depth feature vectors; the skeleton map is input into a skeleton control network to obtain skeleton feature vectors; the depth feature vectors, skeleton feature vectors, and style feature vectors are input into a feature fusion network for fusion to obtain a guide vector.

[0093] In this embodiment, a guiding vector is generated based on the local image to provide guidance for subsequent image restoration, making the restoration result more natural and coherent, and improving the quality and effect of image restoration.

[0094] Step S104: Input the guiding vector and the redraw mask into the diffusion model to obtain the target features.

[0095] In some embodiments, the guiding vector and the redraw mask are input into the diffusion model, which generates a target feature that corresponds to the local image and is used to fill in missing regions. In some examples, the target feature includes depth information, skeleton information, and information such as style, skin color, and texture of the local image.

[0096] In some examples, the text prompts are obtained; the text prompts are input into a text encoder to obtain text vectors; the local image is input into an image encoder to obtain original image features; the text vectors, guide vectors, original image features, and redraw mask are input into a diffusion model to obtain denoised features; and the denoised features and original image features are weighted according to the redraw mask to obtain target features.

[0097] In this embodiment, the guiding vector and the redraw mask are input into the diffusion model to obtain the target features. The diffusion model can produce restoration results that are more consistent with the image context, reduce the inconsistency between the restored and unrestored parts, and make the restoration results more natural and coherent.

[0098] Step S105: Input the target features into the image decoder for decoding to obtain the redrawn image.

[0099] In some embodiments, the target features are input into an image decoder for decoding to obtain a redrawn image.

[0100] In some examples, self-attention mechanisms or Transformer techniques can be applied to the image decoder to improve the effectiveness and accuracy of image inpainting.

[0101] Furthermore, the redrawn image replaces a local portion of the image to be repaired, resulting in the target repaired image.

[0102] In this embodiment, the target features are input into the image decoder for decoding to obtain the redrawn image, which effectively repairs the distorted hand area in the portrait, improves the image quality, and maintains the style consistency of the repair results, thereby improving the repair effect.

[0103] Steps S101 to S105 as shown in the embodiments of this application involve obtaining the target detection box and redraw mask of the image to be repaired; cropping the image to be repaired based on the target detection box to obtain a local image; generating a guiding vector based on the local image; inputting the guiding vector and the redraw mask into a diffusion model to obtain target features, maintaining the style consistency of the repair result and improving the repair effect; and inputting the target features into an image decoder for decoding to obtain a redrawn image, effectively repairing the distorted hand area in the portrait and improving image quality.

[0104] Please refer to Figure 2. In some embodiments, step S101 may include, but is not limited to, steps S201 to S203:

[0105] Step S201: Input the image to be repaired into the hand detection model to obtain the initial detection box.

[0106] In step S201 of some embodiments, the image to be repaired is identified by a hand detection model to obtain an initial detection box.

[0107] Step S202: Expand the initial detection box outward according to a preset multiple to obtain the target detection box.

[0108] In step S202 of some embodiments, the initial detection box is expanded outward by a preset factor to obtain a target detection box. The center coordinates remain unchanged before and after expansion, and the target detection box has a square shape. If the expanded detection box exceeds the image boundary, the pixels exceeding the boundary are filled with black.

[0109] Understandably, the preset multiplier can be determined in advance or dynamically adjusted according to the application scenario and user needs.

[0110] Specifically, the calculation method for the outer expansion of the detection box is as follows:

[0111] Where (x1,y1) and (x′1,y′1) are the coordinates of the upper left corner of the detection box before and after expansion, respectively, and (x2,y2) and (x′2,y′2) are the coordinates of the lower right corner of the detection box before and after expansion, respectively.

[0112] Step S203: Generate an initial mask based on the target detection box, and perform edge feathering operation on the initial mask to obtain a redrawn mask.

[0113] In some embodiments, a zero-based mask with the same size as the target detection box is initialized, and edge feathering is performed on the initial mask to obtain a redrawn mask.

[0114] In this process, edge feathering is used to smooth the edges. Specifically, the edges of the mask are blurred to reduce sharp transition areas, making the mask edges more natural and smooth.

[0115] In step S203 of some embodiments, the expression for redrawing the mask is as follows: M′=M*G

[0116] Where M is the segmentation mask before feathering, M′ is the segmentation mask after feathering (i.e., the redraw mask), G is the Gaussian kernel function, and * indicates the convolution operation.

[0117] Please refer to Figure 3. In some embodiments, step S103 may include, but is not limited to, steps S301 to S306:

[0118] Step S301: Perform 3D reconstruction based on the local image to obtain a 3D model of the human hand, and render the 3D model as a depth map.

[0119] In step S301 of some embodiments, a three-dimensional model of a human hand is obtained by three-dimensional reconstruction of a local image, and then rendered as a depth map. The depth map depicts a non-distorted, reasonable human hand structure from the perspective of depth information.

[0120] Step S302: Input the local image into the human skeleton extractor to obtain the skeleton image of the hand.

[0121] In step S302 of some embodiments, a local image is input into a human skeleton extractor to obtain a skeleton diagram of the hand. The skeleton diagram describes a non-distorted, reasonable human hand structure from the perspective of skeletal information.

[0122] In some examples, human skeleton extractors include, but are not limited to, DWPose, OpenPose, and DeepCut.

[0123] Step S303: Input the local image into the pre-trained style encoder to obtain the style feature vector.

[0124] In step S303 of some embodiments, the local image is input into a pre-trained style encoder to obtain a style feature vector. The style feature vector contains information such as style, skin color, and texture.

[0125] In some examples, the style encoder includes the image encoder part of CLIP, the mapping network part of StyleGAN2, and AdaIN, etc.

[0126] Step S304: Input the depth map into the depth control network to obtain the depth feature vector.

[0127] In this process, a depth feature vector is extracted from the depth map using a depth control network.

[0128] Step S305: Input the skeleton diagram into the skeleton control network to obtain the skeleton feature vector.

[0129] Specifically, a skeleton control network is used to extract skeleton feature vectors from the skeleton graph.

[0130] Step S306: Input the deep feature vector, skeleton feature vector and style feature vector into the feature fusion network for fusion to obtain the guiding vector.

[0131] In step S306 of some embodiments, feature vectors are fused using a feature fusion network.

[0132] Specifically, the guiding vector is obtained by fusing deep feature vectors, skeleton feature vectors, and style feature vectors through a feature fusion network.

[0133] Please refer to Figure 4. In some embodiments, step S104 may include, but is not limited to, steps S401 to S405:

[0134] Step S401: Obtain text prompt words.

[0135] Specifically, the text prompts are used to describe the image to be repaired.

[0136] In step S401 of some embodiments, the text prompt word input by the user is obtained.

[0137] Step S402: Input the text prompt words into the text encoder to obtain the text vector.

[0138] In step S402 of some embodiments, a text vector is extracted from the text prompt word by a text encoder.

[0139] Step S403: Input the local image into the image encoder to obtain the original image features.

[0140] In step S403 of some embodiments, original image features are extracted from a local image by an image encoder.

[0141] Step S404: Input the text vector, guiding vector, original image features and redraw mask into the diffusion model to obtain denoising features.

[0142] In step S404 of some embodiments, the text vector, the guiding vector, the original image features and the redraw mask are input into the diffusion model. The diffusion model adds Gaussian noise to the image features and then performs iterative denoising operations to obtain denoised features.

[0143] Step S405: The target features are obtained by weighting the denoised features and the original image features according to the redraw mask.

[0144] In step S405 of some embodiments, at each time step t, the denoised features and the original image features are weighted according to the redraw mask to obtain the target features, and the expression of the target features is as follows:

[0145] Where ⊙ represents element-wise multiplication, c text Represents a text vector, x known c represents the original image features. guidance Let m represent the guiding vector, and x represent the redraw mask. t x represents the target feature at time step t. t-1 This represents the target feature at time step t-1.

[0146] Figure 5 is an optional flowchart of the image restoration model training method provided in the embodiments of this application. The method in Figure 5 may include, but is not limited to, steps S501 to S504.

[0147] Step S501: Obtain the training dataset.

[0148] Specifically, the image inpainting model includes a diffusion model, a style encoder, a deep control network, a skeleton control network, a feature fusion network, a text encoder, an image encoder, and an image decoder. The training dataset includes left-handed-right-handed paired datasets and normal-distortion paired datasets.

[0149] In step S501 of some embodiments, the training set is obtained by processing the original image-text pair dataset, comprising two parts: a left-hand-right-hand pair dataset and a normal-distortion pair dataset. The purpose of the left-hand-right-hand pair dataset is to encourage the model to focus on style-related information when extracting and applying style vectors, while minimizing the introduction of style-irrelevant interference information. Human body detection and hand detection are performed on the images in the original dataset. If the images contain the left and right hands of the same person, the local images of the left and right hands are cropped out to form image pairs; otherwise, the data is discarded. The purpose of the normal-distortion pair dataset is to simulate inference scenarios, encouraging the model to learn to extract effective style information from distorted input images, thus aligning training and inference. Its construction method involves uncontrolled redrawing of the hand regions of the normal image, often resulting in distortion. Then, the hand regions of the normal image and the redrawn image are cropped out to form image pairs.

[0150] For example, the flowchart for preparing the training dataset is shown in Figure 6.

[0151] Step S502: Train the image restoration model using the training dataset.

[0152] Specifically, the left-handed-right-handed paired dataset includes left-handed-right-handed image pairs, and the normal-distorted paired dataset includes normal-distorted image pairs.

[0153] In step S502 of some embodiments, the first image from the left-hand-right-hand image pair is input into the depth control network, the skeleton control network, and the image encoder, and the second image from the left-hand-right-hand image pair is input into the style encoder. It is understood that since the left and right hands of the same person often have the same style but different hand poses, this difference helps to prompt the model to focus on style-related information when extracting and applying style features.

[0154] Furthermore, the normal image from the normal-distorted image pair is input into the depth control network, the skeleton control network, and the image encoder, while the distorted image from the left-hand-right-hand image pair is input into the style encoder.

[0155] In this embodiment, the image restoration model is trained using a training dataset. This helps to align the model's training and inference.

[0156] Step S503: Determine the denoising loss of the diffusion model and obtain the loss threshold.

[0157] In step S503 of some embodiments, the denoising loss of the diffusion model is calculated, and the loss function expression of the denoising loss is as follows:

[0158] in, The loss function representing the denoising loss, c text Represents a text vector, c guidance Let m represent the guiding vector, t represent the redraw mask, and t represent the time step. Let ∈ represent the expected value, and let ∈ represent the actual noise. θ This represents the prediction noise of the model.

[0159] Step S504: Adjust the image restoration model by denoising loss, return to the step of training the image restoration model with the training dataset, until the denoising loss is less than or equal to the loss threshold, and obtain the trained image restoration model.

[0160] In step S504 of some embodiments, the image inpainting model is iteratively adjusted by denoising loss.

[0161] Understandably, the loss threshold can be predetermined or dynamically adjusted according to the application scenario or user needs.

[0162] It should be noted that during the training process described above, only the deep control network, the skeleton control network, and the feature fusion network were optimized, while the parameters of the remaining parts of the model were frozen.

[0163] Figure 7 is an optional flowchart of the image restoration method provided in the embodiment of this application applied to an image restoration system. The image restoration system includes a detection module, a condition extraction module, and a redrawing module. The method in Figure 7 may include, but is not limited to, the following steps.

[0164] Step 1: Obtain the hand detection bounding box and redraw mask of the image to be repaired.

[0165] It should be noted that the front end obtains the text prompt words input by the user through the user interface, then calls the text-to-image model of the back end to generate an image, and then obtains the optimized image through the image restoration process of this embodiment. Finally, the optimized image is returned to the front end and provided to the user through the interface.

[0166] In some embodiments, obtaining the hand detection bounding box and redraw mask of the image to be repaired includes the following steps:

[0167] 1. Input the image to be repaired into a hand detection model (e.g., YOLO) to obtain the detection box (i.e., the initial detection box).

[0168] 2. Expand the detection box outwards, keeping the center coordinates unchanged, and increase both the length and width to r times the original size (r>1). The purpose of this step is to provide sufficient contextual information for redrawing distorted regions. Considering that mainstream diffusion models used for redrawing are often trained on large amounts of 1:1 image data, to better utilize the capabilities of the diffusion model, the detection box is further expanded into a square, while keeping the center coordinates unchanged before and after expansion. If the expanded detection box (i.e., the target detection box) exceeds the image boundary, the pixels exceeding the boundary are filled with black. The calculation method for the above detection box expansion is as follows:

[0169] Where (x1,y1) and (x′1,y′1) are the coordinates of the upper left corner of the detection box before and after expansion, respectively, and (x2,y2) and (x′2,y′2) are the coordinates of the lower right corner of the detection box before and after expansion, respectively.

[0170] 3. Initialize a mask with all zeros and the same size as the expanded detection bounding box, and assign a value of 1 to the area of ​​the unexpanded detection bounding box inside it. Then, feather the mask at the edges to ensure a smooth blending of the redrawn content and background content. M′=M*G

[0171] Where M is the segmentation mask before feathering, M′ is the segmentation mask after feathering (i.e., the redraw mask), G is the Gaussian kernel function, and * indicates the convolution operation.

[0172] Step 2: Extract the depth map and skeleton map of the hand, and extract the style features of the hand image.

[0173] In some embodiments, extracting a hand depth map and skeleton map, and extracting style features from the hand image includes the following steps:

[0174] 1. A local image is obtained by cropping based on the detection box, and a 3D model of the human hand is obtained by 3D reconstruction. This model is then rendered as a depth map, which depicts a non-distorted and reasonable human hand structure from the perspective of depth information.

[0175] 2. Input the local image into a human skeleton extractor (e.g., DWPose) to obtain a skeleton map of the hand. Similarly, this skeleton map describes a non-distorted and reasonable human hand structure from the perspective of skeleton information.

[0176] 3. Input the local image into a pre-trained style encoder (e.g., the image encoder part of CLIP) to obtain its style feature vector, which contains information such as style, skin color, and texture.

[0177] Step 3: Redraw the hand area within the detection frame.

[0178] In some embodiments, redrawing the hand region within the detection frame includes the following steps:

[0179] 1. The redrawing flowchart is shown in Figure 8. The hand depth map and skeleton map are input into the depth control network and skeleton control network respectively to obtain the depth feature vector and skeleton feature vector. These two feature vectors, together with the style feature vector, are input into the feature fusion network to obtain the fused feature vector as the final guiding vector, which is used to guide the model to redraw a reasonable hand structure and a consistent image style.

[0180] 2. The hand region is redrawn in the local image defined by the detection box, as shown in Figure 8. First, the text prompt is input into the text encoder to obtain the text vector c. text The local image input image encoder obtains the original image features x. known Subsequently, the text vector and the guiding vector c guidance The original image features and the hand region mask m are input into the diffusion model. The diffusion model adds Gaussian noise to the image features and then performs iterative denoising. At each time step t, the updated image features are obtained by weighting the features processed by the diffusion model and the original features according to a mask:

[0181] Where ⊙ represents element-wise multiplication, c text Represents a text vector, x known c represents the original image features. guidance Let m represent the guiding vector, and x represent the redraw mask. t x represents the target feature at time step t. t-1 This represents the target feature at time step t-1.

[0182] After a preset T-step denoising process, the resulting image feature x0 is input into the image decoder for decoding, yielding the redrawn image. Based on the location determined by the detection box, the redrawn image is pasted back into the image to be repaired to replace the original local image, thus obtaining the final repaired image.

[0183] This embodiment designs a repair method based on guided redrawing. By extracting two hand control conditions, depth map and skeleton map, the diffusion model is guided to redraw a reasonable hand structure. Style features are introduced as guidance, and a feature fusion module is designed to adaptively fuse multiple guidance information to maintain the style consistency of the repair results.

[0184] This embodiment provides a method for constructing a training dataset, which can effectively support the training of the repair model.

[0185] This application embodiment obtains the target detection box and redraw mask of the image to be repaired; crops the image to be repaired based on the target detection box to obtain a local image; generates a guiding vector based on the local image; inputs the guiding vector and the redraw mask into a diffusion model to obtain target features, maintaining the style consistency of the repair result and improving the repair effect; inputs the target features into an image decoder for decoding to obtain a redrawn image, effectively repairing the distorted hand area in the portrait and improving image quality.

[0186] Please refer to Figure 9. This application embodiment also provides an image restoration apparatus that can implement the above-described image restoration method. The apparatus includes:

[0187] The detection box acquisition module 901 is used to acquire the target detection box and redraw mask of the image to be repaired.

[0188] Image cropping module 902 is used to crop the image to be repaired according to the target detection box to obtain a local image;

[0189] Vector generation module 903 is used to generate guiding vectors based on local images;

[0190] Feature processing module 904 is used to input the guiding vector and the redraw mask into the diffusion model to obtain the target features;

[0191] The decoding module 905 is used to input the target features into the image decoder for decoding to obtain the redrawn image.

[0192] In some embodiments, the detection box acquisition module is further configured to:

[0193] Input the image to be repaired into the hand detection model to obtain the initial detection box;

[0194] The initial detection box is expanded outward according to a preset multiple to obtain the target detection box;

[0195] An initial mask is generated based on the target detection bounding box. The initial mask is then feathered at the edges to obtain a redrawn mask.

[0196] In some embodiments, the vector generation module is further configured to:

[0197] A 3D model of a human hand is obtained by 3D reconstruction based on local images, and the 3D model is rendered as a depth map;

[0198] Input a local image into the human skeleton extractor to obtain the skeleton image of the hand;

[0199] The local image is input into a pre-trained style encoder to obtain style feature vectors;

[0200] The depth map is input into the depth control network to obtain the depth feature vector;

[0201] Input the skeleton diagram into the skeleton control network to obtain the skeleton feature vector;

[0202] The deep feature vector, skeleton feature vector, and style feature vector are input into the feature fusion network and fused to obtain the guiding vector.

[0203] In some embodiments, the feature processing module is further configured to:

[0204] Get the text prompt words;

[0205] Input the text prompt words into the text encoder to obtain the text vector;

[0206] The local image is input into the image encoder to obtain the original image features;

[0207] The text vector, guide vector, original image features, and redraw mask are input into the diffusion model to obtain denoising features;

[0208] The target features are obtained by weighting the denoised features and the original image features based on the redraw mask.

[0209] In some embodiments, the apparatus further includes an image inpainting module for:

[0210] The redrawn image replaces a local portion of the image to be repaired, resulting in the target repaired image.

[0211] In some embodiments, the image inpainting model includes a diffusion model, a style encoder, a depth control network, a skeleton control network, a feature fusion network, a text encoder, an image encoder, and an image decoder, and the apparatus includes:

[0212] The acquisition module is used to acquire the training dataset, which includes left-handed-right-handed paired datasets and normal-distorted paired datasets.

[0213] The training module is used to train the image restoration model using the training dataset;

[0214] The threshold module is used to determine the denoising loss of the diffusion model and obtain the loss threshold.

[0215] The image restoration model is adjusted by denoising loss, and the steps of training the image restoration model with the training dataset are returned until the denoising loss is less than or equal to the loss threshold. The trained image restoration model is then obtained, which is used to restore images using image restoration methods.

[0216] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0217] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described image restoration method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0218] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0219] Please refer to Figure 10, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes:

[0220] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0221] The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 using the image restoration method of the embodiments of this application.

[0222] Input / output interface 1003 is used to implement information input and output;

[0223] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0224] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0225] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0226] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image restoration method.

[0227] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0228] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0229] The image restoration method, image restoration apparatus, electronic device, and storage medium provided in this application embodiment acquire the target detection box and redraw mask of the image to be restored; crop the image to be restored according to the target detection box to obtain a local image; generate a guiding vector according to the local image; input the guiding vector and the redraw mask into a diffusion model to obtain target features, maintain the style consistency of the restoration result, and improve the restoration effect; input the target features into an image decoder for decoding to obtain a redrawn image, effectively restoring the distorted hand area in the portrait and improving image quality.

[0230] The image restoration model training method provided in this application involves obtaining a training dataset, which includes left-hand-right-hand paired datasets and normal-distorted paired datasets; training the image restoration model using the training dataset; determining the denoising loss of the diffusion model and obtaining a loss threshold; adjusting the image restoration model using the denoising loss; returning to the step of training the image restoration model using the training dataset until the denoising loss is less than or equal to the loss threshold, thus obtaining a trained image restoration model. The image restoration model is used for image restoration, improving the model's generalization ability and effectively restoring distorted hand regions in portraits, thereby improving image quality.

[0231] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0232] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0233] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0234] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0235] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0236] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0237] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0238] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0239] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0240] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0241] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0242] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image restoration method, the method comprising the following steps: Obtain the target detection bounding box and redraw mask of the image to be repaired; A local image is obtained by cropping the image to be repaired based on the target detection box; Generate a guiding vector based on the local image; The guiding vector and the redrawn mask are input into the diffusion model to obtain the target features; The target features are input into an image decoder for decoding to obtain a redrawn image.

2. The method according to claim 1, wherein, The process of acquiring the target detection bounding box and redraw mask of the image to be repaired includes: The image to be repaired is input into the hand detection model to obtain the initial detection box; The initial detection box is expanded outward according to a preset multiple to obtain the target detection box; An initial mask is generated based on the target detection bounding box, and the initial mask is feathered at the edges to obtain the redrawn mask.

3. The method according to claim 1, wherein, The step of generating the guiding vector based on the local image includes: A three-dimensional model of a human hand is obtained by performing three-dimensional reconstruction based on the local image, and the three-dimensional model is rendered as a depth map; The local image is input into the human skeleton extractor to obtain the skeleton image of the hand; The local image is input into a pre-trained style encoder to obtain a style feature vector; The depth map is input into the depth control network to obtain the depth feature vector; The skeleton diagram is input into the skeleton control network to obtain the skeleton feature vector; The deep feature vector, the skeleton feature vector, and the style feature vector are input into a feature fusion network and fused to obtain the guiding vector.

4. The method according to claim 1, wherein, The step of inputting the guiding vector and the redrawn mask into the diffusion model to obtain target features includes: Get the text prompt words; The text prompt words are input into a text encoder to obtain a text vector; The local image is input into an image encoder to obtain the original image features; The text vector, the guiding vector, the original image features, and the redraw mask are input into the diffusion model to obtain the denoising features; The target feature is obtained by weighting the denoised features and the original image features according to the redraw mask.

5. The method according to claim 1, wherein, The method further includes: The redrawn image replaces the local image in the image to be repaired to obtain the target repaired image.

6. A method for training an image restoration model, wherein, The image inpainting model includes a diffusion model, a style encoder, a depth control network, a skeleton control network, a feature fusion network, a text encoder, an image encoder, and an image decoder. The method includes the following steps: Obtain a training dataset, which includes left-handed-right-handed paired datasets and normal-distorted paired datasets; The image restoration model is trained using the training dataset. Determine the denoising loss of the diffusion model and obtain the loss threshold; The image restoration model is adjusted by the denoising loss, and the step of training the image restoration model with the training dataset is returned until the denoising loss is less than or equal to the loss threshold, so as to obtain the trained image restoration model. The image restoration model is used to perform image restoration by the method described in any one of claims 1-5.

7. The method according to claim 6, wherein, The left-handed-right-handed paired dataset includes left-handed-right-handed image pairs, and the normal-distortion paired dataset includes normal-distortion image pairs. Training the image restoration model using the training dataset includes: The first image from the left-hand-right image pair is input into the depth control network, the skeleton control network, and the image encoder, and the second image from the left-hand-right image pair is input into the style encoder; The normal image from the normal-distorted image pair is input into the depth control network, the skeleton control network, and the image encoder, while the distorted image from the left-handed-right-handed image pair is input into the style encoder.

8. The method according to claim 1, wherein, The target detection box includes a square.

9. The method according to claim 1, wherein, The target features include at least one of the following: depth information, skeleton information, style information, skin color information, and texture information of the local image.

10. An image restoration apparatus, the apparatus comprising: The detection box acquisition module is used to acquire the target detection box and redraw mask of the image to be repaired; The image cropping module is used to crop the image to be repaired according to the target detection box to obtain a local image; The vector generation module is used to generate guiding vectors based on the local image; The feature processing module is used to input the guiding vector and the redraw mask into the diffusion model to obtain the target features; The decoding module is used to input the target features into the image decoder for decoding to obtain the redrawn image.

11. The apparatus according to claim 10, wherein, The detection frame acquisition module is also used for: The image to be repaired is input into the hand detection model to obtain the initial detection box; The initial detection box is expanded outward according to a preset multiple to obtain the target detection box; An initial mask is generated based on the target detection bounding box, and the initial mask is feathered at the edges to obtain the redrawn mask.

12. The apparatus according to claim 10, wherein, The vector generation module is also used for: A three-dimensional model of a human hand is obtained by performing three-dimensional reconstruction based on the local image, and the three-dimensional model is rendered as a depth map; The local image is input into the human skeleton extractor to obtain the skeleton image of the hand; The local image is input into a pre-trained style encoder to obtain a style feature vector; The depth map is input into the depth control network to obtain the depth feature vector; The skeleton diagram is input into the skeleton control network to obtain the skeleton feature vector; The deep feature vector, the skeleton feature vector, and the style feature vector are input into a feature fusion network and fused to obtain the guiding vector.

13. The apparatus according to claim 10, wherein, The feature processing module is also used for: Get the text prompt words; The text prompt words are input into a text encoder to obtain a text vector; The local image is input into an image encoder to obtain the original image features; The text vector, the guiding vector, the original image features, and the redraw mask are input into the diffusion model to obtain the denoising features; The target feature is obtained by weighting the denoised features and the original image features according to the redraw mask.

14. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method according to any one of claims 1 to 9.

15. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images