Facial image editing method and apparatus, device, storage medium, and product

Abstract

Embodiments of the present application provide a facial image editing method and apparatus, a device, a storage medium, and a product. In a technical solution provided in an embodiment of the present application, a local image to be processed is extracted from an image to be processed, an image residue corresponding to a preset image effect is generated on the basis of said local image, a target local image is generated on the basis of the image residue and said local image, and the target local image is added to said image to obtain a target image. Thus, it is not necessary to input an entire facial image into a generative model for facial editing, and the area of a face outside said local image in the image does not change. While facial details in the image are preserved, the impact of facial image editing on image resolution is effectively reduced, and the effect of facial image editing is improved.

Description

A method, apparatus, device, storage medium, and product for editing facial images.

[0001] This application claims priority to Chinese Patent Application No. 202411577234.2, filed on November 6, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, and in particular to a method, apparatus, device, storage medium, and product for editing facial images. Background Technology

[0003] With the development of internet technology and image processing technology, people are using digital entertainment products more and more widely, and digital entertainment products are offering more and more functions and ways to play.

[0004] For example, video applications often offer facial image editing features, such as transforming faces in a video into different faces (e.g., turning a face into a child's face, changing the face style, etc.) or editing facial features (e.g., changing single eyelids into double eyelids, and vice versa). Traditional facial image editing methods typically input the entire facial image into a generative model, which then outputs the transformed facial image. However, this method often results in facial images that lose detail and significantly impact image clarity, leading to poor editing results. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and product for editing facial images, in order to solve the technical problem in related technologies that facial image editing easily loses facial details, has a significant impact on image clarity, and results in poor facial image editing effects. It can effectively preserve facial details, reduce the impact on image clarity, and improve the facial image editing effect.

[0006] In a first aspect, embodiments of this application provide a face image editing method, comprising:

[0007] Extract the local image to be processed from the image to be processed;

[0008] Generate an image residual corresponding to a preset image effect based on the local image to be processed;

[0009] Generate a target local image based on the image residual and the local image to be processed;

[0010] The target local image is added to the image to be processed to obtain the target image.

[0011] In a second aspect, embodiments of this application provide a face image editing apparatus, including an image extraction module, a residual generation module, a local processing module, and an image generation module, wherein:

[0012] The image extraction module is configured to extract a local image to be processed from the image to be processed.

[0013] The residual generation module is configured to generate an image residual corresponding to a preset image effect based on the local image to be processed;

[0014] The local processing module is configured to generate a target local image based on the image residual and the local image to be processed.

[0015] The image generation module is configured to add the target local image to the image to be processed to obtain the target image.

[0016] In a third aspect, embodiments of this application provide a face image editing device, including: a memory and one or more processors;

[0017] The memory is used to store one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the face image editing method as described in the first aspect.

[0019] In a fourth aspect, embodiments of this application provide a non-volatile storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the face image editing method as described in the first aspect.

[0020] In a fifth aspect, embodiments of this application provide a computer program product comprising a computer program stored in a computer-readable storage medium, wherein at least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the face image editing method as described in the first aspect.

[0021] This application embodiment extracts a local image to be processed from the image to be processed, generates an image residual corresponding to a preset image effect based on the local image to be processed, generates a target local image based on the image residual and the local image to be processed, and adds the target local image to the image to be processed to obtain the target image. It does not require inputting the entire face image into the generative model for face editing, and does not change the area of ​​the face in the image outside the local image to be processed. It can effectively reduce the impact of face image editing on image clarity while preserving the facial details of the image, and improve the face image editing effect. Attached Figure Description

[0022] Figure 1 is a flowchart of a face image editing method provided in an embodiment of this application;

[0023] Figure 2 is a flowchart of another face image editing method provided in an embodiment of this application;

[0024] Figure 3 is a schematic diagram of the training of an image processing model provided in an embodiment of this application;

[0025] Figure 4 is a schematic diagram of loss calculation for an image processing model provided in an embodiment of this application;

[0026] Figure 5 is a structural schematic diagram of a face image editing device provided in an embodiment of this application;

[0027] Figure 6 is a schematic diagram of the structure of a face image editing device provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but additional steps not included in the drawings may also be present. The above processes can correspond to methods, functions, procedures, subroutines, subroutines, etc.

[0029] The facial image editing method provided in this application can be applied to facial editing in images, such as mobile live streaming, video editing, and image editing. Based on this method, it enables functions such as makeup and special effects on faces. These functions can involve adjusting facial features (eyes, ears, mouth, nose, and eyebrows), for example, changing a single eyelid to a double eyelid. The aim is to generate image residuals corresponding to the local images to be processed extracted from the image to be processed, and to obtain the target image based on these residuals. This eliminates the need to input the entire facial image into a generative model for facial editing, thus improving the facial image editing effect.

[0030] In existing face image editing schemes, the entire face image is typically generated by inputting it into a generative model. This method of generating the entire face easily leads to the loss of facial details and image clarity. Furthermore, the large number of pixels in the entire face image results in a computationally intensive process for the generative model, which can cause overheating and frame drops on low-performance mobile devices. Therefore, this application provides a face image editing method to address the technical problems of existing face image editing methods that easily lose facial details, significantly impact image clarity, and produce poor editing results.

[0031] Figure 1 shows a flowchart of a face image editing method provided in an embodiment of this application. The face image editing method provided in this embodiment of the application can be executed by a face image editing device, which can be implemented by hardware and / or software and integrated into a face image editing device.

[0032] The following description uses a face image editing device to perform a face image editing method as an example. Referring to Figure 1, the face image editing method includes:

[0033] S110: Extract the local image to be processed from the image to be processed.

[0034] For example, the image to be processed, which requires facial editing, is acquired. This image can be a live video frame captured in real-time during a live video broadcast, a video frame from a pre-recorded video, or a single image.

[0035] In one embodiment, after obtaining the image to be processed, a local image to be processed is extracted from the image to be processed. The local image to be processed refers to the region in the image to be processed that requires facial editing. This region can be the area corresponding to the facial features in the image to be processed; for example, the region requiring facial editing can be the area corresponding to one or more combinations of the eyes, ears, mouth, nose, and eyebrows in the image to be processed.

[0036] Optionally, the local image to be processed extracted from the image to be processed can be an image corresponding to a combination of regions including eyes, ears, mouth, nose, and eyebrows. That is, a local image to be processed can include multiple facial features, and in this case, facial image editing of multiple facial features can be achieved simultaneously using a single local image to be processed. Alternatively, the local image to be processed extracted from the image to be processed can be an image corresponding to a region of one of the four facial features: eyes, ears, mouth, nose, and eyebrows. That is, a local image to be processed corresponds to one facial feature, and in this case, facial image editing of one facial feature can be achieved using a single local image to be processed.

[0037] S120: Generate image residuals corresponding to preset image effects based on the local image to be processed.

[0038] For example, after obtaining the local image to be processed, an image residual corresponding to a preset image effect is generated based on the local image to be processed. The preset image effect can be determined based on the effect of editing the face image. For example, the preset image effect can be a facial feature editing effect such as changing single eyelids to double eyelids, double eyelids to single eyelids, adding lipstick, changing lipstick color, making ears more pointed, eyebrows thicker, and nose narrower. The preset image effect can be one or more different facial feature editing effects.

[0039] Optionally, a pre-completed image processing model (such as a GAN (Generative Adversarial Network) model) can be used to process the local image to be processed, generating image residuals corresponding to the preset image effect. In one embodiment, the image processing model can adopt a lightweight model structure, which can effectively reduce the computational load of the model and the computational overhead of the device. Furthermore, by using residual learning on the image processing model, it can effectively preserve the local details of facial features while reducing the difficulty of model training and improving the adaptability of face image editing on different terminals.

[0040] S130: Generate the target local image based on the image residual and the local image to be processed.

[0041] For example, after determining the image residual corresponding to the local image to be processed, a target local image can be generated based on the image residual and the local image to be processed. For instance, the target local image can be obtained by summing the image residual and the local image to be processed. The size of the image residual is the same as the size of the local image to be processed, and the size of the target local image is the same as the size of the local image to be processed.

[0042] The target local image can be understood as an image of a facial feature that has undergone corresponding editing processing. For example, if the local image to be processed is a human eye image with a single eyelid, and the preset image effect is to change a single eyelid into a double eyelid, the target local image is a human eye image with a double eyelid.

[0043] In one possible embodiment, the face image editing method provided by this solution generates a target local image based on the image residual and the local image to be processed. Alternatively, the product of the preset effect weight and the image residual can be summed with the local image to be processed, and the summation result can be determined as the target local image.

[0044] For example, after determining the image residual corresponding to the local image to be processed, the preset effect weight is multiplied by the image residual, and then the product of the preset effect weight and the image residual is added to the local image to be processed to obtain the target local image. For example, suppose the preset effect weight is α, the image residual is R, and the local image to be processed is I. inThe local image of the target can be represented as: I out =I in +αR. This solution allows for flexible configuration of the adjustment level of face image editing through preset effect weights, enabling more flexible adjustment of the target image output.

[0045] S140: Add the target local image to the image to be processed to obtain the target image.

[0046] For example, after obtaining the target local image, the target local image is added to the image to be processed to obtain the target image, that is, the target local image replaces or covers the area corresponding to the local image to be processed in the image to be processed to obtain the target image.

[0047] This solution obtains the target image for face editing by replacing the corresponding region of the local image to be processed in the image to be processed with the target local image that has been edited. It does not require inputting the entire face image into the generative model for image processing, which effectively reduces the amount of data processing on the device. Moreover, it only requires modification of the region corresponding to the preset facial features, without modifying the regions outside the preset facial features, which can effectively preserve the facial details in the regions outside the preset facial features, resulting in a more natural face image editing effect.

[0048] The above describes a method that extracts a local image from the image to be processed, generates an image residual corresponding to a preset image effect based on the local image to be processed, generates a target local image based on the image residual and the local image to be processed, and adds the target local image to the image to be processed to obtain the target image. This method does not require inputting the entire face image into the generative model for face editing, and does not change the area of ​​the face in the image outside the local image to be processed. It can effectively reduce the impact of face image editing on image clarity while preserving the details of the face in the image, thereby improving the face image editing effect.

[0049] Based on the above embodiments, Figure 2 shows a flowchart of another face image editing method provided by an embodiment of this application. This face image editing method is a specific embodiment of the above-described face image editing method. Referring to Figure 2, the face image editing method includes:

[0050] S210: Determine the facial key points in the image to be processed, and extract the local image to be processed from the image to be processed based on the facial key points.

[0051] For example, an image to be processed is acquired, and facial landmark detection is performed on the image to identify multiple facial landmarks. Facial landmarks are important feature points of various parts of the face in the image, typically contour points and corner points, including eyebrows, eyes, nose, and mouth. Facial landmarks can be represented using 68, 106, or 240 points. For example, to balance the accuracy and time efficiency of landmark detection, 106 points can be used to label facial landmarks. Optionally, a pre-trained face detection model can be used to detect facial landmarks in the image to be processed.

[0052] In one embodiment, after determining multiple facial key points in the image to be processed, a local image to be processed is extracted from the image based on the facial features to be edited and the aforementioned multiple facial key points. For example, multiple target facial key points corresponding to the facial features to be edited are determined, and a bounding box corresponding to the facial features to be edited is determined based on these target facial key points (e.g., the maximum and minimum coordinate points on the horizontal and vertical axes of the target facial key points). (Alternatively, the bounding box can be scaled up). The pixel range of the image to be processed corresponding to the bounding box is then extracted to obtain the local image to be processed. This solution accurately extracts the local image to be processed based on the facial key points in the image to be processed, accurately performs facial editing on the local image to be processed, effectively reduces the impact of facial image editing on image clarity while preserving facial details, and ensures the effect of facial image editing.

[0053] In one embodiment, when the facial features to be edited include both left and right parts (e.g., eyes include both eyes, ears include both ears), the bounding box can simultaneously enclose both left and right parts, or it can determine two local images to be processed for the facial key points corresponding to the left and right parts respectively.

[0054] Optionally, when determining two local images to be processed for the facial key points corresponding to the left and right parts respectively, image residuals can be generated for the two local images to be processed respectively, and target local images at the corresponding positions can be generated according to the corresponding image residuals. The target local images can then be used to replace the corresponding regions in the images to be processed to obtain the target image.

[0055] Optionally, when determining two local images to be processed for the facial key points corresponding to the left and right parts respectively, the two local images to be processed can be combined into one local image to be processed. Then, the image residual corresponding to the local image to be processed is determined. Based on the image residual, a target local image containing the editing effect corresponding to the two parts is generated from the synthesized local image to be processed. Then, the target local image is split to obtain the target local images corresponding to the two parts. The corresponding areas in the image to be processed are replaced by the two target local images to obtain the target image.

[0056] In one possible embodiment, the face image editing method provided by this solution extracts a local image to be processed from an image to be processed based on facial key points, including:

[0057] S211: Extract the first local image and the second local image from the image to be processed based on facial key points.

[0058] S212: Combine the first local image and the second local image into a local image to be processed.

[0059] For example, when the facial features to be edited include both left and right parts, a first local image and a second local image can be extracted from the image to be processed based on the facial key points corresponding to the left and right parts. For instance, taking the eyes as an example, the first local image is extracted from the image to be processed based on the facial key points corresponding to the left eye, and the second local image is extracted from the image to be processed based on the facial key points corresponding to the right eye.

[0060] In one embodiment, after obtaining the first local image and the second local image, the first local image and the second local image can be combined to obtain the local image to be processed. Optionally, the first local image and the second local image can be stitched vertically to obtain the local image to be processed, or the first local image and the second local image can be stitched horizontally to obtain the local image to be processed.

[0061] This solution extracts a first and a second local image from the image to be processed based on facial key points. This maximizes the preservation of the facial features that need editing, reducing modifications to areas outside the facial features and preserving greater facial detail and clarity. Simultaneously, the first and second local images are combined into a single local image for unified processing. This reduces the additional computational overhead of separate processing of the first and second local images by the image processing model, significantly reducing the input size of the model and consequently lowering the device's computational cost. This improves facial image editing efficiency, and because image processing is applied only to the facial features, it minimizes the loss of detail in other facial areas, further preserving facial detail and clarity.

[0062] In one possible embodiment, the face image editing method provided by this solution combines a first local image and a second local image into a local image to be processed, including:

[0063] S2121: Perform affine transformation alignment on the first local image and the second local image.

[0064] S2122: Combine the first local image and the second local image after affine transformation alignment into a local image to be processed.

[0065] For example, after obtaining the first and second local images, affine transformation alignment processing can be performed on them. This includes rotating, translating, and cropping the first and / or second local images to align them in height and / or width. The affine-transformed and aligned first and second local images are then combined into the local image to be processed. For instance, taking the scenario of transforming a single eyelid into a double eyelid as an example, the center of the eye in the first and second local images, as well as the size to be cropped, can be calculated based on the key points of the eye. The two eye images are then aligned using affine transformation, and finally, the two eye images are stitched together vertically to obtain the local image to be processed. This solution effectively fixes the image distribution of the input image processing model by performing affine transformation alignment processing on the first and second local images, improving the processing efficiency and accuracy of the image processing model, and effectively reducing the network learning difficulty of the image processing model, thus improving the training efficiency and accuracy of the image processing model.

[0066] S220: Input the local image to be processed into the trained image processing model. The image processing model generates an image residual corresponding to the preset image effect based on the local image to be processed. The objective function of the image processing model is determined based on the generative adversarial network loss function, the absolute value error loss function, and the perceptual loss function.

[0067] For example, this scheme generates the image residual corresponding to the local image to be processed through a trained image processing model. After obtaining the local image to be processed, the local image to be processed is input into the image processing model, which analyzes and processes the local image to obtain the image residual corresponding to the preset image effect.

[0068] In this scheme, the objective function for training the image processing model is determined based on the Generative Adversarial Network (GAN) Loss, the L1 Loss, and the Perceptual Loss. For example, the objective function can be the sum of the GAN Loss, L1 Loss, and Perceptual Loss, or it can be a weighted sum of these three functions.

[0069] This scheme trains the image processing model using an objective function determined by the generative adversarial network loss function, the absolute value error loss function, and the perceptual loss function. The adversarial network loss function and the absolute value error loss function enable the output of the image processing model to be closer to the control sample. Furthermore, the perceptual loss function, based on the calculation of low-level feature loss (pixel color, edges, etc.), calculates the loss by comparing the convolutional output of the original image and the convolutional output of the generated image. This leverages the ability of convolutional layers to abstract high-level features, perceiving the image from a global, high-dimensional perspective. Using perceptual loss can preserve the overall information of the image, effectively ensuring the realism of the generated image, avoiding the generation of unrealistic facial feature images, and improving the quality of face image editing.

[0070] Figure 3 illustrates the training of an image processing model. Taking the transformation of a single eyelid into a double eyelid as an example, the image generation aims to generate a target facial feature image (e.g., a double eyelid image) from an input source facial feature image (e.g., a single eyelid image) through inference by the image processing model. The image processing model uses a Conditional Generative Adversarial Network (cGAN) as a baseline, providing [original image x, baseline image y (GroundTruth, GT)] as training data pairs to train the generator G and discriminator D in the model. Optionally, the adversarial network loss function can be defined as: L cGAN (G,D)=E x,y [logD(x,y)]+E x,z [log(1-D(x,G(x,z)))]

[0071] Where z represents the random noise added to the input image, D(·)∈[0,1] is the discriminator output, representing the probability that the image is true, G(x) is the generated prediction image, and G(x,z) is the prediction image generated when (random) noise z is added. E x,y[logD(x,y)] corresponds to the right side of Figure 3, indicating that when the discriminator is input with a data pair of [x,y], the discriminator should output 1, that is, make D(x,y) as large as possible. E x,z [log(1-D(x,G(x,z)))] corresponds to the left side of Figure 3, indicating that when the discriminator is input with the data pair [x,G(x,z)], the discriminator should output 0, that is, make D(x,G(x,z)) as small as possible. For training the generator G, we want the generated image to be closer to reality, that is, to make D(x,G(x,z)) as large as possible, then L... cGAN The larger (G,D) becomes, the more we need to find the minimum G. For training the discriminator D, the stronger D's discriminative power, the larger D(x,y) becomes, and the smaller D(x,G(x,z)) becomes, then L... cGAN The smaller (G,D) becomes, the more we need to find the maximum value of max D. Based on this, the loss function of the adversarial network can be expressed as:

[0072] In one embodiment, to make the output image G(x,z) of the image processing model closer to the ground truth image y, the L1 distance, i.e., the absolute value error loss function, can be calculated pixel by pixel: L pixel (G)=E x,y,z [||yG(x,z)||1]

[0073] Meanwhile, perceptual loss leverages the ability of convolutional layers to abstract high-level features, perceiving the image from a global, high-dimensional perspective. Using perceptual loss preserves the overall information of the image, effectively ensuring the realism of the generated image and avoiding the generation of unrealistic facial feature images. Optionally, a VGG-19 network can be used to calculate the perceptual loss, which can be expressed as:

[0074] In this scheme, the L1 distance of multi-layer convolution is used as the perceptual loss, F i (·) represents the feature map of the i-th convolutional layer, λ i Let be the loss weights for the i-th layer convolutional feature map. Based on this, the objective function for training the image processing model in this scheme can be expressed as:

[0075] Where, λ pixel λperceptual is the first loss weight of the absolute value error loss function, and λperceptual is the second loss weight of the perceptual loss.

[0076] In one possible embodiment, the training process of the image processing model provided by this solution may include: inputting sample images into the image processing model, generating sample residuals corresponding to preset image effects through the image processing model based on the sample images; obtaining generated images based on the sample residuals and sample images, and optimizing the parameters of the image processing model based on the objective function and the label images corresponding to the generated images and sample images.

[0077] For example, when training an image processing model, sample images are input into the model, which then generates sample residuals corresponding to a preset image effect based on the sample images. After obtaining the sample residuals, a generated image is obtained based on the sample residuals and the sample images, for example, by adding the sample residuals and the sample images together.

[0078] In one embodiment, after obtaining the generated image, the model loss is calculated based on the objective function and the label images corresponding to the generated image and the sample image, and the parameters of the image processing model are optimized based on the model loss.

[0079] Figure 4 shows a schematic diagram of loss calculation for an image processing model. The image processing model processes the sample image (input image x) through a generator, outputs the sample residual (Net_{out}) from the generator, and adds the sample residual to the sample image to obtain the generated image (model output G(x, z)). The loss can be calculated based on the objective function, the generated image, and the label image, and the parameters of the image processing model can be optimized based on the loss calculation results.

[0080] This approach improves image generation quality by learning the differences (i.e., residuals) between the input and target images. Since directly generating the target image is difficult, especially when the image details are complex, this approach effectively reduces the learning difficulty of the network by learning the residuals. The image processing model only needs to learn how to extract missing details and features from the target image from the input image, rather than generating the entire image from scratch. Simultaneously, residual learning helps the image processing model better focus on detail differences, making the image obtained based on the output residuals closer to the target image in detail, effectively improving the overall quality of image generation. Because the image processing model learns the variations or details of the image, rather than the entire image, it can better adapt to changes in the input image, providing better generalization ability. The generator in the image processing model learns the image residuals, keeping the overall network structure of the image processing model unchanged. The input image is added to the result of the forward propagation as the network output, and the loss function is calculated, i.e., from the original G(x,z) = Net... out It becomes G(x,z)=x+Net out It still represents the output of the model and can be used to calculate the loss with the labeled image to optimize the parameters of the image processing model.

[0081] In one embodiment, to meet the real-time face editing function requirements of mobile devices, it is necessary to consider the computational overhead of the model and avoid consuming too much computing resources of the mobile device. This solution will introduce a depthwise separable convolution module based on the U-Net network structure commonly used in generator G in generative adversarial networks, while reducing the number of convolution channels and convolution depth, effectively reducing the number of floating-point operations (FLOPs) of model computation, reducing the computational overhead of the model before image processing, and ensuring the smooth operation of the image processing model on mobile devices.

[0082] S230: Generate the target local image based on the image residual and the local image to be processed.

[0083] S240: Add the target local image to the image to be processed to obtain the target image.

[0084] In one embodiment, the face image editing method provided by this solution adds a target local image to an image to be processed to obtain a target image, including:

[0085] S241: When the local image to be processed is obtained based on the first local image and the second local image, the first target local image and the second target local image are obtained based on the target local image.

[0086] S242: Add the first target local image and the second target local image to the image to be processed to obtain the target image.

[0087] For example, when the local image to be processed is obtained based on the first local image and the second local image, after obtaining the target local image, the target local image can be split into a first target local image and a second target local image according to the stitching method of the first local image and the second local image, wherein the size of the first target local image and the second target local image is the same as the size of the first local image and the second local image.

[0088] For example, when stitching the first local image and the second local image vertically to obtain the local image to be processed, the target local image is split vertically into the first target local image and the second target local image; when stitching the first local image and the second local image horizontally to obtain the local image to be processed, the target local image is split horizontally into the first target local image and the second target local image.

[0089] In one embodiment, after obtaining the first target local image and the second target local image, the first and second target local images are added to the image to be processed according to their positions on the image to be processed, respectively, to obtain the target image. At this time, the image in the target image corresponding to the first and second local images is replaced with the image corresponding to the first and second target local images. This solution, by splitting the target local image into the first and second target local images and adding them to the image to be processed when the local image to be processed is obtained based on the first and second local images, can retain the image of the facial features that need to be edited to a greater extent, reduce the modification of areas other than the facial features that need to be edited, and retain the details and clarity of the face to a greater extent.

[0090] In one possible embodiment, the face image editing method provided by this solution adds a first target local image and a second target local image to an image to be processed to obtain a target image. This can be achieved by: performing an inverse affine transformation alignment process on the first target local image and the second target local image; and adding the first target local image and the second target local image after the inverse affine transformation alignment process to the image to be processed to obtain the target image.

[0091] In this scheme, the local image to be processed is synthesized from a first local image and a second local image after affine transformation alignment. Correspondingly, after splitting the target local image into a first target local image and a second target local image, this scheme performs an inverse affine transformation alignment on the first and second target local images based on the affine transformation alignment method applied to them. This transforms the specifications of the first and second target local images to be consistent with those of the first and second local images before affine transformation alignment. After completing the inverse affine transformation alignment of the first and second target local images, the inverse affine transformation aligned first and second target local images can be added to the image to be processed to obtain the target image. At this point, the first and second target local images can correctly replace the corresponding regions of the first and second local images in the image to be processed, resulting in a target image where facial features have been edited while preserving facial details.

[0092] The above describes a method that extracts a local image from the image to be processed, generates an image residual corresponding to a preset image effect based on the local image, generates a target local image based on the image residual and the local image to be processed, and adds the target local image to the image to be processed to obtain the target image. This method eliminates the need to input the entire face image into the generative model for face editing, and does not alter areas of the face outside the local image to be processed. It effectively reduces the impact of face image editing on image clarity while preserving facial details, thus improving the face image editing effect. Furthermore, by splitting the target local image into a first target local image and a second target local image and adding them to the image to be processed when the local image to be processed is obtained based on the first and second local images, the method can retain the images of the facial features that need editing to the greatest extent, reducing modifications to areas outside the facial features that need editing, and preserving more facial details and clarity. Furthermore, by splitting the target local image into a first target local image and a second target local image and adding them to the image to be processed when the local image to be processed is obtained based on the first local image and the second local image, the target image can be obtained by maximizing the preservation of the facial features that need to be edited, reducing modifications to areas other than the facial features that need to be edited, and preserving more facial details and clarity. This solution only modifies the high-frequency changes and edge details of the facial features in the image to be processed. Compared with existing face editing solutions that generate the entire face image, this solution can better preserve facial feature details, while also completely preserving information about other facial parts besides the facial features.

[0093] Figure 5 is a schematic diagram of a face image editing device provided in an embodiment of this application. Referring to Figure 5, the face image editing device includes an image extraction module 51, a residual generation module 52, a local processing module 53, and an image generation module 54.

[0094] The image extraction module 51 is configured to extract a local image to be processed from the image to be processed; the residual generation module 52 is configured to generate an image residual corresponding to a preset image effect based on the local image to be processed; the local processing module 53 is configured to generate a target local image based on the image residual and the local image to be processed; and the image generation module 54 is configured to add the target local image to the image to be processed to obtain the target image.

[0095] The above describes a method that extracts a local image from the image to be processed, generates an image residual corresponding to a preset image effect based on the local image to be processed, generates a target local image based on the image residual and the local image to be processed, and adds the target local image to the image to be processed to obtain the target image. This method does not require inputting the entire face image into the generative model for face editing, and does not change the area of ​​the face in the image outside the local image to be processed. It can effectively reduce the impact of face image editing on image clarity while preserving the details of the face in the image, thereby improving the face image editing effect.

[0096] In one possible embodiment, the image extraction module 51 extracts a local image to be processed from the image to be processed, configured to: determine facial key points in the image to be processed, and extract the local image to be processed from the image to be processed based on the facial key points.

[0097] In one possible embodiment, the image extraction module 51 extracts a local image to be processed from the image to be processed based on facial key points, configured as follows:

[0098] Extract the first local image and the second local image from the image to be processed based on facial key points;

[0099] The first local image and the second local image are combined into a local image to be processed.

[0100] In one possible embodiment, the image extraction module 51 combines the first local image and the second local image into a local image to be processed, configured as follows:

[0101] Perform affine transformation alignment on the first and second local images;

[0102] The first and second local images, after affine transformation alignment, are combined into the local image to be processed.

[0103] In one possible embodiment, the image generation module 54 adds the target local image to the image to be processed to obtain the target image, configured as follows:

[0104] When the local image to be processed is obtained based on the first local image and the second local image, the first target local image and the second target local image are obtained based on the target local image;

[0105] The target image is obtained by adding the first target local image and the second target local image to the image to be processed.

[0106] In one possible embodiment, the image generation module 54 adds the first target local image and the second target local image to the image to be processed to obtain the target image, configured as follows:

[0107] Perform inverse affine transformation alignment on the local images of the first and second targets;

[0108] The target image is obtained by adding the first and second target local images, which have been aligned by inverse affine transformation, to the image to be processed.

[0109] In one possible embodiment, the residual generation module 52 generates an image residual corresponding to a preset image effect based on the local image to be processed, configured as follows:

[0110] The local image to be processed is input into the trained image processing model. The image processing model generates an image residual corresponding to the preset image effect based on the local image to be processed. The objective function of the image processing model is determined based on the generative adversarial network loss function, the absolute value error loss function, and the perceptual loss function.

[0111] In one possible embodiment, the face image editing apparatus further includes a model training module configured to train an image processing model, wherein the training process of the image processing model is configured as follows:

[0112] The sample image is input into the image processing model, which then generates the sample residual corresponding to the preset image effect based on the sample image.

[0113] The generated image is obtained based on the sample residual and the sample image. The parameters of the image processing model are optimized based on the objective function and the label images corresponding to the generated image and the sample image.

[0114] In one possible embodiment, the local processing module 53 generates a target local image based on the image residual and the local image to be processed, configured to: sum the product of the preset effect weight and the image residual with the local image to be processed, and determine the summation result as the target local image.

[0115] It is worth noting that in the embodiments of the above-mentioned face image editing device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this application.

[0116] This application also provides a face image editing device, which can integrate the face image editing apparatus provided in this application. Figure 6 is a schematic diagram of the structure of a face image editing device provided in this application. Referring to Figure 6, the face image editing device is configured as follows: an input device 63, an output device 64, a memory 62, and one or more processors 61; the memory 62 is used to store one or more programs; when one or more programs are executed by one or more processors 61, the one or more processors 61 implement the face image editing method provided in the above embodiments. The face image editing apparatus, device, and computer provided above can be used to execute the face image editing method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0117] This application also provides a non-volatile storage medium storing computer-executable instructions, which, when executed by a computer processor, are used to perform the face image editing method provided in the above embodiments. Of course, the computer-executable instructions provided in this application are not limited to the face image editing method provided above; they can also perform related operations in the face image editing method provided in any embodiment of this application. The face image editing apparatus, device, and storage medium provided in the above embodiments can execute the face image editing method provided in any embodiment of this application. Technical details not described in detail in the above embodiments can be found in the face image editing method provided in any embodiment of this application.

[0118] Based on the above embodiments, this application also provides a computer program product. The technical solution of this application, in essence or in other words, the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes several instructions to cause a computer device, mobile terminal, or processor therein to execute all or part of the steps of the face image editing method provided in the various embodiments of this application.

Claims

1. A method for editing facial images, wherein, include: Extract the local image to be processed from the image to be processed; Generate an image residual corresponding to a preset image effect based on the local image to be processed; Generate a target local image based on the image residual and the local image to be processed; The target local image is added to the image to be processed to obtain the target image.

2. The face image editing method according to claim 1, wherein, The step of extracting the local image to be processed from the image to be processed includes: Identify facial key points in the image to be processed, and extract the local image to be processed from the image to be processed based on the facial key points.

3. The face image editing method according to claim 2, wherein, The step of extracting the local image to be processed from the image to be processed based on the facial key points includes: Based on the facial key points, extract a first local image and a second local image from the image to be processed; The first local image and the second local image are combined into a local image to be processed.

4. The face image editing method according to claim 3, wherein, The step of combining the first local image and the second local image into a local image to be processed includes: Affine transformation alignment processing is performed on the first local image and the second local image; The first local image and the second local image, after affine transformation alignment, are combined into a local image to be processed.

5. The face image editing method according to claim 1, wherein, The step of adding the target local image to the image to be processed to obtain the target image includes: When the local image to be processed is obtained based on the first local image and the second local image, the first target local image and the second target local image are obtained based on the target local image; The first target local image and the second target local image are added to the image to be processed to obtain the target image.

6. The face image editing method according to claim 5, wherein, The step of adding the first target local image and the second target local image to the image to be processed to obtain the target image includes: The first target local image and the second target local image are aligned by inverse affine transformation. The first target local image and the second target local image, after being aligned by inverse affine transformation, are added to the image to be processed to obtain the target image.

7. The face image editing method according to claim 1, wherein, The step of generating an image residual corresponding to a preset image effect based on the local image to be processed includes: The local image to be processed is input into the trained image processing model. The image processing model generates an image residual corresponding to a preset image effect based on the local image to be processed. The objective function of the image processing model is determined based on the generative adversarial network loss function, the absolute value error loss function, and the perceptual loss function.

8. The face image editing method according to claim 7, wherein, The training process of the image processing model includes: The sample image is input into the image processing model, and the image processing model generates a sample residual corresponding to a preset image effect based on the sample image. The generated image is obtained based on the sample residual and the sample image. The parameters of the image processing model are optimized based on the objective function and the label images corresponding to the generated image and the sample image.

9. The face image editing method according to claim 1, wherein, The step of generating a target local image based on the image residual and the local image to be processed includes: The product of the preset effect weight and the image residual is summed with the local image to be processed, and the summation result is determined as the target local image.

10. A face image editing device, wherein, It includes an image extraction module, a residual generation module, a local processing module, and an image generation module, wherein: The image extraction module is configured to extract a local image to be processed from the image to be processed. The residual generation module is configured to generate an image residual corresponding to a preset image effect based on the local image to be processed; The local processing module is configured to generate a target local image based on the image residual and the local image to be processed. The image generation module is configured to add the target local image to the image to be processed to obtain the target image.

11. A face image editing device, wherein, include: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the face image editing method as described in any one of claims 1-9.

12. A non-volatile storage medium for storing computer-executable instructions, wherein, The computer-executable instructions, when executed by a computer processor, are used to perform the face image editing method as described in any one of claims 1-9.

13. A computer program product comprising a computer program, wherein, When the computer program is executed by the processor, it implements the face image editing method according to any one of claims 1-9.

Images