Image processing method and apparatus, electronic device, storage medium, and computer program product

Abstract

The present application relates to the technical field of images, and discloses an image processing method and apparatus, an electronic device, a storage medium, and a computer program product. The method comprises: inputting a first image and a second image into a first matting model, and acquiring a first feature map on the basis of the first image and the second image by means of the first matting model, the second image being an image obtained on the basis of an image element in the first image; processing the first feature map by means of the first matting model to obtain at least one of a detail feature map and an edge feature map; and on the basis of at least one of the detail feature map and the edge feature map, and the first feature map, outputting a first matted image by means of the first matting model, wherein the first matting model is obtained by training on the basis of detail features and edge features of images.

Description

Image processing methods, apparatuses, electronic devices, storage media, and computer program products

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202510071618.5, filed in China on January 16, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application belongs to the field of image technology, specifically relating to an image processing method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0004] Typically, after a user takes a picture with their phone, the user can trigger the phone to process the image.

[0005] For example, if a user finds that the position of the subject in the captured image does not meet their needs, the user can use the background removal function of the image processing software on their mobile phone to remove the subject. The user can then move the removed image to the desired position in the captured image, so that the processed image meets the user's needs.

[0006] However, the image cutout function in the image processing software has limited accuracy, which may result in the cutout image not meeting the user's needs. This leads to poor image quality in the cutout image acquired by the electronic device. Summary of the Invention

[0007] The purpose of this application is to provide an image processing method, apparatus, electronic device, storage medium, and computer program product that can improve the image quality of cutout images acquired by electronic devices.

[0008] In a first aspect, embodiments of this application provide an image processing method, which includes: inputting a first image and a second image into a first matting model; obtaining a first feature map based on the first image and the second image using the first matting model, wherein the second image is an image obtained based on image elements in the first image; processing the first feature map using the first matting model to obtain at least one of a detail feature map and an edge feature map; and outputting a first matted image using the first matting model based on at least one of the detail feature map and the edge feature map, and the first feature map; wherein the first matting model is trained based on the detail features and edge features of the image.

[0009] Secondly, embodiments of this application provide an image processing apparatus, comprising: an acquisition module, a processing module, and an output module; the acquisition module is configured to input a first image and a second image into a first matting model, and acquire a first feature map based on the first image and the second image through the first matting model, wherein the second image is an image obtained based on image elements in the first image; the processing module is configured to process the first feature map through the first matting model to obtain at least one of a detail feature map and an edge feature map; the output module is configured to output a first matted image through the first matting model, based on at least one of the detail feature map and the edge feature map, and the first feature map; wherein the first matting model is trained based on the detail features and edge features of the image.

[0010] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0011] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0012] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0013] In a sixth aspect, embodiments of this application provide a computer program / program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0014] In this embodiment, the electronic device can input a first image and a second image into a first matting model to obtain a first feature map based on the first image and the second image, wherein the second image is an image obtained based on image elements in the first image; then, the electronic device can process the first feature map through the first matting model to obtain at least one of a detail feature map and an edge feature map; and output a first matted image based on at least one of the detail feature map and the edge feature map, and the first feature map; wherein, the first matting model is trained based on the detail features and edge features of the image. In this scheme, since the second image is obtained based on the image elements that need to be cut out from the first image, the second image includes the image elements that need to be cut out. Therefore, the electronic device can process the second image and the original first image to further obtain at least one of the detail feature map and the edge feature map. In other words, the first cut-out image output by the first cut-out model can accurately retain at least one of the detail image content and edge image content of the image elements that the user needs to cut out from the first image, based on the second image. This allows the first cut-out image to meet the user's requirements for the fineness of the cut-out image, thereby improving the image quality of the cut-out image obtained by the electronic device. Attached Figure Description

[0015] Figure 1 is a flowchart of one of the image processing methods provided in the embodiments of this application;

[0016] Figure 2 is a schematic diagram of an example of a cutout image provided in an embodiment of this application;

[0017] Figure 3 is one of the structural diagrams of a first image matting model provided in an embodiment of this application;

[0018] Figure 4A is a second structural diagram of a first image matting model provided in an embodiment of this application;

[0019] Figure 4B is a third structural diagram of a first image matting model provided in an embodiment of this application;

[0020] Figure 5A is one of the schematic diagrams of a masking image provided in an embodiment of this application;

[0021] Figure 5B is one of the schematic diagrams of a masking image provided in an embodiment of this application;

[0022] Figure 6 is a fourth structural diagram of a first image matting model provided in an embodiment of this application;

[0023] Figure 7 is the fifth structural diagram of a first image matting model provided in an embodiment of this application;

[0024] Figure 8 is a second flowchart of an image processing method provided in an embodiment of this application;

[0025] Figure 9A is one of the schematic diagrams of an example of displaying a cutout image on a mobile phone according to an embodiment of this application;

[0026] Figure 9B is a second example of a mobile phone displaying a cutout image provided in an embodiment of this application;

[0027] Figure 9C is a third example of a mobile phone displaying a cutout image according to an embodiment of this application;

[0028] Figure 10A is a fourth example of a mobile phone displaying a cutout image provided in an embodiment of this application;

[0029] Figure 10B is the fifth example of a mobile phone displaying a cutout image provided in the embodiments of this application;

[0030] Figure 11 is a third flowchart of an image processing method provided in an embodiment of this application;

[0031] Figure 12 is a sixth example of a mobile phone displaying a cutout image provided in the embodiments of this application;

[0032] Figure 13A is the seventh example of a mobile phone displaying a cutout image according to an embodiment of this application;

[0033] Figure 13B is an eighth example of a mobile phone displaying a cutout image provided in an embodiment of this application;

[0034] Figure 14 is a flowchart of a matting model training method provided in an embodiment of this application;

[0035] Figure 15 is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0036] Figure 16 is one of the hardware structure diagrams of an electronic device provided in an embodiment of this application;

[0037] Figure 17 is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0039] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0040] The terms "at least one," "at least one," etc., used in this application's specification refer to any one, any two, or a combination of two or more of the included objects. For example, "at least one of a, b, and c" can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more, and its meaning is similar to that of "at least one."

[0041] The following is a definition of the technical terms used in the embodiments of this application:

[0042] Secondary composition: The process of moving the position of the subject in an image stored in an electronic device to obtain a new composition with the subject in a different position.

[0043] The image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0044] The image processing method provided in this application can be applied to scenarios where secondary image composition is required.

[0045] The image processing method provided in this application embodiment will be illustrated below with examples of specific scenarios.

[0046] Scenario 1: Recomposing the image of the person in it.

[0047] If a user wants to recompose an image, such as adjusting the position of person 1 in image 1 relative to the background building, the user can trigger an electronic device to input image 1 and the corresponding cutout image into the first cutout model, so that the first cutout model outputs a first cutout image with detailed features and edge features corresponding to person 1 based on the input image.

[0048] Scenario 2: Recomposing the image of objects.

[0049] If a user wants to perform secondary composition on an image, such as adjusting the display position and size of an object in image 2, the user can trigger an electronic device to input image 2 and the corresponding cutout image into the first cutout model, so that the first cutout model outputs a first cutout image with detailed features and edge features corresponding to object 2 based on the input image.

[0050] It should be noted that the above scenarios 1 and 2 are merely exemplary examples of some scenarios that may be applied to the embodiments of this application. In actual implementation, the embodiments of this application can also be applied to any possible scenarios that require adjustment of parameters such as the display size and display position of image elements contained in the image. The embodiments of this application are not limited here.

[0051] Based on the above-described scenario applied in the embodiments of this application, the image processing method provided in the embodiments of this application allows an electronic device to input a first image and a second image into a first matting model. The first matting model, based on the first image and the second image, obtains a first feature map, where the second image is an image obtained based on image elements in the first image. Then, the electronic device can process the first feature map using the first matting model to obtain at least one of a detail feature map and an edge feature map. Finally, the first matting model, based on at least one of the detail feature map and the edge feature map, and the first feature map, outputs a first matted image. The first matting model is trained based on the detail features and edge features of the image. In this scheme, since the second image is obtained based on the image elements that need to be cut out from the first image, the second image includes the image elements that need to be cut out. Therefore, the electronic device can process the second image and the original first image to further obtain at least one of the detail feature map and the edge feature map. In other words, the first cut-out image output by the first cut-out model can accurately retain at least one of the detail image content and edge image content of the image elements that the user needs to cut out from the first image, based on the second image. This allows the first cut-out image to meet the user's requirements for the fineness of the cut-out image, thereby improving the image quality of the cut-out image obtained by the electronic device.

[0052] The image processing method provided in this application is executed by an image processing device, which can be an electronic device, or a functional module or entity within an electronic device. This application does not limit the specific implementation of this method. The following will use an electronic device as an example to illustrate the image processing method provided in this application.

[0053] This application provides an image processing method. Figure 1 shows a flowchart of an image processing method provided by this application. As shown in Figure 1, the image processing method provided by this application may include the following steps 201 to 203.

[0054] Step 201: The electronic device inputs the first image and the second image into the first matting model, and obtains the first feature map based on the first image and the second image through the first matting model.

[0055] In this embodiment of the application, the second image can be an image obtained based on image elements in the first image.

[0056] In some embodiments of this application, the image elements included in the second image may be determined based on the user's requirements for cutout of the first image.

[0057] In some embodiments of this application, the second image described above may have the same image size as the first image.

[0058] In some embodiments of this application, the second image described above may be a mask image corresponding to the first image.

[0059] It should be noted that the above mask can be a simple black and white mask, or a complex mask based on the layer content contained in the first image.

[0060] In some embodiments of this application, the mask described above can be used to define the display area for image elements that need to be cut out from the first image.

[0061] For example, suppose the first image contains a head image of a male user wearing glasses, and the user wants to obtain this head image through an electronic device. The user can first trigger the electronic device to perform image cutout processing based on the head image, so that the electronic device outputs a black and white mask image corresponding to the head image as shown in Figure 2, i.e., the aforementioned second image. In the black and white mask image, the white area is the image area the user wants to cut out, i.e., the head image of the male user wearing glasses; the black area is the image area the user does not need to cut out, i.e., the background area in the first image.

[0062] In this embodiment, the first image matting model is trained based on the detail features and edge features of the image.

[0063] In some embodiments of this application, when an electronic device inputs a first image and a second image into a first matting model, the first matting model can first merge the first image and the second image into a single image by superimposing the image channels, and then perform processing on the merged image to obtain a first feature map.

[0064] In some embodiments of this application, the first matting model described above may include an input convolutional layer and N convolutional modules.

[0065] In some embodiments of this application, when the electronic device inputs a first image and a second image into a first matting model, the electronic device may first perform downsampling processing on the image merged from the first image and the second image by inputting a convolutional layer to reduce the resolution of the image.

[0066] For example, assuming the resolution of the first and second images is 1024*1024, the resolution of the output feature map can be reduced to 512*512 after downsampling through the input convolutional layer.

[0067] In some embodiments of this application, the first image and the second image can undergo matrix calculation based on Feature = w0x + b0 in the input convolutional layer to extract features from the input image, thereby outputting a feature map corresponding to the input convolutional layer. Here, x can be image data obtained from the first image and the second image, w0 can be the weight value corresponding to the input convolutional layer, and b0 can be the bias value corresponding to the input convolutional layer.

[0068] In this way, electronic devices can reduce the computational load of image processing and the number of parameters used in model calculations by downsampling the input image to adjust the size of the input image, thereby increasing the robustness and generalization ability of the model.

[0069] In some embodiments of this application, the electronic device can input the feature map output by the input convolutional layer into the convolutional module in the first matting model to perform subsequent image processing on the image.

[0070] It should be noted that the number of convolutional modules included in the first matting model and the parameters of each convolutional model can be determined according to actual needs, and this application does not impose any restrictions here.

[0071] In some embodiments of this application, the step 201 above, "obtaining a first feature map based on the first image and the second image", can be specifically implemented through the following steps 201a to 201c.

[0072] Step 201a: The electronic device performs downsampling processing on the first image and the second image through the first convolution module in the first matting model, and outputs the first feature vector.

[0073] Step 201b: The electronic device performs downsampling on the (i-1)th feature vector through the i-th convolutional module in the first matting model and outputs the i-th feature vector.

[0074] Where i∈[2,N], and i is an integer, and N is the number of convolutional modules.

[0075] Step 201c: Determine the Nth feature vector as the first feature map.

[0076] In some embodiments of this application, the electronic device can use N convolutional modules to perform N downsampling processes on the feature map output by the above-mentioned input convolutional layer in order to extract the semantic information of the first image and the second image, namely the above-mentioned feature vector.

[0077] It should be noted that the semantic information of the above images may include, but is not limited to, basic information such as the outlines and categories of the image elements contained in the image; it may also include information such as the attributes, positional relationships, and context of the image elements.

[0078] The attributes of an image element may include, but are not limited to, at least one of the following: color, shape, and size.

[0079] In some embodiments of this application, during the downsampling process of the input image data by the convolution module, the electronic device can increase the number of convolution kernels and the number of output channels.

[0080] In this way, since each convolutional kernel can extract a specific image feature and output a corresponding feature map, more types of features can be extracted from the image, enhancing the richness and diversity of feature representation, so that the model can better understand and recognize different elements and patterns in the image, thereby further avoiding the loss of image feature information.

[0081] For example, as shown in Figure 3, if N=4, the electronic device can first downsample the first and second images in the first matting model through the input convolutional layer to obtain a feature map with a resolution of 512*512, which is obtained by combining images from 32 channels. Next, the electronic device can downsample the 512*512 feature map output by the input convolutional layer through the first convolutional module, i.e., convolutional module 1, to obtain a feature map with a resolution of 256*256, which is obtained by combining images from 64 channels, i.e., the first feature vector. Then, the electronic device can downsample the 256*256 feature map output by convolutional module 1 through the second convolutional module, i.e., convolutional module 2. A feature map with a resolution of 128*128 is obtained, which is the second feature vector obtained by combining images from 128 channels. Then, the electronic device can use a third convolution module, namely convolution module 3, to downsample the feature map with a resolution of 128*128 output from convolution module 2, resulting in a feature map with a resolution of 64*64, which is the third feature vector obtained by combining images from 256 channels. Finally, the electronic device can use a fourth convolution module, namely convolution module 4, to downsample the feature map with a resolution of 64*64 output from convolution module 3, resulting in a feature map with a resolution of 32*32, which is the fourth feature vector obtained by combining images from 512 channels, also known as the first feature map.

[0082] In some embodiments of this application, the first matting model can be based on the included i-th convolutional module, through Feature i =w i (w i-1 (…(w1x+b1)…)+b i-1 )+b i Matrix calculations are performed to extract features from the input image, resulting in an output feature map. i `x` can be the feature vector output by the i-th convolutional module; `x` can be the image data received by the first convolutional module, such as the feature map output by the input convolutional layer; `w` i b can be the weight value corresponding to the i-th convolutional module; i It can be the bias value corresponding to the i-th convolutional module.

[0083] It should be noted that the weight and bias values ​​for each convolutional module can be different. The weight and bias values ​​can be the values ​​determined during the training of the first matting model; specific values ​​are not limited in this application.

[0084] In this way, the electronic device can use the N convolutional modules included in the first matting model to perform N downsampling processes on the image received by the convolutional modules to adjust the size of the input image, thereby reducing the amount of computation for image processing in the model and the number of parameters used in the model calculation, while increasing the robustness and generalization ability of the model.

[0085] Step 202: The electronic device processes the first feature map using the first matting model to obtain at least one of the detail feature map and the edge feature map.

[0086] In some embodiments of this application, the first matting model may include N deconvolution modules.

[0087] In some embodiments of this application, the electronic device can perform upsampling processing on the first feature map through a deconvolution module to improve the resolution of the first feature map, so that the subsequent first cutout image can output an image with the same resolution as the first image and the second image.

[0088] In some embodiments of this application, the above-mentioned detailed feature map can be understood as a feature map with image details obtained by finely segmenting the image elements that the user needs to cut out in the first image.

[0089] In some embodiments of this application, the aforementioned edge feature map can be understood as a fine segmentation of the edge regions of the image elements that the user needs to cut out in the first image, resulting in a feature map that significantly improves the display effect of the edge regions.

[0090] It should be noted that the aforementioned refined segmentation can be understood as a method with segmentation accuracy higher than existing segmentation methods. For example, a segmentation method that can achieve hair-level accuracy.

[0091] The aforementioned existing segmentation methods may include, but are not limited to, any of the following: image segmentation methods that segment using a single RGB image; image segmentation methods that segment using a single RGB image and its corresponding trima.

[0092] In some embodiments of this application, in conjunction with the above steps 201a to 201c, the "processing of the first feature map to obtain a detail feature map" in step 202 can be specifically implemented through the following steps 202a1 to 202a3.

[0093] Step 202a1: The electronic device upsamples the first feature map through the first deconvolution module in the first matting model to obtain the first detail feature vector.

[0094] In some embodiments of this application, the first deconvolution module in the first matting model, namely deconvolution module 1, can be... Matrix calculations are performed to upsample the first feature map, resulting in the first detail feature vector, TransFeature1. , can be the weight value corresponding to deconvolution module 1; x can be the first feature map received by deconvolution module 1; This can be the bias value corresponding to the first deconvolution module.

[0095] In some embodiments of this application, the resolution of the image corresponding to the first detail feature vector may be greater than the resolution of the first feature map.

[0096] In some embodiments of this application, the electronic device can perform a convolution operation by performing the reverse operation of the convolution module, using the transposed convolution kernel to perform a convolution operation, thereby achieving upsampling processing of the first feature map and obtaining the first detail feature vector.

[0097] In some embodiments of this application, a deconvolution module in the first matting model can be connected to a convolution module via a skip connection. Thus, the skip connection can first linearly add the detail feature vector output by the connected deconvolution module and the feature vector output by the convolution module, and then pass them to the subsequent deconvolution module for processing.

[0098] Thus, the first matting model can enhance the image details in the feature maps and improve the predicted details of the images in the feature maps by fusing feature maps with information from different semantic levels.

[0099] For example, referring to Figure 3 and as shown in Figure 4A, after the convolution module 4 outputs a first feature map of 32*32*512, the electronic device can upsample the first feature map through the first deconvolution module, i.e., deconvolution module 1, to obtain a feature map with a resolution of 64*64. This feature map is obtained by combining images from 64 channels, i.e., the first detail feature vector. Then, the electronic device can linearly add the TransFeature1 output by deconvolution module 1 and the Feature3 output by convolution module 3 through skip1, i.e., skip1 = TransFeature1 + Feature3, as the input to the second deconvolution module for subsequent processing.

[0100] Step 202a2: The electronic device upsamples the Nj-th feature vector and the (j-1)-th detail feature vector through the j-th deconvolution module in the first matting model, and outputs the j-th detail feature vector.

[0101] Where j∈[2,N].

[0102] In some embodiments of this application, while the deconvolution module upsamples the input image data to increase the resolution of the feature map, the electronic device can compress the dimension of the output feature map to reduce the number of image channels, thereby facilitating the subsequent output of a single matted image by the first matting model, thus meeting the user's application needs.

[0103] In some embodiments of this application, methods for reducing the number of image channels may include, but are not limited to, any of the following: color space conversion, 1*1 convolution.

[0104] In some embodiments of this application, the j-th deconvolution module in the first matting model can be... Matrix calculations are performed to upsample the received feature vectors, resulting in the j-th detailed feature vector, TransFeature. j .in, It can be the weight value corresponding to the j-th deconvolution module; x can be the Nj-th feature vector and the (j-1)-th detail feature vector; It can be the bias value corresponding to the j-th deconvolution module.

[0105] For example, referring to Figure 4A, the second deconvolution module in the first matting model, namely deconvolution module 2, can receive the image data transmitted by skip1, i.e., skip1 = TransFeature1 + Feature3. Therefore, deconvolution module 2 can... Matrix calculations are performed to upsample the feature vector corresponding to the received skip1, resulting in the second detailed feature vector, TransFeature2. These can be the weight values ​​corresponding to deconvolution module 2; This can be the bias value corresponding to deconvolution module 2. Furthermore, the second detail feature vector, TransFeature2, is a feature map with a resolution of 128*128, and this feature map is obtained by combining images from 32 channels.

[0106] Step 202a3: The electronic device determines the Nth detail feature vector as a detail feature map.

[0107] In some embodiments of this application, the electronic device can determine the detail feature vector output by the last deconvolution module as a detail feature map.

[0108] For example, referring to Figure 4A and as shown in Figure 4B, the electronic device can first linearly add the TransFeature2 output by the deconvolution module 2 and the Feature2 output by the convolution module 2 via skip2, i.e., skip2 = TransFeature2 + Feature2, as the input to the third deconvolution module, i.e., deconvolution module 3. Thus, deconvolution module 3 can... Matrix calculations are performed to upsample the feature vector corresponding to the received skip2, resulting in a third detailed feature vector, TransFeature3. These can be the weight values ​​corresponding to deconvolution module 3; This can be the bias value corresponding to deconvolution module 3. Furthermore, the third detail feature vector, TransFeature3, is a feature map with a resolution of 256*256, and this feature map is obtained by combining images from 16 channels. Then, the electronic device can linearly add the TransFeature3 output by deconvolution module 3 and the Feature1 output by convolution module 1 via skip3, i.e., skip3 = TransFeature3 + Feature1. Therefore, the fourth deconvolution module in the first matting model, i.e., deconvolution module 4, can receive the image data transmitted by skip3. Thus, deconvolution module 4 can... Matrix calculations are performed to upsample the feature vector corresponding to the received skip3, resulting in the fourth detail feature vector, TransFeature4. These can be the weight values ​​corresponding to deconvolution module 4; This can be the bias value corresponding to deconvolution module 4. Furthermore, the fourth detail feature vector, TransFeature4, is a feature map with a resolution of 512*512, and this feature map is obtained by combining images from 16 channels. Therefore, the electronic device can determine the fourth detail feature vector, TransFeature4, as a detail feature map.

[0109] In this way, electronic devices can improve the segmentation accuracy of the first matting model in terms of the detailed positions of the image elements to be matted through convolution and deconvolution modules, thereby obtaining matted images with high accuracy directly without the need for users to manually draw the edge positions of image elements. This reduces the steps that users need to perform manually. Furthermore, since the deconvolution module performs feature fusion processing on multiple feature maps during the calculation process, it can enrich the details of the feature output of each layer in the calculation process of the first matting model, thereby improving the level of refinement.

[0110] In some embodiments of this application, after step 202a2 above, the image processing method provided by the embodiments of this application further includes the following steps A1 and A2.

[0111] Step A1: The electronic device uses the j / 2 edge attention module in the first matting model to perform feature fusion on the (j-1)th detail feature vector and the jth detail feature vector to obtain the j / 2th edge feature vector.

[0112] Where j is an even number.

[0113] In some embodiments of this application, the aforementioned edge attention module can improve the segmentation details at the edges in the input image, so that the first matting model can determine the central region through high-level features. This allows the deconvolution module following the edge attention module in the first matting model to focus on training the edge regions of image elements in the image, so as to output a feature map with fine edge effects.

[0114] In some embodiments of this application, the above-described feature fusion method may include, but is not limited to, at least one of the following: linearly multiplying feature vectors or linearly adding feature vectors.

[0115] It should be noted that the specific feature fusion method can be determined according to actual needs, and this application does not limit it here.

[0116] In some embodiments of this application, since the resolutions of the feature maps corresponding to the (j-1)th detail feature vector and the jth detail feature vector may be different, the first matting model can first perform upsampling processing on the feature map with smaller resolution so that the resolutions of the feature maps received by the edge attention module are all the same, thereby enabling the edge attention module to perform related processing of feature fusion.

[0117] In some embodiments of this application, the first matting model may first preprocess the feature map corresponding to the j-th detail feature vector to output two mask images.

[0118] For example, assume j = 2. The first edge attention module in the first matting model, i.e., edge attention module 1, can first, based on the second detail feature vector output by the second deconvolution module, whose feature map can be an image similar to the image shown in Figure 2, set the image regions between 0 and 1 in the feature map of the second detail feature vector to white and the other regions to black, to obtain a mask image as shown in Figure 5A, i.e., mask1. Then, the first matting model can perform color inversion processing based on mask1 to obtain a mask image as shown in Figure 5B, i.e., mask2. Thus, the input of edge attention module 1 is: the first detail feature vector output by deconvolution module 1, such as TransFeature1; the second detail feature vector output by deconvolution module 2, such as TransFeature2; mask1 and mask2.

[0119] Furthermore, the edge attention module 1 can perform linear multiplication based on TransFeature1 and mask2 to retain the image portion with higher certainty in the central region of the feature map, and can perform linear multiplication based on TransFeature2 and mask1 to obtain the image portion of the edge region of the image element in the accurate feature map. Then, the edge attention module 1 can linearly add the results of the above linear multiplication, i.e., EdgeAtt1 = TransFeature1 * mask2 + TransFeature2 * mask1, combining the central and edge regions of the image element in the feature map, thereby improving the fineness of the edge matting of the output first edge feature vector EdgeAtt1.

[0120] Step A2: The electronic device upsamples the Nj-th feature vector, the j-th detail feature vector, and the j / 2-th edge feature vector through the (j+1)-th deconvolution module in the first matting model, and outputs the (j+1)-th detail feature vector.

[0121] In some embodiments of this application, the (j+1)th deconvolution module in the first matting model can be... Matrix calculations are performed to upsample the received feature vectors, resulting in the (j+1)th detail feature vector, TransFeature. j+1 .in, It can be the weight value corresponding to the (j+1)th deconvolution module; x can be the Njth feature vector, the jth detail feature vector, and the j / 2th edge feature vector; It can be the bias value corresponding to the j-th deconvolution module.

[0122] For example, referring to Figure 4A, as shown in Figure 6, the electronic device can first linearly add the TransFeature2 output by the deconvolution module 2, the Feature2 output by the convolution module 2, and the EdgeAtt1 output by the attention edge module 1 via skip2, i.e., skip2 = TransFeature2 + Feature2 + EdgeA1, as the input to the third deconvolution module, i.e., deconvolution module 3. Thus, deconvolution module 3 can... Matrix calculations are performed to upsample the feature vector corresponding to the received skip2, resulting in a third detailed feature vector, TransFeature3. These can be the weight values ​​corresponding to deconvolution module 3; This can be the bias value corresponding to deconvolution module 3. Furthermore, the third detail feature vector, TransFeature3, is a feature map with a resolution of 256*256, and this feature map is obtained by combining images from 16 channels.

[0123] It should be noted that during the execution of steps 202a2, A1, and A2 above, the electronic device executes the processing procedures corresponding to each module sequentially based on the connection order of the modules in the first matting model. Simultaneously, all even-numbered deconvolution modules in the N deconvolution models execute the content of step 202a2, while all odd-numbered deconvolution modules execute the content of steps A1 and A2. Then, the electronic device can determine the detail feature vector output by the last deconvolution module as the detail feature map.

[0124] For example, when the first matting model includes 6 deconvolution modules and 3 edge attention modules, the electronic device can execute step 202a2 through deconvolution module 2 after executing step 202a1, then execute step A1 through edge attention module 1, and then execute step A2 through deconvolution module 3; further, execute step 202a2 through deconvolution module 4, then execute step A1 through edge attention module 2, and then execute step A2 through deconvolution module 5; and even further, execute step 202a2 through deconvolution module 6, and then execute step A1 through edge attention module 3.

[0125] For example, referring to Figures 4B and 6, as shown in Figure 7, the electronic device can linearly add the TransFeature3 output by the deconvolution module 3 and the Feature1 output by the convolution module 1 via skip3, i.e., skip3 = TransFeature3 + Feature1. Therefore, the fourth deconvolution module in the first matting model, i.e., deconvolution module 4, can receive the image data transmitted by skip3. Thus, deconvolution module 4 can... Matrix calculations are performed to upsample the feature vector corresponding to the received skip3, resulting in the fourth detail feature vector, TransFeature4. These can be the weight values ​​corresponding to deconvolution module 4; This can be the bias value corresponding to deconvolution module 4. Furthermore, the fourth detail feature vector, TransFeature4, is a feature map with a resolution of 512*512, and this feature map is obtained by combining images from 16 channels. Therefore, the electronic device can determine the fourth detail feature vector, TransFeature4, as a detail feature map.

[0126] In some embodiments of this application, in conjunction with the above steps 201a to 201c, the step 202 "processing the first feature map to obtain an edge feature map" can be specifically implemented through the following steps 202b1 and 202b2.

[0127] Step 202b1: The electronic device performs feature fusion on the (j-1)th detail feature vector and the jth detail feature vector through the j / 2th edge attention module in the first matting model to obtain the j / 2th edge feature vector.

[0128] In this embodiment of the application, the (j-1)th detail feature vector is the detail feature vector output by the (j-1)th deconvolution module in the first matting model, and the jth detail feature vector is the detail feature vector output by the jth deconvolution module in the first matting model.

[0129] Where j∈[2,N], and j is an even number.

[0130] It should be noted that the specific process of the electronic device obtaining the j / 2th edge feature vector in step 202b1 can be found in the relevant descriptions in steps A1 and A2 above, and will not be repeated here.

[0131] Step 202b2: The electronic device determines the edge feature vector corresponding to the last edge attention module in the first matting model as the edge feature map.

[0132] In some embodiments of this application, when N is even, the last edge attention module can be the N / 2th edge attention module; when N is odd, the last edge attention module can be (N-1) / 2 edge attention modules.

[0133] For example, referring to Figure 7, the electronic device can use the second edge attention module in the first matting model, namely edge attention module 2, to set the image regions between 0 and 1 in the feature map of TransFeature3 (output of deconvolution module 3) to white and other regions to black, to obtain a mask image similar to that shown in Figure 5A, namely mask3. Then, the first matting model can perform color inversion processing based on mask3 to obtain a mask image similar to that shown in Figure 5B, namely mask4. Thus, the inputs of edge attention module 2 are: TransFeature3 output by deconvolution module 3, TransFeature4 output by deconvolution module 4, mask3, and mask4. Edge attention module 2 can perform linear multiplication based on TransFeature3 and mask4 to retain the image portion with high certainty in the central region of the feature map, and can also perform linear multiplication based on TransFeature4 and mask3 to obtain the image portion of the edge region of the image element in the accurate feature map. Then, edge attention module 2 can linearly add the results of the above linear multiplication based on EdgeAtt2 = TransFeature3*mask4 + TransFeature4*mask3 to combine the central and edge regions of the image element in the feature map, thereby improving the fineness of the edge location matting of the image element in the output second edge feature vector EdgeAtt2. At the same time, the electronic device can determine the edge feature vector corresponding to the last edge attention module, that is, the EdgeAtt2 corresponding to edge attention module 2, as the edge feature map.

[0134] In this way, electronic devices can improve the segmentation accuracy of the first matting model in terms of the details and edges of the image elements to be matted through the deconvolution module and the edge attention module. This results in a matted image with high accuracy that can be obtained directly without the need for the user to manually draw the edge positions of the image elements, reducing the steps that the user needs to perform manually. Furthermore, since the deconvolution module and the edge attention module perform feature fusion processing on multiple feature maps during the calculation process, the details of the feature output of each layer in the calculation process of the first matting model can be enriched, thereby improving the level of refinement.

[0135] Step 203: The electronic device outputs a first matted image based on at least one of the detail feature map and the edge feature map, and the first feature map, using the first matting model.

[0136] In some embodiments of this application, the electronic device may include an output convolutional layer.

[0137] In some embodiments of this application, the electronic device can output a first matted image by performing feature fusion and upsampling processing based on a first feature map, a detail feature map, and an edge feature map through an output convolutional layer.

[0138] For example, referring to Figure 7, the electronic device can first use skip4 to linearly add the detail feature map TransFeature4 output by the deconvolution module 4, the Feature0 output by the input convolutional layer (i.e., the feature map obtained based on the first and second images), and the edge feature map EdgeAtt2 output by the attention edge module 2, i.e., skip4 = TransFeature4 + Feature0 + EdgeA2, as the input to the output convolutional layer. Thus, the output convolutional layer can... Matrix calculations are performed to upsample the feature vector corresponding to the received skip4, resulting in the feature vector TransFeature0, which is the first matted image mentioned above. This can be the weight values ​​corresponding to the output convolutional layer; This can be the bias value corresponding to the output convolutional layer. Furthermore, the resolution of the first matted image is 512*512, and the first matted image is obtained by combining images from 4 channels.

[0139] In some embodiments of this application, the electronic device can output a first matted image by performing feature fusion and upsampling processing based on the edge feature map and the first feature map through the output convolutional layer.

[0140] For example, referring to Figure 7, the electronic device can linearly add the edge feature map EdgeAtt2 output by the attention edge module 2 and the Feature0 output by the input convolutional layer through skip4, that is, the feature map obtained based on the first image and the second image, i.e., skip4 = Feature0 + EdgeAtt2, as the input of the output convolutional layer, i.e., the first cutout image mentioned above.

[0141] In some embodiments of this application, the electronic device can output a corresponding first matted image by performing feature fusion and upsampling processing based on the detail feature map and the first feature map through the output convolutional layer.

[0142] For example, as shown in FIG4B, the electronic device can linearly add the detail feature map TransFeature4 output by the deconvolution module 4 and the Feature0 output by the input convolutional layer, i.e. the feature map obtained based on the first image and the second image, by skip4, i.e., skip4 = TransFeature4 + Feature0, as the input of the output convolutional layer, i.e. the first cutout image mentioned above.

[0143] In the image processing method provided in this application embodiment, since the second image is obtained based on the image elements that need to be cut out in the first image, the second image includes the image elements that need to be cut out. Therefore, the electronic device can process the second image and the original first image to further obtain at least one of the detail feature map and the edge feature map. That is, the first cut-out image output by the first cut-out model can accurately retain at least one of the detail image content and the edge image content of the image elements that the user needs to cut out in the first image, based on the second image, so that the first cut-out image can meet the user's requirements for the fineness of the cut-out image. In this way, the image quality of the cut-out image obtained by the electronic device is improved.

[0144] In some embodiments of this application, referring to FIG1 and FIG8, before step 201 above, the image processing method provided in the embodiments of this application further includes the following steps 301 to 303.

[0145] Step 301: The electronic device performs image cutout processing on the first image elements in the first image to obtain the second cutout image.

[0146] In this embodiment of the application, the second cutout image may include the first image element.

[0147] In some embodiments of this application, the first image element can be an image element in the first image that the user needs to cut out.

[0148] In some embodiments of this application, the above-described image cutout process can be an "instant image cutout" method.

[0149] It should be noted that the algorithm mainly involved in the above-mentioned "instant image matting" method is the edge-side salient object detection (SOD) algorithm, which refers to the method of segmenting the main image elements in the image to be matted in order to extract the approximate segmentation result of the image elements selected by the user.

[0150] In some embodiments of this application, when an electronic device displays a first image that needs to be cut out, the electronic device can receive a second input from the user to the first image, and determine the first image element based on the image area corresponding to the second input, thereby performing cutout processing on the first image element.

[0151] In some embodiments of this application, the second input may include any of the following: click input, swipe input, long press input, or other feasible inputs, which are not limited in this application embodiment.

[0152] In some embodiments of this application, the above-mentioned click input can be any number of clicks.

[0153] In some embodiments of this application, the above-mentioned sliding operation can be a sliding operation in any direction, such as sliding up, sliding down, sliding left, or sliding right, etc., and this is not limited in the embodiments of this application.

[0154] In some embodiments of this application, the electronic device can determine the image region corresponding to the second input based on the input parameters of the second input described above, thereby determining the first image element.

[0155] In some embodiments of this application, the above-mentioned input parameters may include, but are not limited to, at least one of the following: input position, input trajectory.

[0156] For example, a mobile phone is used as an example for illustration. As shown in Figure 9A, the mobile phone can display an image editing interface 10 to display an image 1 that the user wants to edit. In image 1, a man holding a beverage cup is taking a photo with buildings in the background. If the user wants to adjust the position of the man in image 1, as shown in Figure 9B, the user can long press the "man" in image 1 to trigger the mobile phone to determine the image of the "man" in image 1 that the user wants to obtain based on the position of the user's press. Then, the "instant cutout" method can be used to cut out the image area of ​​the "man" selected by the user, to cut out the gray area shown in Figure 9C, to obtain the cutout image corresponding to the "man", that is, the second cutout image.

[0157] Step 302: The electronic device segments the first image based on the image elements in the first image to obtain at least two segmented images.

[0158] In some embodiments of this application, the electronic device may determine the image elements contained in the first image based on the type of image elements.

[0159] In some embodiments of this application, the types of the above-mentioned image elements may include, but are not limited to, any of the following: people, animals, objects, buildings, and natural environments.

[0160] In some embodiments of this application, when an image element contains multiple people, the electronic device can segment the image corresponding to the multiple people based on information such as gender, age group, person's action, and person's clothing.

[0161] In some embodiments of this application, when an image element contains an animal, the electronic device can segment the image corresponding to the animal based on information such as the animal's species, color, and posture.

[0162] In some embodiments of this application, when an image element contains an object, the electronic device can segment the image corresponding to the object based on information such as the object's shape, color, and purpose.

[0163] In some embodiments of this application, the electronic device can classify image elements in a first image by using an end-to-end object detector.

[0164] For example, the end-to-end object detector described above can be the Dino classification method, or it can be a classification method that is an improvement on the Dino classification method. For example, the Dino model can be fine-tuned so that the fine-tuned classification model is more suitable for classifying image elements of images stored in an album.

[0165] In some embodiments of this application, the electrons can still be segmented in the first image based on the type of image elements using the Segment Anything Model (SAM).

[0166] In some embodiments of this application, the above-mentioned at least two segmented images may include: a segmented image corresponding to at least one image element included in the first image, and a segmented image corresponding to the image background of the first image.

[0167] In some embodiments of this application, after the electronic device obtains at least two segmented images, the electronic device can add a label to each segmented image that includes image elements to annotate the image information corresponding to each image element.

[0168] In some embodiments of this application, the above-mentioned image information may include, but is not limited to, at least one of the following: type and location.

[0169] For example, referring to Figure 9A, the mobile phone can determine, based on the Dino method, that Image 1 contains three image elements, such as a "man," a beverage cup, and a building in the background. Then, the mobile phone can segment these three image elements based on the SAM model to obtain four segmented images, such as segmented images 1 to 4. Segmented image 1 can be the segmented image including the "man," segmented image 2 can be the segmented image including the beverage cup, segmented image 3 can be the segmented image including the building, and segmented image 4 can be the segmented image corresponding to the remaining image in Image 1 excluding the three image elements. Then, the mobile phone can add labels to the segmented images containing image elements, i.e., segmented images 1 to 3, to indicate the type of the image elements. For example, the mobile phone can add a "person" label to segmented image 1 to indicate that the image element included in segmented image 1 is a person; or, the mobile phone can add specific labels such as "young" or "man" to segmented image 1 to describe the person's information.

[0170] Step 303: The electronic device determines the second image based on the area relationship between the first segmented image and the second matted image.

[0171] In this embodiment of the application, the first segmented image can be one of the at least two segmented images that includes the first image element.

[0172] In some embodiments of this application, the method by which the electronic device determines the first segmented image may include, but is not limited to, any of the following:

[0173] The electronic device can determine the first image element that the user needs to cut out based on the input position of the second input mentioned above, and then determine the first segmented image based on the image elements contained in each of the at least two segmented images;

[0174] The electronic device can determine the first segmented image based on the input position of the second input and the original display position corresponding to the at least two segmented images;

[0175] The electronic device can also determine the largest bounding box based on the bounding box corresponding to the first image element in the second matted image and the bounding boxes corresponding to the segmented images containing the first image element in the at least two segmented images, so as to perform image segmentation on the image elements contained in the largest bounding box and determine the first segmented image.

[0176] In some embodiments of this application, the electronic device can represent the display position of the outer frame in the image by the coordinates of the upper left and lower right vertices of the outer frame.

[0177] In some embodiments of this application, the electronic device may determine the above-mentioned maximum outer frame based on the following set of formulas (1).

[0178] Wherein, (new_x1, new_y1) can be the top-left vertex of the maximum bounding box, (new_x2, new_y2) can be the bottom-right vertex of the maximum bounding box; (sod_x1, sod_y1) can be the top-left vertex of the bounding box of the first image element in the second cutout image, (sod_x2, sod_y2) can be the bottom-right vertex of the first image element in the second cutout image; (label_x1, label_y1) can be the top-left vertex of the partition image corresponding to the first image element, (label_x2, label_y2) can be the bottom-right vertex of the partition image corresponding to the first image element; img_width can be the width of the first image, and img_height can be the height of the first image.

[0179] In some embodiments of this application, in the above formula group (1), the electronic device can determine the minimum horizontal value and the minimum vertical value based on the coordinates of the vertex of the upper left corner of the second cutout image and the coordinates of the vertex of the upper left corner of the segmented image corresponding to the first image element, so as to combine them into the coordinates of the vertex of the upper left corner of the maximum bounding box. Since the coordinates of the vertex of the upper left corner of the maximum bounding box are located inside the first image, the minimum value of the coordinates of the vertex of the upper left corner of the maximum bounding box is 0, that is, it is located at the upper left corner of the first image.

[0180] In some embodiments of this application, in the above formula group (1), the electronic device can determine the maximum horizontal value and the maximum vertical value based on the coordinates of the lower right corner vertex of the second cutout image and the coordinates between the lower right corner vertex of the segmented image corresponding to the first image element, so as to combine them into the coordinates of the lower right corner vertex of the maximum bounding box. Since the coordinates of the lower right corner vertex of the maximum bounding box are located inside the first image, the horizontal coordinate of the lower right corner vertex of the maximum bounding box is less than the width of the first image and the vertical coordinate is less than the height of the first image.

[0181] For example, assume the width of the first image is 12 and the height is 15. Referring to Figure 9C, the second cutout image can be the cutout image 11 shown in Figure 10A. The mobile phone can establish a coordinate system based on the upper left corner of the cutout image 11 as the origin, with the horizontal axis as the x-axis and the vertical axis as the y-axis. Then, the mobile phone can use the coordinates (sod_x1, sod_y1) of the upper left vertex A of the outer frame 12, such as (5,7); and the coordinates (sod_x2, sod_y2) of the lower right vertex B, such as (10,12), to represent the display position of the outer frame 12. The segmented image corresponding to the first image element can be segmented image 13 as shown in Figure 10B. The mobile phone can establish a coordinate system based on the upper left corner of segmented image 13 as the origin, with the horizontal axis as the x-axis and the vertical axis as the y-axis. Then, the mobile phone can use the coordinates (label_x1, label_y1) of the upper left vertex C of the outer frame 14, such as (5,6), and the coordinates (label_x2, label_y2) of the lower right vertex D, such as (10,12), to represent the display position of the outer frame 14. Thus, the mobile phone can perform the following calculations: new_x1 = max(0,min(5,5)) = 5 new_y1 = max(0,min(7,6)) = 6 new_x2 = min(12,max(10,10)) = 10 new_y2 = min(15,max(12,12)) = 12

[0182] The coordinates of the top-left vertex of the maximum bounding box are determined to be (5,6), and the coordinates of the bottom-right vertex are determined to be (10,12), which are also the vertex coordinates of bounding box 14. Then, the mobile phone can determine bounding box 14 as the maximum bounding box and perform image segmentation based on the image elements contained in the maximum bounding box to determine the first segmented image mentioned above.

[0183] In some embodiments of this application, when the ratio between the first segmented image and the second matted image is within a preset range, the electronic device can determine the first segmented image as the second image; when the ratio between the first segmented image and the second matted image is not within the aforementioned preset range, the electronic device can determine the second matted image as the second image.

[0184] For example, assume that the maximum connected component area of ​​the second cutout image is area1, and the maximum connected component area of ​​the first segmented image is area2.

[0185] That is, when the area of ​​the first segmented image is greater than 1.75 times the area of ​​the second cutout image, the electronic device can determine the second cutout image as the second image; when the area of ​​the first segmented image is less than 0.5 times the area of ​​the second cutout image, the electronic device can determine the second cutout image as the second image; when the area of ​​the first segmented image is between 0.5 and 1.75 times the area of ​​the second cutout image, the electronic device can determine the first segmented image as the second image.

[0186] It should be noted that the 0.5 times and 1.75 times mentioned above are only for illustration. The specific value setting can be determined according to the user's needs in actual application. This application does not limit it here.

[0187] In this embodiment, the electronic device can obtain multiple cutout results using various cutout methods. This avoids situations where, during the process of directly obtaining a cutout result using a single method, the stored cutout result may contain unwanted image portions or may not fully include the desired image portions. Simultaneously, the electronic device can determine the second image to be used subsequently by comparing the area ratio between the first segmented image and the second cutout image. This prevents excessive discrepancies between the two segmentation results, which could lead to a significant deviation between the cutout result and the content required by the user.

[0188] In some embodiments of this application, referring to FIG1 and FIG11, after step 203 above, the image processing method provided by the embodiments of this application further includes the following steps 401 to 403.

[0189] Step 401: The electronic device displays the first cutout image in the first image.

[0190] Step 402: The electronic device receives the first input to the first cutout image.

[0191] In some embodiments of this application, the first input described above may be an input that triggers the electronic device to adjust the display position of the first cutout image in the first image.

[0192] In some embodiments of this application, the first input may include any of the following: click input, swipe input, gesture input, or other feasible inputs, which are not limited in this embodiment.

[0193] In some embodiments of this application, the above-mentioned click input can be any number of clicks.

[0194] In some embodiments of this application, the above-mentioned sliding operation can be a sliding operation in any direction, such as sliding up, sliding down, sliding left, or sliding right, etc., and this is not limited in the embodiments of this application.

[0195] In some embodiments of this application, the above-mentioned gesture input may include, but is not limited to, at least one of the following: click gesture, swipe gesture, drag gesture, pressure recognition gesture, long press gesture, area change gesture, double press gesture, double tap gesture, specific gesture input or other possible gesture inputs. The specific gesture input form can be determined according to actual needs, and is not limited in this embodiment of the application.

[0196] Step 403: In response to the first input, the electronic device adjusts the display position of the first cutout image in the first image and fills the original position of the first cutout image with an image.

[0197] In some embodiments of this application, during the process of adjusting the display position of the first cutout image by the electronic device, the electronic device can adjust the display size of the first cutout image according to the user's input.

[0198] In some embodiments of this application, after the electronic device adjusts the display position of the first cutout image according to the user's input, if the electronic device does not receive any user input within a preset time period, the electronic device can begin to fill the original position of the first cutout image. For example, the preset time period can be 3 seconds.

[0199] It should be noted that the above-mentioned preset time period setting can be the default time length set by the electronic device, or it can be set according to the user's needs. This application does not limit it here.

[0200] In some embodiments of this application, the electronic device can fill the original position of the first matted image using a diffusion model. This diffusion model (Stable Diffusion) may include an inpaint module and an outpaint module.

[0201] It should be noted that the aforementioned diffusion model is a deep learning method based on diffusion models. It achieves the conversion from text to image by learning the mapping relationship between a large amount of image and text data. Furthermore, it maintains the structural and texture consistency of the image during the image generation process based on the diffusion model.

[0202] The working principle of the Inpaint module can be summarized in two steps: First, the user needs to provide an image to be repaired and draw a mask on the image to indicate the area that needs to be repaired; then, the deep learning algorithm can generate a new image based on the provided image and mask, in which the defective areas are repaired or redrawn.

[0203] The processing flow of the Outpaint module is similar to that of the Inpaint module, but the goal of the Outpaint module's algorithm is to complete the details of areas outside the image.

[0204] For example, in conjunction with 9A, if a user wants to adjust the display position of "man" in image 1 and reduce the display size of "man", the user can long-press the display area of ​​"man" in image 1 to trigger the phone to obtain the corresponding first cutout image based on "man" using the first cutout model. Then, the user can use two-finger pinch input to trigger the phone to reduce the display size of the first cutout image according to the user's needs. Then, the user can use drag input to trigger the phone to adjust the display position of the first cutout image. Then, the phone can fill the original position of the first cutout image to display image 15 as shown in Figure 12.

[0205] For example, as shown in Figure 13A, assuming the first image is image 16 containing a teddy bear and a building, and the teddy bear is located in the right edge area of ​​image 16, if the user wants to adjust the display position of the teddy bear to the center area of ​​image 16, the user can long-press the display area of ​​the teddy bear in image 16 to trigger the phone to obtain the corresponding cutout image based on the teddy bear using the first cutout model. Then, the user can drag the input to trigger the phone to adjust the display position of the cutout image corresponding to the teddy bear. Then, the phone can fill the original position of the cutout image corresponding to the teddy bear with the background building, and can complete the teddy bear based on the content in the cutout image, thereby displaying image 17 as shown in Figure 13B.

[0206] In this embodiment, the electronic device can adjust the display parameters of the first cutout image in the first image, and can fill the original display position of the first cutout image with an image, so that the user can perform secondary composition on the first image to make the adjusted first image meet the user's actual needs. This improves the convenience of image processing by the electronic device.

[0207] In some embodiments of this application, in conjunction with steps 401 to 403 above, the image processing method provided in this application embodiment further includes the following step 404.

[0208] Step 404: The electronic device adjusts the display parameters of the first cutout image based on the display parameters of the adjacent areas of the first cutout image after the position adjustment.

[0209] In some embodiments of this application, the above display parameters may include, but are not limited to, any of the following: image brightness, image saturation, and image color levels.

[0210] In some embodiments of the present application, before adjusting the display parameters of the first matte image, the electronic device can perform image processing on the edge region of the first matte image to weaken the sense of jerk at the edge of the first matte image.

[0211] In some embodiments of the present application, the electronic device can calculate the central brightness of the image based on the main region of the first image and the original image background region corresponding to the first matte image after the display position is adjusted, so as to adjust the tone curve of the first matte image, so that the brightness of the first matte image is similar to that of the adjacent region.

[0212] In some embodiments of the present application, the electronic device can represent the first image through the LAB color model, and then extract the L channel in the LAB color model, that is, the brightness channel, so as to define the average value of the brightness values of all pixels in the first image through the average brightness mean_light. Thus, different tone outputs can be adjusted through different average brightness values, so that the main color of the first matte image is adapted to the color of the adjacent region after the adjustment position. Among them,

[0213] The Lab color model consists of three elements. L represents brightness, and A and B are two color channels. Among them, the colors included in A are from dark green to gray to bright pink, that is, from low brightness values to medium brightness values to high brightness values; B is from bright blue to gray to yellow, that is, from low brightness values to medium brightness values to high brightness values.

[0214] In some embodiments of the present application, when the difference between the average brightness of the first matte image and the average brightness of the adjacent region of the first matte image after the adjustment position is greater than a preset threshold, the electronic device can adjust the display parameters of the first matte image. For example, the above preset threshold can be 50.

[0215] Exemplarily, assume that the average brightness of the first matte image is mean_light, and the average brightness of the adjacent region of the first matte image after the adjustment position is mean_light2. Then, when mean_light < mean_light2 - 50 or mean_light > mean_light2 + 50, the electronic device can adjust the tone range of the first matte image.

[0216] It should be noted that the setting of the above preset threshold can be determined according to actual needs, and the present application does not limit this here.

[0217] In the embodiments of the present application, the electronic device can adjust the display parameters of the first matte image, so that the first matte image after movement can match the display parameters of the adjacent region after movement, thereby improving the display effect of the adjusted first image.

[0218] In some embodiments of this application, the image processing method provided in this application further includes the following steps 501 to 503.

[0219] Step 501: The electronic device inputs the sample image into the second matting model, performs matting processing on the second image elements in the sample image, and obtains the third matted image.

[0220] Step 502: The electronic device calculates the loss value based on the third matted image and the target matted image.

[0221] In this embodiment of the application, the loss value described above represents the difference in detail features and edge features between the third matted image and the target matted image.

[0222] Step 503: Based on the loss value, the electronic device adjusts the model parameters in the second matting model to obtain the first matting model.

[0223] In some embodiments of this application, the sample images described above can be images with a sufficiently large number of samples. For example, 5000 sample images.

[0224] It should be noted that the specific number of sample images can be determined according to the actual needs during the model training process, and this application does not limit it here.

[0225] In some embodiments of this application, the structure of the second matting model is largely the same as that of the first matting model. For details, please refer to the description of the above embodiments. The embodiments of this application will not be repeated here.

[0226] In some embodiments of this application, the target cutout image can be a cutout image obtained by manual drawing, which has a high degree of precision.

[0227] It should be noted that the smaller the difference between the third cutout image and the target cutout image, the closer the display effect of the third cutout image is to the cutout image required by the user, and the smaller the loss value.

[0228] In some embodiments of this application, the electronic device may use a backpropagation algorithm based on the loss value to iteratively update the weight values ​​and bias values ​​in the second matting model until convergence, so as to train the first matting model.

[0229] For example, suppose the second matting model includes one input convolutional layer, four convolutional modules, four deconvolutional modules, one output convolutional layer, and two edge attention modules. The second matting model can perform five downsampling operations on the input image using the one input convolutional layer and four convolutional modules. Then, the first feature map output by the fourth convolutional module is input to deconvolutional module 1 (one of the four deconvolutional modules) for upsampling, resulting in detail feature vector 1. Next, the electronic device can linearly add the detail feature vector 1 output by deconvolutional module 1 and the feature vector 3 output by convolutional module 3 via skip connection 1, using this as input to deconvolutional module 2 for upsampling, resulting in detail feature vector 2. Then, the electronic device can input detail feature vector 1 and detail feature vector 2 to edge attention module 1 for feature fusion, outputting edge feature vector 1. Finally, skip connection 2 can perform upsampling based on the detail feature vector 2 output by deconvolutional module 2 and the feature vector 2 output by convolutional module 2. The edge feature vector 1 output by edge attention module 1 is linearly added to the feature vector 1 output by edge attention module 1, and then upsampled through deconvolution module 3 to obtain detail feature vector 3. Then, the electronic device can linearly add the detail feature vector 3 output by deconvolution module 3 and the feature vector 1 output by convolution module 1 through skip connection 3, and use it as input to deconvolution module 4 to perform upsampling to obtain detail feature vector 4. Then, the electronic device can input detail feature vector 3 and detail feature vector 4 into edge attention module 2 for feature fusion and output edge feature vector 2. Then, skip connection 4 can linearly add the detail feature vector 4 output by deconvolution module 4, the feature vector output by the input convolution layer and the edge feature vector 2 output by edge attention module 2, and perform upsampling through the output convolution layer to output the third matted image.

[0230] Furthermore, the second detection model can calculate a loss value based on the third matted image and the target matted image. Then, based on the loss value, the backpropagation algorithm is used to iteratively update the weights and biases in the second matting model until convergence, thereby training the first matting model.

[0231] To illustrate the various scenarios in which the embodiments of this application can be applied, and in conjunction with the various implementation schemes of the embodiments of this application described above, specific examples are given below to explain the implementation process of the embodiments of this application in various scenarios. A mobile phone is used as an example for illustration.

[0232] Example 1: Recompose the image of the person in the picture.

[0233] If a user wants to recompose an image, such as adjusting the position of person 1 in image 1 relative to the background buildings, the user can trigger an electronic device to acquire a cutout image corresponding to image 1 based on existing "instant cutout". Image 1 and its corresponding cutout image are then input into a first cutout model. This model outputs a first cutout image with detailed and edge features corresponding to person 1, such as cutout image 1. The electronic device can then adjust the display position of cutout image 1 based on the user's input, fill the original display position of cutout image 1 with images based on the background buildings, and adjust parameters such as brightness of the moved cutout image 1. Thus, the user obtains the desired image 1, achieving a secondary composition of image 1.

[0234] Example 2: Re-composition of objects in an image.

[0235] If a user wants to recompose an image, such as adjusting the display position and size of an object in image 2, the user can trigger the electronic device to obtain a cutout image corresponding to image 2 based on the existing "instant cutout". Image 2 and its corresponding cutout image are then input into a first cutout model. This model outputs a first cutout image with detailed and edge features corresponding to object 2, such as cutout image 2. The electronic device can then adjust the display position and size of cutout image 2 based on the user's input. Furthermore, it can fill the original display position of cutout image 2 with other image elements and adjust parameters such as brightness after the movement. This allows the user to obtain the desired image 2, thus achieving a secondary composition of image 2.

[0236] This application provides a method for training an image matting model. Figure 14 shows a flowchart of such a method, which can be applied to electronic devices. As shown in Figure 14, the image matting model training method provided in this application may include steps 601 to 603 as described below.

[0237] Step 601: The electronic device inputs the sample image into the second matting model, performs matting processing on the second image elements in the sample image, and obtains the third matted image.

[0238] Step 602: The electronic device calculates the loss value based on the third matted image and the target matted image.

[0239] In this embodiment of the application, the loss value described above represents the difference in detail features and edge features between the third matted image and the target matted image.

[0240] Step 603: Based on the loss value, the electronic device adjusts the model parameters in the second matting model to obtain the first matting model.

[0241] It should be noted that the image matting model training method in steps 601 to 603 above can be found in the description of steps 501 to 503 in the above embodiments, and will not be repeated here.

[0242] In the matting model training method provided in this application embodiment, since the first matting model can be trained by combining the detail features and edge features of the matted image on the basis of the second matting model, in actual use, the electronic device can input the second image obtained based on the image elements to be matted and the original first image into the first matting model, so that the user can obtain a first matted image with at least one of the detail features and edge features, thereby enabling the first matted image to meet the user's requirements for the precision of the matted image. In this way, the image quality of the matted image obtained by the image processing device can be improved.

[0243] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there are no contradictions, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0244] It should be noted that the image processing method provided in this application embodiment can be executed by an image processing device. This application embodiment uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application embodiment.

[0245] Figure 15 shows a possible structural schematic diagram of the image processing apparatus according to an embodiment of this application. As shown in Figure 15, the image processing apparatus 70 may include: an acquisition module 71, a processing module 72, and an output module 73;

[0246] The acquisition module 71 is used to input the first image and the second image into the first matting model, and obtain the first feature map based on the first image and the second image through the first matting model. The second image is an image obtained based on the image elements in the first image.

[0247] The processing module 72 is used to process the first feature map obtained by the acquisition module 71 through the first matting model to obtain at least one of the detail feature map and the edge feature map;

[0248] Output module 73 is used to output a first matted image based on at least one of the detail feature map and edge feature map obtained by the acquisition module, and the first feature map, through the first matting model;

[0249] The first image matting model is trained based on the image's detail and edge features.

[0250] In one possible implementation, the image processing apparatus 70 provided in this application embodiment further includes: a determining module; the processing module 72 is further configured to, before the acquisition module 71 inputs the first image and the second image into the first matting model, perform matting processing on the first image elements in the first image to obtain a second matted image, the second matted image including the first image elements; and segment the first image based on the image elements in the first image to obtain at least two segmented images; the determining module is configured to determine the second image based on the area relationship between the first segmented image and the second matted image obtained by the processing module 72; wherein, the first segmented image is a segmented image including the first image elements among the at least two segmented images.

[0251] In one possible implementation, the acquisition module 71 is specifically used to: perform downsampling processing on the first image and the second image through the first convolution module in the first matting model, and output the first feature vector; and perform downsampling processing on the (i-1)th feature vector through the i-th convolution module in the first matting model, and output the i-th feature vector, where i∈[2,N], and i is an integer, and N is the number of convolution modules; and determine the N-th feature vector as the first feature map.

[0252] In one possible implementation, the processing module 72 is specifically used to: perform upsampling processing on the first feature map through the first deconvolution module in the first matting model to obtain the first detail feature vector; and perform upsampling processing on the Nj-th feature vector and the (j-1)-th detail feature vector through the j-th deconvolution module in the first matting model to output the j-th detail feature vector, where j∈[2,N]; and determine the N-th detail feature vector as the detail feature map.

[0253] In one possible implementation, the processing module 72 is further configured to: after upsampling the Nj-th feature vector and the (j-1)-th detail feature vector to output the j-th detail feature vector, perform feature fusion on the (j-1)-th detail feature vector and the j-th detail feature vector through the (j / 2)-th edge attention module in the first matting model to obtain the (j / 2)-th edge feature vector, where j is an even number; and perform upsampling on the Nj-th feature vector, the j-th detail feature vector and the (j / 2)-th edge feature vector through the (j+1)-th deconvolution module in the first matting model to output the (j+1)-th detail feature vector.

[0254] In one possible implementation, the processing module 72 is specifically used to: perform feature fusion on the (j-1)th detail feature vector and the jth detail feature vector through the j / 2th edge attention module in the first matting model to obtain the j / 2th edge feature vector, where the (j-1)th detail feature vector is the detail feature vector output by the (j-1)th deconvolution module in the first matting model, and the jth detail feature vector is the detail feature vector output by the jth deconvolution module in the first matting model, where j∈[2,N] and j is an even number; and determine the edge feature vector corresponding to the last edge attention module in the first matting model as the edge feature map.

[0255] In one possible implementation, the image processing apparatus 70 provided in this application embodiment further includes: a display module and a receiving module; the display module is configured to display the first cutout image in a first image after outputting a first cutout image based on at least one of a detail feature map and an edge feature map, and a first feature map; the receiving module is configured to receive a first input to the first cutout image displayed by the display module; the processing module 72 is further configured to adjust the display position of the first cutout image in the first image and perform image filling on the original position of the first cutout image in response to the first input received by the receiving module.

[0256] In one possible implementation, the processing module 72 is further configured to adjust the display parameters of the first cutout image based on the display parameters of the adjacent regions of the first cutout image after the position adjustment.

[0257] In one possible implementation, the image processing apparatus 70 provided in this application embodiment further includes: a calculation module; the aforementioned acquisition module 71 is further configured to input a sample image into a second matting model, perform matting processing on the second image elements in the sample image, and obtain a third matted image; the calculation module is configured to calculate a loss value based on the third matted image and the target matted image acquired by the acquisition module 71, the loss value representing the difference in detail features and edge features between the third matted image and the target matted image; the aforementioned processing module 72 is further configured to adjust the model parameters in the second matting model based on the loss value, and obtain a first matting model.

[0258] In the image processing apparatus provided in this application embodiment, since the second image is obtained based on the image elements that need to be cut out in the first image, the second image includes the image elements that need to be cut out. Therefore, the image processing apparatus can process based on the second image and the original first image to further obtain at least one of the detail feature map and the edge feature map. That is, the first cut-out image output by the first cut-out model can accurately retain at least one of the detail image content and the edge image content of the image elements that the user needs to cut out in the first image, based on the second image, so that the first cut-out image can meet the user's requirements for the fineness of the cut-out image. In this way, the image quality of the cut-out image obtained by the image processing apparatus is improved.

[0259] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0260] The image processing device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0261] The image processing apparatus provided in this application embodiment can implement the various processes implemented in the above method embodiments, and will not be described again here to avoid repetition.

[0262] Optionally, as shown in FIG16, this application embodiment also provides an electronic device 90, including a processor 91 and a memory 92. The memory 92 stores a program or instructions that can run on the processor 91. When the program or instructions are executed by the processor 91, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0263] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0264] Figure 17 is a schematic diagram of the hardware structure of an electronic device that implements an embodiment of this application.

[0265] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0266] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for powering various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The electronic device structure shown in Figure 17 does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0267] The processor 110 is configured to input a first image and a second image into a first matting model, obtain a first feature map based on the first image and the second image through the first matting model, wherein the second image is an image obtained based on image elements in the first image; process the first feature map through the first matting model to obtain at least one of a detail feature map and an edge feature map; and output a first matted image based on at least one of the detail feature map and the edge feature map and the first feature map through the first matting model; wherein the first matting model is trained based on the detail features and edge features of the image.

[0268] Optionally, the processor 110 is further configured to: perform matting processing on the first image elements in the first image before inputting the first image and the second image into the first matting model to obtain a second matted image, the second matted image including the first image elements; and segment the first image based on the image elements in the first image to obtain at least two segmented images, the at least two segmented images including: a segmented image corresponding to at least one image element included in the first image and a segmented image corresponding to the image background of the first image; and determine the second image based on the area relationship between the first segmented image and the second matted image; wherein the first segmented image is a segmented image including the first image elements among the at least two segmented images.

[0269] Optionally, the processor 110 is specifically configured to perform downsampling processing on the first image and the second image through the first convolutional module in the first matting model, and output the first feature vector; and perform downsampling processing on the (i-1)th feature vector through the i-th convolutional module in the first matting model, and output the i-th feature vector, where i∈[2,N], and i is an integer, N is the number of convolutional modules; and determine the N-th feature vector as the first feature map.

[0270] Optionally, the processor 110 is specifically configured to perform upsampling processing on the first feature map through the first deconvolution module in the first matting model to obtain the first detail feature vector; and to perform upsampling processing on the Nj-th feature vector and the (j-1)-th detail feature vector through the j-th deconvolution module in the first matting model to output the j-th detail feature vector, where j∈[2,N]; and to determine the N-th detail feature vector as the detail feature map.

[0271] Optionally, the processor 110 is further configured to, after upsampling the Nj-th feature vector and the (j-1)-th detail feature vector to output the j-th detail feature vector, perform feature fusion on the (j-1)-th detail feature vector and the j-th detail feature vector through the (j / 2)-th edge attention module in the first matting model to obtain the (j / 2)-th edge feature vector, where j is an even number; and perform upsampling on the Nj-th feature vector, the j-th detail feature vector and the (j / 2)-th edge feature vector through the (j+1)-th deconvolution module in the first matting model to output the (j+1)-th detail feature vector.

[0272] Optionally, the processor 110 is specifically configured to: perform feature fusion on the (j-1)th detail feature vector and the jth detail feature vector through the j / 2th edge attention module in the first matting model to obtain the j / 2th edge feature vector, where the (j-1)th detail feature vector is the detail feature vector output by the (j-1)th deconvolution module in the first matting model, and the jth detail feature vector is the detail feature vector output by the jth deconvolution module in the first matting model, where j∈[2,N] and j is an even number; and determine the edge feature vector corresponding to the last edge attention module in the first matting model as the edge feature map.

[0273] Optionally, the display unit 106 is configured to display the first cutout image in the first image after outputting the first cutout image based on at least one of the detail feature map and the edge feature map, and the first feature map; the user input unit 107 is configured to receive a first input to the first cutout image; and the display unit 106 is further configured to adjust the display position of the first cutout image in the first image in response to the first input, and to perform image filling on the original position of the first cutout image.

[0274] Optionally, the processor 110 is also configured to adjust the display parameters of the first cutout image based on the display parameters of the adjacent regions of the first cutout image after the position adjustment.

[0275] Optionally, the processor 110 is further configured to input the sample image into the second matting model, perform matting processing on the second image elements in the sample image to obtain a third matted image; and calculate a loss value based on the third matted image and the target matted image, the loss value representing the difference in detail features and edge features between the third matted image and the target matted image; and adjust the model parameters in the second matting model based on the loss value to obtain a first matting model.

[0276] In the electronic device provided in this application embodiment, since the second image is obtained based on the image elements that need to be cut out in the first image, the second image includes the image elements that need to be cut out. Therefore, the electronic device can process the second image and the original first image to further obtain at least one of the detail feature map and the edge feature map. That is, the first cut-out image output by the first cut-out model can accurately retain at least one of the detail image content and the edge image content of the image elements that the user needs to cut out in the first image, based on the second image. This allows the first cut-out image to meet the user's requirements for the fineness of the cut-out image, thereby improving the image quality of the cut-out image obtained by the electronic device.

[0277] The electronic device provided in this application embodiment can implement the various processes implemented in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0278] For details on the beneficial effects of the various implementation methods in this embodiment, please refer to the beneficial effects of the corresponding implementation methods in the above method embodiments. To avoid repetition, these will not be repeated here.

[0279] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0280] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0281] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0282] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0283] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0284] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0285] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0286] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above method embodiments and achieve the same technical effects. To avoid repetition, it will not be described again here.

[0287] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0288] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0289] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

An image processing method, the method comprising: The first image and the second image are input into the first matting model. Based on the first image and the second image, the first feature map is obtained through the first matting model. The second image is an image obtained based on the image elements in the first image. The first feature map is processed using the first matting model to obtain at least one of the detail feature map and the edge feature map. Using the first matting model, based on at least one of the detail feature map and the edge feature map, and the first feature map, a first matted image is output; The first image matting model is trained based on the detail features and edge features of the image. The method of claim 1, wherein, Before inputting the first image and the second image into the first matting model, the method further includes: The first image element in the first image is cut out to obtain a second cut-out image, and the second cut-out image includes the first image element; Based on the image elements in the first image, the first image is segmented to obtain at least two segmented images; The second image is determined based on the area relationship between the first segmented image and the second matted image; Wherein, the first segmented image is a segmented image that includes the first image element among the at least two segmented images. The method of claim 1, wherein, The step of obtaining a first feature map based on the first image and the second image using the first image matting model includes: The first convolutional module in the first image matting model performs downsampling on the first image and the second image, and outputs the first feature vector. The i-th convolutional module in the first matting model performs downsampling on the (i-1)-th feature vector and outputs the i-th feature vector, where i∈[2,N], i is an integer, and N is the number of convolutional modules; The Nth feature vector is determined as the first feature map. The method according to claim 3, wherein, The step of processing the first feature map using the first matting model to obtain a detail feature map includes: The first feature map is upsampled by the first deconvolution module in the first matting model to obtain the first detail feature vector. The j-th deconvolution module in the first matting model performs upsampling on the Nj-th feature vector and the (j-1)-th detail feature vector, and outputs the j-th detail feature vector, where j∈[2,N]. The Nth detail feature vector is determined as the detail feature map. The method according to claim 4, wherein, After upsampling the Nj-th feature vector and the (j-1)-th detail feature vector to output the j-th detail feature vector, the method further includes: The (j / 2)th edge attention module in the first matting model performs feature fusion on the (j-1)th detail feature vector and the jth detail feature vector to obtain the (j / 2)th edge feature vector, where j is an even number; The (j+1)th deconvolution module in the first matting model performs upsampling on the Nj-th feature vector, the j-th detail feature vector, and the j / 2-th edge feature vector, and outputs the (j+1)-th detail feature vector. The method according to any one of claims 3 to 5, wherein, The step of processing the first feature map using the first matting model to obtain an edge feature map includes: The (j / 2)th edge attention module in the first matting model performs feature fusion on the (j-1)th detail feature vector and the jth detail feature vector to obtain the (j / 2)th edge feature vector. The (j-1)th detail feature vector is the detail feature vector output by the (j-1)th deconvolution module in the first matting model, and the jth detail feature vector is the detail feature vector output by the jth deconvolution module in the first matting model, where j∈[2,N] and j is an even number. The edge feature vector corresponding to the last edge attention module in the first matting model is determined as the edge feature map. According to the method of claim 1, wherein, After outputting a first cutout image based on at least one of the detail feature map and the edge feature map, and the first feature map, the method further includes: Display the first cutout image in the first image; Receive the first input to the first cutout image; In response to the first input, the display position of the first cutout image is adjusted in the first image, and the original position of the first cutout image is filled with an image. The method according to claim 7, wherein, The method further includes: Based on the display parameters of the adjacent regions of the first cutout image after the position adjustment, the display parameters of the first cutout image are adjusted. According to the method of claim 1, wherein, The method further includes: The sample image is input into the second image matting model, and the second image elements in the sample image are matted to obtain the third matted image. Based on the third matted image and the target matted image, a loss value is calculated, which represents the difference in detail features and edge features between the third matted image and the target matted image; Based on the loss value, the model parameters in the second matting model are adjusted to obtain the first matting model. An image processing apparatus, the image processing apparatus comprising: Acquisition module, processing module, and output module; The acquisition module is used to input the first image and the second image into the first matting model, and obtain a first feature map based on the first image and the second image through the first matting model, wherein the second image is an image obtained based on the image elements in the first image; The processing module is used to process the first feature map obtained by the acquisition module through the first matting model to obtain at least one of the detail feature map and the edge feature map; The output module is used to output a first matted image based on at least one of the detail feature map and the edge feature map obtained by the acquisition module, and the first feature map, through the first matting model; The first image matting model is trained based on the detail features and edge features of the image. The apparatus according to claim 10, wherein, The image processing device further includes: a determination module; The processing module is further configured to perform image matting processing on the first image elements in the first image before the acquisition module inputs the first image and the second image into the first matting model, to obtain a second matted image, the second matted image including the first image elements; and to segment the first image based on the image elements in the first image to obtain at least two segmented images; The determining module is used to determine the second image based on the area relationship between the first segmented image and the second cutout image obtained by the processing module; Wherein, the first segmented image is a segmented image that includes the first image element among the at least two segmented images. The apparatus according to claim 10, wherein, The acquisition module is specifically used for: The first convolutional module in the first image matting model performs downsampling on the first image and the second image, and outputs the first feature vector. The i-th convolutional module in the first matting model performs downsampling on the (i-1)-th feature vector and outputs the i-th feature vector, where i∈[2,N], i is an integer, and N is the number of convolutional modules; The Nth feature vector is determined as the first feature map. The apparatus according to claim 12, wherein, The processing module is specifically used for: The first feature map is upsampled by the first deconvolution module in the first matting model to obtain the first detail feature vector. The j-th deconvolution module in the first matting model performs upsampling on the Nj-th feature vector and the (j-1)-th detail feature vector, and outputs the j-th detail feature vector, where j∈[2,N]. The Nth detail feature vector is determined as the detail feature map. The apparatus according to claim 13, wherein, The processing module is further configured to: After upsampling the Nj-th feature vector and the (j-1)-th detail feature vector to output the j-th detail feature vector, the (j / 2)-th edge attention module in the first matting model performs feature fusion on the (j-1)-th detail feature vector and the j-th detail feature vector to obtain the j / 2-th edge feature vector, where j is an even number; The (j+1)th deconvolution module in the first matting model performs upsampling on the Nj-th feature vector, the j-th detail feature vector, and the j / 2-th edge feature vector, and outputs the (j+1)-th detail feature vector. The apparatus according to any one of claims 12 to 14, wherein, The processing module is specifically used for: The (j / 2)th edge attention module in the first matting model performs feature fusion on the (j-1)th detail feature vector and the jth detail feature vector to obtain the (j / 2)th edge feature vector. The (j-1)th detail feature vector is the detail feature vector output by the (j-1)th deconvolution module in the first matting model, and the jth detail feature vector is the detail feature vector output by the jth deconvolution module in the first matting model, where j∈[2,N] and j is an even number. The edge feature vector corresponding to the last edge attention module in the first matting model is determined as the edge feature map. The apparatus according to claim 10, wherein, The image processing device further includes: a display module and a receiving module; The display module is configured to output a first cutout image based on at least one of the detail feature map and the edge feature map, and the first feature map, and then display the first cutout image in the first image. The receiving module is used to receive a first input to the first cutout image displayed by the display module; The processing module is further configured to, in response to the first input received by the receiving module, adjust the display position of the first cutout image in the first image, and fill the original position of the first cutout image with an image. The apparatus according to claim 16, wherein, The processing module is further configured to adjust the display parameters of the first cutout image based on the display parameters of the adjacent regions of the first cutout image after the position adjustment. The apparatus according to claim 10, wherein, The image processing device further includes: a computing module; The acquisition module is further configured to input the sample image into the second matting model, perform matting processing on the second image elements in the sample image, and obtain a third matted image; The calculation module is used to calculate a loss value based on the third matted image and the target matted image acquired by the acquisition module. The loss value represents the difference in detail features and edge features between the third matted image and the target matted image. The processing module is further configured to adjust the model parameters in the second matting model based on the loss value to obtain the first matting model. An electronic device includes a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the image processing method as described in any one of claims 1 to 9. A readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the image processing method as described in any one of claims 1 to 9. A computer program product stored in a storage medium, the program product being executed by at least one processor to implement the steps of the image processing method as described in any one of claims 1 to 9.

Images