Image processing method, training method for optical flow prediction model, and related apparatuses

Abstract

The present application relates to the field of artificial intelligence, and in particular to the technical fields of image processing, computer vision, deep learning, data processing, etc. Provided are an image processing method, a training method for an optical flow prediction model, and related apparatuses. The image processing method comprises: acquiring a portrait image to be processed; performing resolution scaling processing on said portrait image, so as to obtain a target-resolution image; on the basis of skeleton points and contour points of a target human body in the target-resolution image, acquiring a PAFs image corresponding to the target-resolution image; on the basis of the PAFs image and a skeleton map, performing optical flow prediction on the target-resolution image, so as to obtain a target optical flow map corresponding to the target-resolution image, wherein the skeleton map is obtained on the basis of the skeleton points; and on the basis of the target optical flow map, performing shaping processing on said portrait image, so as to obtain a shaped target image corresponding to said portrait image. The present application can perform shaping processing more accurately, and has a relatively good generalization performance.

Description

Image processing methods, training methods for optical flow prediction models, and related devices

[0001] This application claims priority to Chinese Patent Application No. 202510193249.7, filed on February 20, 2025, entitled "Image Processing Method, Training Method for Optical Flow Prediction Model and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the technical fields of image processing, computer vision, deep learning and data processing in the field of artificial intelligence, and in particular to an image processing method, a training method for an optical flow prediction model and related apparatus. Background Technology

[0003] Artificial Intelligence (AI) adaptive reshaping technology for portrait images is a technique that uses artificial intelligence algorithms to automatically adjust and optimize portrait images. It has important applications in fields such as portrait photography retouching and online social networking.

[0004] Currently, keypoint-based 2D image deformation schemes are commonly used to reshape portrait images. Specifically, for the portrait image to be processed, the contour points of the human body in the image are first detected using a contour point detection model; the position coordinates of each contour point after reshaping are obtained according to a preset reshaping rule; and a transformation matrix is ​​obtained based on the position coordinates of each contour point before and after reshaping; then, the human body in the portrait image is reshaped according to the transformation matrix to obtain the target image. However, this method of reshaping suffers from poor generalization. Summary of the Invention

[0005] This application provides an image processing method, an optical flow prediction model training method, and related apparatus for improving the generalization of image reshaping processing of human portrait images.

[0006] According to a first aspect of this application, an image processing method is provided, comprising:

[0007] Acquire the portrait image to be processed;

[0008] The image of the person to be processed is scaled up to obtain the image at the target resolution.

[0009] Based on the skeletal points and contour points of the target human body in the target resolution image, obtain the component affinity field (PAF) image corresponding to the target resolution image;

[0010] Optical flow prediction is performed on the target resolution image based on PAFs images and skeleton map to obtain the target optical flow map corresponding to the target resolution image. The skeleton map is obtained based on the skeleton points.

[0011] The target optical flow map is used to reshape the portrait image to be processed, and the reshaped target image corresponding to the portrait image to be processed is obtained.

[0012] According to a second aspect of this application, a method for training an optical flow prediction model is provided, comprising:

[0013] Obtain training sample pairs, which include portrait image samples and corresponding label images. The label images are reference images obtained after reshaping the portrait image samples.

[0014] The portrait image samples and label images are scaled down to obtain target resolution image samples and target resolution label images.

[0015] Based on the skeletal points and contour points of the human body sample in the target resolution image sample, obtain the component affinity field (PAF) image sample corresponding to the target resolution image sample;

[0016] The target resolution image sample, PAFs image sample and skeleton map sample are input into the optical flow prediction model to perform optical flow prediction, and the predicted optical flow map output by the optical flow prediction model is obtained. The skeleton map sample is obtained based on the skeleton points.

[0017] The predicted optical flow map is applied to the target resolution image sample to reshape the image sample, and the reshaped image sample corresponding to the target resolution image sample is obtained.

[0018] The parameters of the optical flow prediction model are adjusted based on the shaped image samples and the target resolution label image.

[0019] According to a third aspect of this application, an image processing apparatus is provided, comprising:

[0020] The first acquisition unit is used to acquire the portrait image to be processed;

[0021] The resolution processing unit is used to perform resolution scaling on the portrait image to be processed, so as to obtain a target resolution image.

[0022] The second acquisition unit is used to acquire the component affinity field (PAF) image corresponding to the target resolution image based on the skeletal points and contour points of the target human body in the target resolution image;

[0023] The optical flow prediction unit is used to predict the optical flow of the target resolution image based on the PAFs image and the skeleton map to obtain the target optical flow map corresponding to the target resolution image. The skeleton map is obtained based on the skeleton points.

[0024] The shaping processing unit is used to perform shaping processing on the portrait image to be processed based on the target optical flow map, so as to obtain the shaped target image corresponding to the portrait image to be processed.

[0025] According to a fourth aspect of this application, a training apparatus for an optical flow prediction model is provided, comprising:

[0026] The first acquisition unit is used to acquire training sample pairs, which include portrait image samples and label images corresponding to the portrait image samples. The label images are reference images obtained after the portrait image samples are shaped.

[0027] The resolution processing unit is used to perform resolution scaling on the portrait image sample and the label image respectively to obtain the target resolution image sample and the target resolution label image;

[0028] The second acquisition unit is used to acquire the component affinity field (PAF) image sample corresponding to the target resolution image sample based on the skeletal points and contour points of the human body sample in the target resolution image sample.

[0029] The optical flow prediction unit is used to input the target resolution image sample, PAF image sample and skeleton map sample into the optical flow prediction model to perform optical flow prediction and obtain the predicted optical flow map output by the optical flow prediction model. The skeleton map sample is obtained based on the skeleton points.

[0030] The shaping processing unit is used to apply the predicted optical flow map to shape the target resolution image sample to obtain the shaped image sample corresponding to the target resolution image sample.

[0031] The adjustment unit is used to adjust the parameters of the optical flow prediction model based on the shaped image samples and the target resolution label image.

[0032] According to a fifth aspect of this application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the image processing method of the first aspect or to perform the training method of the optical flow prediction model of the second aspect.

[0033] According to a sixth aspect of this application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the image processing method described in the first aspect or to perform the training method of the optical flow prediction model described in the second aspect.

[0034] According to a seventh aspect of this application, a computer program product is provided, the computer program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, the at least one processor executing the computer program causing the electronic device to perform the image processing method of the first aspect or the training method of the optical flow prediction model of the second aspect.

[0035] The technology of this application solves the problem of poor generalization in current methods of shaping human images. By scaling the resolution of the human image to be processed, a target resolution image is obtained. Based on the skeletal points and contour points of the target human body in the target resolution image, a corresponding PAF (Plan-Aspect-Oriented File) image is obtained. The obtained PAF image provides richer and more accurate information, accurately representing the position, contour, and body shape of various parts of the target human body. Therefore, optical flow prediction based on the PAF image and skeleton map can more accurately obtain the target optical flow map corresponding to the target resolution image. Based on the target optical flow map, the human image to be processed is then shaped to obtain the shaped target image. This allows for adaptive shaping to different degrees according to different postures and body shapes, achieving consistent results and demonstrating good generalization.

[0036] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0037] The accompanying drawings are provided for better understanding of the scheme and do not constitute a limitation of this application. Wherein:

[0038] Figure 1 is a schematic diagram of an application scenario to which this application applies;

[0039] Figure 2 is a schematic diagram according to the first embodiment of this application;

[0040] Figure 3 is a schematic diagram according to the second embodiment of this application;

[0041] Figure 4 is a schematic diagram according to the third embodiment of this application;

[0042] Figure 5 is a schematic diagram according to the fourth embodiment of this application;

[0043] Figure 6 is a schematic diagram according to the fifth embodiment of this application;

[0044] Figure 7 is a schematic diagram according to the sixth embodiment of this application;

[0045] Figure 8 is a schematic diagram according to the seventh embodiment of this application;

[0046] Figure 9 is a schematic diagram according to the eighth embodiment of this application;

[0047] Figure 10 is a schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of this application. Detailed Implementation

[0048] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0049] This application provides an image processing method, an optical flow prediction model training method, and related apparatus, which are applied to the fields of image processing, computer vision, deep learning, and data processing in the field of artificial intelligence, in order to improve the generalization of shaping processing of human portrait images.

[0050] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0051] AI adaptive reshaping technology for portrait images (such as portrait photos) has very important applications in fields such as portrait photography retouching and online social networking.

[0052] Currently, keypoint-based 2D image deformation schemes are commonly used to reshape portrait images. Specifically, for the portrait image to be processed, a contour point detection model is first used to detect the contour points of the human body in the image, resulting in, for example, 70 contour points used to mark the entire contour of the human body. For example, contour point 1 represents the contour point on the left side of the neck, contour point 2 represents the contour point on the right side of the neck, and so on. Based on a preset shaping rule (e.g., the left side of the neck shrinks inward by 30%), the position coordinates of each contour point after shaping are obtained. Based on the position coordinates of each contour point before and after shaping, a transformation matrix is ​​obtained. Then, based on the transformation matrix, the human body in the portrait image to be processed is shaped to obtain the target image. However, keypoint-based 2D image deformation schemes are only applicable to standard poses, thus exhibiting poor generalization.

[0053] In addition, optical flow-based schemes can be used to reshape portrait images. Optical flow is a concept in motion detection of objects within a visual field, used to describe the motion of an observed target, surface, or edge relative to the observer. In computer vision, optical flow refers to the motion vector field of pixels in an image, describing the displacement of each pixel between two frames. Optical flow-based schemes utilize human pose recognition models to detect human skeletal points, obtain skeletal structure information, extract depth features coupled with the skeletal structure from the input portrait image to be processed, predict the optical flow deformation field, and map it back to the portrait image to be processed. However, optical flow-based schemes have poor adaptability to different body shapes.

[0054] In summary, due to the complex structure and posture of the human body, as well as the diverse clothing and scenes of human figures, current image processing methods cannot achieve good shaping effects.

[0055] To address the aforementioned issues, this application provides an image processing method that performs resolution scaling on the human portrait image to be processed, obtaining a target resolution image. Based on the skeletal points and contour points of the target human body in the target resolution image, a part affinity field (PAF) image corresponding to the target resolution image is obtained, which can accurately express the position, contour, and body shape information of various parts of the target human body, thereby enabling a more accurate acquisition of the target optical flow map corresponding to the target resolution image. Furthermore, based on the target optical flow map, the human portrait image to be processed is sculpted to obtain a sculpted target image corresponding to the human portrait image to be processed. This method achieves adaptive sculpting to different degrees according to different postures and body shapes of the human body, exhibiting good generalization ability.

[0056] Figure 1 is a schematic diagram of an application scenario to which this application applies. In this application scenario, the user uploads a portrait image to be processed to the server 102 through the client 101. After obtaining the portrait image to be processed, the server 102 performs shaping processing on the portrait image to be processed according to the image processing method provided in this application to obtain the shaped target image corresponding to the portrait image to be processed.

[0057] It should be noted that Figure 1 is only a schematic diagram of an application scenario provided by an embodiment of this application. This embodiment of the application does not limit the devices included in Figure 1, nor does it limit the positional relationship between the devices in Figure 1.

[0058] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0059] Figure 2 is a schematic diagram according to the first embodiment of this application. As shown in Figure 2, the image processing method provided in the first embodiment of this application can be applied to an electronic device, which may be a server or a server cluster, etc. The image processing method provided in the first embodiment of this application includes:

[0060] S201. Obtain the portrait image to be processed.

[0061] In this embodiment, the image to be processed may be input by a user to an electronic device executing this method embodiment, or it may be sent by another device to the electronic device executing this method embodiment. The image to be processed may be, for example, a 2D portrait photograph containing a target human body, and it may be a high-resolution image.

[0062] S202. Perform resolution scaling on the portrait image to be processed to obtain the target resolution image.

[0063] For example, the portrait image to be processed is a high-resolution image. The resolution of the portrait image to be processed is reduced to obtain a target resolution image. The resolution of the target resolution image is lower than that of the portrait image to be processed; for example, the resolution of the target resolution image is 384*384. It can be understood that this step is used to uniformly scale the resolution of the portrait image to be processed in order to obtain a target resolution image of the same resolution.

[0064] S203. Based on the skeletal points and contour points of the target human body in the target resolution image, obtain the PAFs image corresponding to the target resolution image.

[0065] For example, the skeletal points and contour points of the target human body in the target resolution image are obtained through a human keypoint detection model. The PAFs image corresponding to the target resolution image can be obtained by connecting adjacent skeletal points and contour points of the target human body.

[0066] It is understandable that PAFs (Programmable Frame Images) are used to help identify relationships between key points in the human body. Compared with current related technologies, PAFs in related technologies are obtained by connecting skeletal points and expanding these connections according to the size of the bounding boxes of all skeletal points. This aims to express the position and body shape information of various parts of the human body (such as shoulders, torso, waist, arms, and legs). However, the width information of the connections after the expansion operation is insufficient to accurately express the body shape information of various parts of the human body, resulting in inaccurate optical flow maps obtained from PAFs for shaping processing. In contrast, the PAFs images in this application are obtained based on the skeletal points and contour points of the target human body in the target resolution image. The obtained PAFs images contain richer and more accurate information, accurately expressing the position, contour, and body shape information of various parts of the target human body, which helps to make optical flow prediction more accurate.

[0067] S204. Based on PAFs images and skeleton maps, perform optical flow prediction on the target resolution image to obtain the target optical flow map corresponding to the target resolution image. The skeleton map is obtained based on the skeleton points.

[0068] For example, the skeleton map is obtained by connecting the skeletal points of the target human body in the target resolution image. After acquiring the PAFs image, optical flow prediction can be performed on the target resolution image based on the PAFs image and the skeleton map to obtain the target optical flow map corresponding to the target resolution image. For example, the target resolution image, PAFs image, and skeleton map are input into the optical flow prediction model to obtain the target optical flow map output by the optical flow prediction model. Since the PAFs image provides richer and more accurate information, and can accurately express the position, contour, and body shape information of various parts of the target human body, the target optical flow map obtained in this embodiment is more accurate.

[0069] S205. Based on the target optical flow map, perform shaping processing on the portrait image to be processed to obtain the shaped target image corresponding to the portrait image to be processed.

[0070] For example, after obtaining the target optical flow map corresponding to the target resolution image, the resolution of the target optical flow map is the same as that of the target resolution image, both being low-resolution images. Therefore, the target optical flow map can be scaled to obtain a resolution-scaled optical flow map, whose resolution is the same as that of the portrait image to be processed. For instance, the resolution-scaled optical flow map can be applied to the portrait image to be processed using the warp function to achieve a shaping process, thereby obtaining the shaped target image corresponding to the portrait image to be processed. Here, the warp function is a transformation function used to perform geometric transformations on an image to change its shape or viewpoint.

[0071] After obtaining the shaped target image corresponding to the portrait image to be processed, the target image can be output, such as displaying the target image to the user for easy viewing, or performing relevant processing on the target image.

[0072] In this embodiment, the resolution of the human image to be processed is scaled to obtain a target resolution image. Based on the skeletal points and contour points of the target human body in the target resolution image, a PAFs image corresponding to the target resolution image is obtained. The obtained PAFs image has richer and more accurate information, and can accurately express the position, contour, and body shape of various parts of the target human body. Therefore, optical flow prediction is performed on the target resolution image based on the PAFs image and the skeleton map, which can more accurately obtain the target optical flow map corresponding to the target resolution image. Based on the target optical flow map, the human image to be processed is shaped to obtain the shaped target image corresponding to the human image to be processed. This achieves adaptive shaping to different degrees according to different human postures and body shape, achieving a consistent effect and having good generalization.

[0073] Figure 3 is a schematic diagram according to the second embodiment of this application. Based on the above embodiments, this application further describes the image processing method. As shown in Figure 3, the image processing method provided by the second embodiment of this application may include:

[0074] S301. Obtain the portrait image to be processed.

[0075] The implementation principle and technical effect of S301 can be referred to the embodiment of step S201 in Figure 2 above, and will not be repeated here.

[0076] S302. Perform resolution scaling on the portrait image to be processed to obtain the target resolution image.

[0077] The implementation principle and technical effect of S302 can be referred to the embodiment of step S202 in Figure 2 above. In this embodiment, the resolution of the target resolution image is the resolution of the input image corresponding to the optical flow prediction model. For example, the resolution of the target resolution image is 384*384. By reducing the resolution, the computing power requirement of the optical flow prediction model can be reduced. While achieving the same effect, the performance in terms of memory usage and time consumption can be optimized.

[0078] S303. Input the target resolution image into the human key point detection model to perform human key point detection, and obtain the skeletal points and contour points of the target human body output by the human key point detection model.

[0079] In this step, the human keypoint detection model can output not only human skeletal points but also human contour points. For details on how to obtain the human keypoint detection model, please refer to current related technologies. For example, by inputting a target resolution image into the human keypoint detection model for human keypoint detection, multiple skeletal points and multiple contour points of the target human body in the target resolution image can be obtained.

[0080] In this embodiment of the application, step S203 in Figure 2 may further include the following two steps S304 and S305:

[0081] S304. Based on the preset parts of the target human body, connect the adjacent bone points and contour points to obtain multiple contour lines.

[0082] For example, the preset parts of the target human body may include the shoulders, torso, waist, arms, and legs. For each part, adjacent skeletal points and contour points are connected to obtain the contour line corresponding to each part, thereby obtaining multiple contour lines. For example, Figure 4 is a schematic diagram according to the third embodiment of this application. As shown in Figure 4, for the same target human body, the current related technology uses the method of connecting skeletal points to obtain skeletal point connection diagram 401, which mainly includes the shoulders, torso, waist, arms, and legs of the target human body; while the embodiment of this application uses the method of connecting skeletal points and contour points together to obtain the contour lines corresponding to each part of the target human body, that is, to obtain contour line diagram 402.

[0083] Optionally, each contour line can be a different color, with different colors used to represent different parts of the contour. This allows the optical flow prediction model to determine the corresponding part based on the color of each contour line.

[0084] S305. Based on multiple contour lines, obtain the PAFs image corresponding to the target resolution image.

[0085] In this step, after obtaining multiple contour lines corresponding to various preset parts of the target human body, a PAF image corresponding to the target resolution image can be obtained based on these contour lines. Compared with current related technologies, the PAF images in related technologies are obtained by connecting skeletal points and expanding the skeletal point connections according to the size of the bounding boxes of all skeletal points, aiming to express the position and body shape information of various parts of the human body (such as shoulders, torso, waist, arms, and legs). However, the width information of the connections after the expansion operation is insufficient to accurately express the body shape information of various parts of the human body, resulting in inaccurate optical flow maps obtained from the PAF images for shaping processing. In contrast, the PAF images in this embodiment are obtained by connecting the skeletal points and contour points of the human body, which can output PAF images with richer and more accurate information, thereby accurately expressing the position, contour, and body shape information of each part, and making the optical flow map output by the optical flow prediction model more accurate.

[0086] In this embodiment of the application, step S204 in FIG2 may further include the following step S306:

[0087] S306. Input the target resolution image, PAFs image and skeleton map into the optical flow prediction model. The optical flow prediction model performs optical flow prediction on the target resolution image based on the PAFs image and skeleton map to obtain the target optical flow map. The skeleton map is obtained based on the skeleton points.

[0088] Among them, the optical flow prediction model is trained based on human image samples, the predicted optical flow map corresponding to the human image samples and the label image. The predicted optical flow map is obtained based on PAFs image samples, and the PAFs image samples are obtained based on the skeletal points and contour points of human body samples in human image samples.

[0089] For example, the optical flow prediction model is trained based on portrait image samples, corresponding predicted optical flow maps, and label images. For details on how the optical flow prediction model is trained, please refer to subsequent embodiments. Inputting the target resolution image, PAFs image, and skeleton map into the optical flow prediction model yields the target optical flow map output by the model. The resolution of the target optical flow map is the same as that of the target resolution image; both are low-resolution images.

[0090] In this embodiment of the application, step S205 in Figure 2 may further include the following three steps S307 to S309:

[0091] S307. Smooth the target optical flow map to obtain the smoothed optical flow map.

[0092] For example, after obtaining the target optical flow map, the target optical flow map can be Gaussian smoothed to make the shaping effect map after optical flow more smooth and uniform, thus obtaining the smoothed optical flow map.

[0093] S308. Perform resolution scaling on the smoothed optical flow map to obtain a resolution-scaled optical flow map. The resolution of the resolution-scaled optical flow map is the same as the resolution of the portrait image to be processed.

[0094] In this step, after obtaining the smoothed optical flow map, a resolution scaling process can be performed on the smoothed optical flow map to obtain a resolution-scaled optical flow map. The resolution of the resolution-scaled optical flow map is the same as the resolution of the portrait image to be processed. For example, if the smoothed optical flow map is a low-resolution image, a resolution upscaling process can be performed on the smoothed optical flow map to obtain a high-resolution resolution-scaled optical flow map with the same resolution as the portrait image to be processed.

[0095] S309. Apply the resolution scaling optical flow map to reshape the portrait image to be processed, and obtain the reshaped target image corresponding to the portrait image to be processed.

[0096] For example, a resolution-scaled optical flow map can be applied to the portrait image to be processed using a warp function to reshape the image, thereby obtaining the reshaped target image corresponding to the portrait image to be processed. After obtaining the reshaped target image corresponding to the portrait image to be processed, the target image can be output.

[0097] In this embodiment, the image of the human figure to be processed is scaled down to obtain a target resolution image, which can meet the low resolution requirement of the optical flow prediction model. The target resolution image is input into the human key point detection model to detect human key points, and the skeletal points and contour points of the target human figure in the target resolution image are obtained. According to each preset part of the target human figure, adjacent skeletal points and contour points are connected to obtain multiple contour lines. Based on the multiple contour lines, the PAFs image corresponding to the target resolution image is obtained. The obtained PAFs image has richer and more accurate information and can accurately express the position, contour and body shape information of each preset part of the target human figure. Thus, the target resolution image, PAFs image and skeleton image are input into the optical flow prediction model, so that the optical flow prediction model outputs the target optical flow map more accurately. The target optical flow map is smoothed to obtain a smoother and more uniform smoothed optical flow map. The smoothed optical flow map is scaled down to obtain a scaled optical flow map with the same resolution as the image of the human figure to be processed. Then, the scaled optical flow map is applied to the image of the human figure to be processed to reshape it, and the reshaped target image corresponding to the image of the human figure to be processed is obtained. The embodiments of this application can adaptively shape the body to different degrees according to different postures and body sizes, achieving a consistent effect and demonstrating good generalization ability.

[0098] Based on the above embodiments, considering the case where the portrait image to be processed contains multiple target human bodies, the image processing method provided in this application embodiment may further include: for each target human body among the multiple target human bodies, obtaining the target optical flow map corresponding to the target human body; performing weighted summation processing on the multiple target optical flow maps to obtain a weighted optical flow map; and performing shaping processing on the portrait image to be processed based on the weighted optical flow map to obtain the shaped target image corresponding to the portrait image to be processed.

[0099] For example, when the portrait image to be processed contains multiple target bodies, a transformation weight corresponding to each target body can be pre-set, such as a user-configured weight. For instance, if the portrait image contains two target bodies, target body A and target body B, the user might configure the transformation weight for target body A as 50% and the transformation weight for target body B as 80%. Using the method described above, target optical flow map A corresponding to target body A and target optical flow map B corresponding to target body B are obtained. The two target optical flow maps are then weighted and summed, i.e., target optical flow map A and target optical flow map B are multiplied by their respective transformation weights and summed to obtain the weighted optical flow map for the entire image. Based on the weighted optical flow map, the portrait image to be processed is then shaped to obtain the shaped target image.

[0100] Based on the above embodiments, Figure 5 is a schematic diagram according to the fourth embodiment of this application. As shown in Figure 5, the training method of the optical flow prediction model provided in the fourth embodiment of this application can be applied to electronic devices, such as servers or server clusters. The training method of the optical flow prediction model provided in the fourth embodiment of this application includes:

[0101] S501. Obtain training sample pairs. The training sample pairs include portrait image samples and corresponding label images. The label images are reference images obtained after reshaping the portrait image samples.

[0102] In this embodiment, the portrait image sample is a high-resolution portrait image to be shaped; the label image is a reference image obtained after shaped processing of the portrait image sample. Multiple training sample pairs can be obtained based on the portrait image sample and its corresponding label image.

[0103] S502. Perform resolution scaling on the portrait image sample and the label image respectively to obtain the target resolution image sample and the target resolution label image.

[0104] For example, the portrait image sample and the label image are respectively downscaled to obtain the target resolution image sample and the target resolution label image. The resolution of the target resolution image sample and the target resolution label image is, for example, 384*384, which can meet the resolution requirements of the optical flow prediction model for the input image. It can be understood that by downscaling, the computational power requirement of the optical flow prediction model can be reduced, and the performance in terms of memory usage and time consumption can be optimized while achieving the same effect.

[0105] S503. Based on the skeletal points and contour points of the human body sample in the target resolution image sample, obtain the PAFs image sample corresponding to the target resolution image sample.

[0106] In this step, the skeletal points and contour points of the human body sample in the target resolution image sample can be combined to obtain the PAFs image sample corresponding to the target resolution image sample.

[0107] Optionally, before obtaining the PAFs image sample corresponding to the target resolution image sample based on the skeletal points and contour points of the human body sample in the target resolution image sample, the training method of the optical flow prediction model provided in this application embodiment may further include: inputting the target resolution image sample into the human body key point detection model to perform human body key point detection, and obtaining the skeletal points and contour points of the human body sample output by the human body key point detection model.

[0108] For example, the human keypoint detection model can output not only human skeletal points but also human contour points. For details on how to obtain the human keypoint detection model, please refer to current related technologies. For example, by inputting a target resolution image sample into the human keypoint detection model for human keypoint detection, the skeletal points and contour points of the human sample output by the human keypoint detection model can be obtained.

[0109] Further, optionally, obtaining PAFs image samples corresponding to the target resolution image sample based on the skeletal points and contour points of the human body sample in the target resolution image sample may include: connecting adjacent skeletal points and contour points according to each preset part of the human body sample to obtain multiple contour lines, and obtaining PAFs image samples based on the multiple contour lines.

[0110] For example, the preset parts of the human body sample may include the shoulders, torso, waist, arms, and legs of the target human body. For each part, adjacent bone points and contour points can be connected to obtain multiple contour lines, thereby obtaining PAFs image samples corresponding to the target resolution image sample based on the multiple contour lines. Optionally, each contour line has a different color, and different colored contour lines are used to represent the contours of different parts, so that the optical flow prediction model can determine the part corresponding to each contour line based on the color.

[0111] S504. Input the target resolution image sample, PAFs image sample and skeleton map sample into the optical flow prediction model to perform optical flow prediction, and obtain the predicted optical flow map output by the optical flow prediction model. The skeleton map sample is obtained based on the skeleton points.

[0112] In this step, the skeleton map sample is obtained by connecting the skeletal points of the human body sample in the target resolution image sample. After obtaining the PAFs image samples, the target resolution image sample, PAFs image samples, and skeleton map samples can be input into the optical flow prediction model to perform optical flow prediction, resulting in the predicted optical flow map output by the optical flow prediction model. The resolution of the predicted optical flow map is the same as that of the target resolution image sample, both being low-resolution images.

[0113] S505. Apply the predicted optical flow map to reshape the target resolution image sample to obtain the reshaped image sample corresponding to the target resolution image sample.

[0114] For example, the predicted optical flow map can be applied to the target resolution image sample through a warp function to achieve the shaping process of the target resolution image sample, thereby obtaining the shaped image sample corresponding to the target resolution image sample.

[0115] S506. Based on the shaped image samples and the target resolution label image, adjust the parameters of the optical flow prediction model.

[0116] In this step, after obtaining the shaped image sample corresponding to the target resolution image sample, the shaped image sample and the target resolution label image can be compared to obtain the loss value. Based on the loss value, the parameters of the optical flow prediction model are adjusted to obtain the trained optical flow prediction model.

[0117] In this embodiment, resolution scaling is applied to both the human image sample and the label image to obtain target resolution image samples and target resolution label images. This reduces the computational requirements of the optical flow prediction model, optimizing performance in terms of memory usage and time consumption while achieving the same effect. Based on the skeletal points and contour points of the human sample in the target resolution image sample, corresponding PAFs image samples are obtained, accurately representing the position, contour, and body shape information of various parts of the target human body. Inputting the target resolution image sample, PAFs image samples, and skeleton map samples into the optical flow prediction model for optical flow prediction yields a more accurate predicted optical flow map. The predicted optical flow map is then used to reshape the target resolution image sample, obtaining a reshaped image sample. Based on the reshaped image sample and the target resolution label image, the parameters of the optical flow prediction model are adjusted. The trained optical flow prediction model can output predicted optical flow maps more accurately based on different human postures and body shapes, exhibiting good generalization ability.

[0118] Based on the embodiment shown in Figure 5, Figure 6 is a schematic diagram according to the fifth embodiment of this application. As shown in Figure 6, the optical flow prediction model provided in the fifth embodiment of this application includes an encoder, an attention module, and a decoder. The encoder contains the same number of modules and network layers as the first number of modules, and the decoder contains the same number of modules and network layers as the second number of network layers. Specifically, the encoder encodes the image obtained by combining the target resolution image samples and skeleton map samples through channel processing to obtain a first implicit feature map; the attention module extracts attention information based on the first implicit feature map and PAFs image samples to obtain a second implicit feature map; and the decoder decodes the second implicit feature map to obtain a predicted optical flow map.

[0119] For example, the network structure of the optical flow prediction model in this application embodiment is simpler. For instance, the encoder has 3 modules (i.e., the first number of modules) and 6 network layers (i.e., the first number of network layers); the decoder has 3 modules (i.e., the second number of modules) and 6 network layers (i.e., the second number of network layers). By simplifying the network structure of the optical flow prediction model, the performance of the optical flow prediction model can be further optimized while improving the adaptability and robustness of the effect. For the target resolution image sample, PAFs image sample, and skeleton map sample input to the optical flow prediction model, the encoder can encode the image obtained by channel combination of the target resolution image sample and skeleton map sample to obtain a first implicit feature map. The attention module extracts attention information based on the first implicit feature map and PAFs image sample to obtain a second implicit feature map, and then the decoder decodes the second implicit feature map to obtain the predicted optical flow map.

[0120] Based on the above embodiments, Figure 7 is a schematic diagram according to the sixth embodiment of this application. As shown in Figure 7, a high-resolution human portrait image to be processed is subjected to resolution scaling (downsampling) to obtain a target resolution image, which is a low-resolution image (e.g., 384*384). The target resolution image is input into a human keypoint detection model to detect human keypoints, obtaining the skeletal points and contour points of the target human body in the target resolution image. A skeleton map is obtained based on the skeletal points; a PAFs image corresponding to the target resolution image is obtained based on the skeletal points and contour points. The target resolution image and the skeleton map are concatted to obtain a combined image. The encoder of the optical flow prediction model encodes the image to obtain a first implicit feature map. The attention module of the optical flow prediction model extracts attention information based on the first implicit feature map and the PAFs image to obtain a second implicit feature map. The decoder of the optical flow prediction model decodes the second implicit feature map to obtain a predicted optical flow map, which is a low-resolution optical flow map. The predicted optical flow map is upsampled (resolution scaling) to obtain a high-resolution optical flow map with the same resolution as the subject image. The subject image and the high-resolution optical flow map are then input into a warp function for shaping, resulting in the shaped target image corresponding to the subject image.

[0121] The embodiments of this application can achieve better portrait image shaping effects, especially for portrait images of people with different body shapes, it can adaptively perform different degrees of shaping to achieve a consistent effect; at the same time, it can optimize performance in terms of memory usage and time consumption, such as saving the computing and memory overhead of the computer's central processing unit (CPU) / graphics processing unit (GPU), and realize local deployment on the client.

[0122] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0123] Figure 8 is a schematic diagram according to the seventh embodiment of this application. As shown in Figure 8, the image processing apparatus 800 provided in the seventh embodiment of this application includes: a first acquisition unit 801, a resolution processing unit 802, a second acquisition unit 803, an optical flow prediction unit 804, and a shaping processing unit 805. Wherein:

[0124] The first acquisition unit 801 is used to acquire the portrait image to be processed.

[0125] The resolution processing unit 802 is used to perform resolution scaling processing on the portrait image to be processed to obtain a target resolution image.

[0126] The second acquisition unit 803 is used to acquire the PAFs image corresponding to the target resolution image based on the skeletal points and contour points of the target human body in the target resolution image.

[0127] The optical flow prediction unit 804 is used to predict the optical flow of the target resolution image based on the PAFs image and the skeleton map to obtain the target optical flow map corresponding to the target resolution image. The skeleton map is obtained based on the skeleton points.

[0128] The shaping processing unit 805 is used to perform shaping processing on the portrait image to be processed based on the target optical flow map, so as to obtain the shaped target image corresponding to the portrait image to be processed.

[0129] In some embodiments, the second acquisition unit 803 includes: a line processing module (not shown in the figure), used to connect adjacent bone points and contour points according to various preset parts of the target human body to obtain multiple contour lines; and an image acquisition module (not shown in the figure), used to obtain PAFs images based on the multiple contour lines.

[0130] In some embodiments, the optical flow prediction unit 804 includes: an optical flow prediction module (not shown in the figure), used to input a target resolution image, a PAF image, and a skeleton map into an optical flow prediction model, wherein the optical flow prediction model performs optical flow prediction on the target resolution image based on the PAF image and the skeleton map to obtain a target optical flow map; the optical flow prediction model is obtained by training based on human image samples, the predicted optical flow map corresponding to the human image samples, and a label image, wherein the predicted optical flow map is obtained based on PAF image samples, and the PAF image samples are obtained based on the skeleton points and contour points of human body samples in the human image samples.

[0131] In some embodiments, the shaping processing unit 805 includes: a smoothing processing module (not shown in the figure) for smoothing the target optical flow map to obtain a smoothed optical flow map; a resolution processing module (not shown in the figure) for performing resolution scaling processing on the smoothed optical flow map to obtain a resolution-scaled optical flow map, wherein the resolution of the resolution-scaled optical flow map is the same as the resolution of the portrait image to be processed; and a first shaping processing module (not shown in the figure) for applying the resolution-scaled optical flow map to shape the portrait image to be processed to obtain a shaped target image corresponding to the portrait image to be processed.

[0132] In some embodiments, the portrait image to be processed contains multiple target human bodies, and the shaping processing unit 805 further includes: an optical flow map acquisition module (not shown in the figure), used to acquire a target optical flow map corresponding to each of the multiple target human bodies; a weighted processing module (not shown in the figure), used to perform weighted summation processing on the multiple target optical flow maps to obtain a weighted optical flow map; and a second shaping processing module (not shown in the figure), used to perform shaping processing on the portrait image to be processed based on the weighted optical flow map to obtain a shaped target image corresponding to the portrait image to be processed.

[0133] In some embodiments, the second acquisition unit 803 further includes: a key point detection module (not shown in the figure), which is used to input the target resolution image into the human key point detection model to perform human key point detection before acquiring the PAFs image corresponding to the target resolution image based on the skeletal points and contour points of the target human body in the target resolution image, so as to obtain the skeletal points and contour points of the target human body output by the human key point detection model.

[0134] The image processing apparatus shown in Figure 8 can execute the steps in the method embodiment corresponding to the above image processing method. Its implementation principle and technical effect are similar, and will not be described again here.

[0135] Figure 9 is a schematic diagram according to the eighth embodiment of this application. As shown in Figure 9, the training device 900 for the optical flow prediction model provided in the eighth embodiment of this application includes: a first acquisition unit 901, a resolution processing unit 902, a second acquisition unit 903, an optical flow prediction unit 904, a shaping processing unit 905, and an adjustment unit 906. Wherein:

[0136] The first acquisition unit 901 is used to acquire training sample pairs, which include portrait image samples and label images corresponding to the portrait image samples. The label images are reference images obtained after the portrait image samples have been shaped.

[0137] The resolution processing unit 902 is used to perform resolution scaling processing on the portrait image sample and the label image respectively to obtain the target resolution image sample and the target resolution label image.

[0138] The second acquisition unit 903 is used to acquire PAFs image samples corresponding to the target resolution image sample based on the skeletal points and contour points of the human body sample in the target resolution image sample.

[0139] The optical flow prediction unit 904 is used to input the target resolution image sample, PAFs image sample and skeleton map sample into the optical flow prediction model to perform optical flow prediction and obtain the predicted optical flow map output by the optical flow prediction model. The skeleton map sample is obtained based on the skeleton points.

[0140] The shaping processing unit 905 is used to apply the predicted optical flow map to shape the target resolution image sample to obtain the shaped image sample corresponding to the target resolution image sample.

[0141] The adjustment unit 906 is used to adjust the parameters of the optical flow prediction model based on the shaped image samples and the target resolution label image.

[0142] In some embodiments, the second acquisition unit 903 includes: a connection processing module (not shown in the figure), used to connect adjacent bone points and contour points according to each preset part of the human body sample to obtain multiple contour lines; and an image sample acquisition module (not shown in the figure), used to obtain PAFs image samples according to the multiple contour lines.

[0143] In some embodiments, the second acquisition unit 903 further includes: a key point detection module (not shown in the figure), which is used to input the target resolution image sample into the human key point detection model to perform human key point detection before acquiring the PAFs image sample corresponding to the target resolution image sample based on the skeletal points and contour points of the human sample in the target resolution image sample, so as to obtain the skeletal points and contour points of the human sample output by the human key point detection model.

[0144] In some embodiments, the optical flow prediction model includes an encoder, an attention module, and a decoder. The encoder contains the same number of modules and network layers as a first number, and the decoder contains the same number of modules and network layers as a second number. The encoder encodes an image obtained by combining target resolution image samples and skeleton map samples through channel processing to obtain a first implicit feature map. The attention module extracts attention information based on the first implicit feature map and PAFs image samples to obtain a second implicit feature map. The decoder decodes the second implicit feature map to obtain a predicted optical flow map.

[0145] The training device for the optical flow prediction model shown in Figure 9 can execute the steps in the method embodiment corresponding to the above-described optical flow prediction model training method. Its implementation principle and technical effect are similar, and will not be repeated here.

[0146] According to embodiments of this application, this application also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the scheme provided in any of the above embodiments.

[0147] According to embodiments of this application, this application also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the scheme provided in any of the above embodiments.

[0148] According to an embodiment of this application, this application also provides a computer program product, which includes: a computer program stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the solution provided in any of the above embodiments.

[0149] Figure 10 is a schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0150] As shown in Figure 10, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) (Figure 10 uses ROM 1002 as an example) or a computer program loaded from a storage unit 1008 into a random access memory (RAM) (Figure 10 uses RAM 1003 as an example). The RAM 1003 can also store various programs and data required for the operation of the electronic device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface (Figure 10 uses I / O interface 1005 as an example) is also connected to the bus 1004.

[0151] Multiple components in electronic device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of displays, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0152] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit, a graphics processing unit, various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as image processing methods and optical flow prediction model training methods. For example, in some embodiments, the image processing methods and optical flow prediction model training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the image processing methods and optical flow prediction model training methods described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured by any other suitable means (e.g., by means of firmware) to perform image processing methods and training methods for optical flow prediction models.

[0153] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0154] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0155] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0156] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0157] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0158] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0159] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0160] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, integrations, sub-integrations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image processing method, comprising: Acquire the portrait image to be processed; The image of the person to be processed is subjected to resolution scaling to obtain a target resolution image; Based on the skeletal points and contour points of the target human body in the target resolution image, obtain the component affinity field (PAF) image corresponding to the target resolution image; Optical flow prediction is performed on the target resolution image based on the PAFs image and the skeleton map to obtain the target optical flow map corresponding to the target resolution image. The skeleton map is obtained based on the skeleton points. The target optical flow map is used to reshape the portrait image to be processed, thereby obtaining the reshaped target image corresponding to the portrait image to be processed.

2. The image processing method according to claim 1, wherein, The step of obtaining the component affinity field (PAF) image corresponding to the target resolution image based on the skeletal points and contour points of the target human body in the target resolution image includes: Based on the various preset parts of the target human body, adjacent bone points and contour points are connected to obtain multiple contour lines. The PAFs image is obtained based on the multiple contour lines.

3. The image processing method of claim 1, wherein, The step of performing optical flow prediction on the target resolution image based on the PAFs image and skeleton map to obtain the target optical flow map corresponding to the target resolution image includes: The target resolution image, the PAFs image, and the skeleton map are input into the optical flow prediction model. The optical flow prediction model performs optical flow prediction on the target resolution image based on the PAFs image and the skeleton map to obtain the target optical flow map. The optical flow prediction model is trained based on human image samples, the predicted optical flow map corresponding to the human image samples, and the label image. The predicted optical flow map is obtained based on PAFs image samples, which are obtained based on the skeleton points and contour points of the human body samples in the human image samples.

4. The image processing method of claim 1, wherein, The step of shaping the portrait image to be processed based on the target optical flow map to obtain the shaped target image corresponding to the portrait image to be processed includes: The target optical flow map is smoothed to obtain a smoothed optical flow map; The smoothed optical flow map is subjected to resolution scaling to obtain a resolution-scaled optical flow map, the resolution of which is the same as the resolution of the portrait image to be processed. The resolution-scaled optical flow map is applied to reshape the portrait image to be processed, thereby obtaining the reshaped target image corresponding to the portrait image to be processed.

5. The image processing method according to any one of claims 1 to 4, wherein, The image of the human figure to be processed contains multiple target human figures, and the method further includes: For each of the plurality of target human bodies, obtain the target optical flow map corresponding to the target human body; The optical flow maps of the multiple targets are weighted and summed to obtain the weighted optical flow map. The image to be processed is shaped based on the weighted optical flow map to obtain the shaped target image corresponding to the image to be processed.

6. The image processing method according to any one of claims 1 to 4, wherein, Before obtaining the component affinity field (PAF) image corresponding to the target resolution image based on the skeletal points and contour points of the target human body in the target resolution image, the method further includes: The target resolution image is input into the human key point detection model to detect human key points, and the skeletal points and contour points of the target human body are obtained from the output of the human key point detection model.

7. A training method for an optical flow prediction model, comprising: Obtain training sample pairs, wherein the training sample pairs include portrait image samples and label images corresponding to the portrait image samples, and the label images are reference images obtained after the portrait image samples are shaped; The portrait image sample and the label image are respectively subjected to resolution scaling processing to obtain a target resolution image sample and a target resolution label image; Based on the skeletal points and contour points of the human body sample in the target resolution image sample, obtain the component affinity field (PAF) image sample corresponding to the target resolution image sample; The target resolution image sample, the PAFs image sample, and the skeleton map sample are input into the optical flow prediction model to perform optical flow prediction, and the predicted optical flow map output by the optical flow prediction model is obtained. The skeleton map sample is obtained based on the skeleton points. The predicted optical flow map is applied to the target resolution image sample to perform a reshaping process, thereby obtaining the reshaped image sample corresponding to the target resolution image sample; Based on the shaped image samples and the target resolution label image, the parameters of the optical flow prediction model are adjusted.

8. The training method of claim 7, wherein, The step of obtaining the component affinity field (PAF) image sample corresponding to the target resolution image sample based on the skeletal points and contour points of the human body sample in the target resolution image sample includes: Based on the various preset parts of the human body sample, adjacent bone points and contour points are connected to obtain multiple contour lines. Based on the multiple contour lines, the PAFs image samples are obtained.

9. The training method of claim 7 or 8, wherein, Before obtaining the component affinity field (PAF) image sample corresponding to the target resolution image sample based on the skeletal points and contour points of the human body sample in the target resolution image sample, the method further includes: The target resolution image sample is input into the human key point detection model to detect human key points, and the skeletal points and contour points of the human sample are output by the human key point detection model.

10. The training method according to claim 7 or 8, wherein, The optical flow prediction model includes an encoder, an attention module, and a decoder. The encoder contains a first number of modules and a first number of network layers, and the decoder contains a second number of modules and a second number of network layers. The encoder encodes the image obtained by combining the target resolution image sample and the skeleton map sample through channel processing to obtain a first implicit feature map. The attention module extracts attention information based on the first implicit feature map and the PAFs image sample to obtain a second implicit feature map. The decoder decodes the second implicit feature map to obtain the predicted optical flow map.

11. An image processing apparatus, comprising: The first acquisition unit is used to acquire the portrait image to be processed; A resolution processing unit is used to perform resolution scaling on the portrait image to be processed to obtain a target resolution image; The second acquisition unit is used to acquire the component affinity field (PAF) image corresponding to the target resolution image based on the skeletal points and contour points of the target human body in the target resolution image; An optical flow prediction unit is used to perform optical flow prediction on the target resolution image based on the PAFs image and the skeleton map to obtain a target optical flow map corresponding to the target resolution image, wherein the skeleton map is obtained based on the skeleton points; A shaping processing unit is used to perform shaping processing on the portrait image to be processed based on the target optical flow map, so as to obtain the shaped target image corresponding to the portrait image to be processed.

12. A training device for an optical flow prediction model, comprising: The first acquisition unit is used to acquire training sample pairs, the training sample pairs including portrait image samples and label images corresponding to the portrait image samples, the label images being reference images obtained after shaping the portrait image samples; A resolution processing unit is used to perform resolution scaling processing on the portrait image sample and the label image respectively to obtain a target resolution image sample and a target resolution label image; The second acquisition unit is used to acquire the component affinity field (PAF) image sample corresponding to the target resolution image sample based on the skeletal points and contour points of the human body sample in the target resolution image sample. An optical flow prediction unit is used to input the target resolution image sample, the PAFs image sample, and the skeleton map sample into an optical flow prediction model to perform optical flow prediction, and obtain a predicted optical flow map output by the optical flow prediction model. The skeleton map sample is obtained based on the skeleton points. A shaping processing unit is used to apply the predicted optical flow map to shape the target resolution image sample to obtain a shaped image sample corresponding to the target resolution image sample. The adjustment unit is used to adjust the parameters of the optical flow prediction model based on the shaped image sample and the target resolution label image.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the image processing method according to any one of claims 1 to 6 or the training method for the optical flow prediction model according to any one of claims 7 to 10.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the image processing method according to any one of claims 1 to 6 or the training method of the optical flow prediction model according to any one of claims 7 to 10.

15. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the image processing method according to any one of claims 1 to 6 or the steps of the training method for the optical flow prediction model according to any one of claims 7 to 10.

Images