A device and method for image processing

Abstract

A method of generating a data set for training an AI model to generate caricatures by interactive editing of an input reference image of a subject. The method comprises processing a facial image to identify facial landmarks and defining the landmarks as a plurality of source points; defining a plurality of target points corresponding to a location of the source points after deformation of the facial image; relating the source points to landmark information comprising a text-based description of the facial feature in which the source points are located; and relating the target points to distorted landmark information comprising a text-based description of a distorted facial feature in which the source points were located. There is also provided a method of training an AI model to generate caricatures, a method of implementing an AI model trained to generate caricatures, and a device configured to operate to implement the method.

Description

[0001] A DEVICE AND METHOD FOR IMAGE PROCESSING

[0002] FIELD OF THE INVENTION

[0003] This invention relates to generating caricatures from facial images. The caricatures may be two-dimensional and realistic in style, cartoon-like in style, or three-dimensional avatars.

[0004] BACKGROUND

[0005] Caricature generation is a unique artistic process that requires a trade-off between preserving the identity of the subject and distorting facial features to be as exaggerated as possible. Therefore, this has traditionally been a manual task performed by skilled artists.

[0006] In this field there are three central creative rules in caricature generation most commonly adhered to. The first is the ID of generated results must be identical to that of the reference image. That is, it must be possible to identify the subject. The second is the caricature generation models should support exaggerated deformations as much as possible. And the third is that the model should also have the ability to create various styles, brushes, backgrounds, expressions, and other attributes.

[0007] Caricature synthesis aims to exaggerate or distort facial features for a stylised yet recognisable portrayal of the subject. Early approaches typically involved a deformation stage followed by image-to-image translation. A generative adversarial network (GAN)-based framework (as described in “CariGANs: Unpaired photo-to-caricature translation” by Cao et al. ACM TOG, 2018) introduced facial landmarks to guide these deformations. This was later improved by automating control point prediction for warping and incorporating a discriminator to preserve identity (as described in “Warpgan: Automatic caricature generation” by Shi et al. In CVPR, 2019). Subsequent work has diversified caricature generation by exploring multiple facial exaggeration types and utilising StyleGAN with GAN inversion for greater control (as described in “Stylecarigan: caricature generation via stylegan feature map modulation” by Jang et al. In SIGGRAPH, 2021). In spatial manipulation, Semantics CariGAN (as described in “Learning to Caricature via Semantic Shape Transform” by Chu et al. HCV, 2021) applies semantic shape transformations using warped maps within a segmentation-guided dual domain framework. Recent advances in diffusion models have inspired diffusion-based caricature synthesis. DemoCaricature (as described in “Democaricature: Democratising caricature generation with a rough sketch” by Chen et al. In CVPR, 2024) preserves identity by fine-tuning the diffusion model and controls the layout with a text-to-image (T2I) adapter (as described in “T2I- Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models” by Mou et al. arXiv preprint arXiv:2302.08453, 2023).

[0008] With a limited set of reference images, T2I personalisation aims to adapt pretrained T2I models to specific concepts while maintaining generalisability. Among the proposed strategies, Textual Inversion (as described in “An Image is Worth One Word: Personalizing Text-to-image Generation using Textual Inversion” by Gal et al. In ICLR, 2023) optimizes text embeddings to capture new concepts, while DreamBooth (as described in “DreamBooth: Fine Tuning Text-to-image Diffusion Models for Subject-Driven Generation” by Ruiz et al. In CVPR, 2023) personalises outputs by fine-tuning entire latent diffusion models (LDMs).

[0009] Research into Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA (as described in “LoRA: Low-Rank Adaptation of Large Language Models” by Hu et al.. In ICLR, 2022), seeks to reduce the computational cost during training. CustomDiffusion (as described in “Multi-concept customization of text-to-image diffusion” by Kumari, et al. In CVPR, 2023) fine-tunes only cross-attention layers, while Perfusion (as described in “Key-Locked Rank One Editing for Text-to-image Personalization” by Tewel et al. In SIGGRAPH, 2023) introduces Rank-1 Model Editing (ROME) (as described in “Locating and Editing Factual Associations in GPT” by Meng et al. In NeurlPS, 2022), optimising the Value-pathway in cross-attention. DemoCaricature enhances personalisation by fine-tuning both a learnable text token and cross-attention layers. A notable contribution, InstantID (as described in “Instantid: Zero-shot identity-preserving generation in seconds” by Wang et al. arXiv:2401.07519, 2024), presents an ID-preserving adaptation method inspired by IP-Adapter (as described in “Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models” by Ye et al. arxiv:2308.06721, 2023), effectively balancing fidelity and efficiency.

[0010] Most of the previous approaches are based on generative adversarial networks (GANs). These models require both human face and caricature datasets and the image-to-image translation between the two is performed by cycleGAN (as described in “Unpaired image-to-image translation using cycle-consistent adversarial networks” by Zhu et al., International Conference on Computer Vision (ICCV), 2017). Although they do not require strict matching pairs of face images and caricatures, the huge gap between the amount of these two types of data is still the main reason why these models are difficult to train at a large scale. The GAN-based synthesis approaches deficiency result in generated caricatures lacking expressiveness and missing creative features like local abstraction, hairstyle variation, and view changes, which goes against the third creative rule mentioned above.

[0011] Therefore, more recent works turn their attention to latent diffusion models (LDM) in the hope of solving the above problems through the powerful generative capabilities of diffusion models. However, these works usually have two disadvantages. Firstly, for each new ID added, this approach requires fine tuning the model, which is very time-consuming. Secondly, this approach utilises vanilla T2I-Adapter or ControlNet to manipulate the layout. However, these layout conditioners are not trained on caricature data. As a result, it is difficult for them to balance between maintaining the subject’s identity and exaggerated deforming of the faces as mentioned above.

[0012] SUMMARY OF THE INVENTION

[0013] According to one aspect there is provided a method of generating a data set for training an Al model to generate caricatures by interactive editing of an input reference image of a subject, the method comprising: processing a facial image to identify a plurality of facial landmarks and defining the landmarks as a plurality of source points; defining a plurality of target points corresponding to a location of the source points after deformation of the facial image; relating the source points to landmark information comprising a text-based description of the facial feature in which the source points are located; and relating the target points to distorted landmark information comprising a text-based description of a distorted facial feature in which the source points were located.

[0014] In an embodiment, the deformation of the facial image may be obtained by applying a mapping algorithm to the facial image.

[0015] In an embodiment, the mapping algorithm may be a thin plate spline algorithm.

[0016] In an embodiment, the landmark information related to the source points may comprise any of the following facial features including eyes, nose, mouth, ears, jawline, etc.

[0017] In an embodiment, the distorted landmark information related to the target points may comprise any of the following facial feature descriptions including enlarging eyes, stretching the nose bridge, sliming the whole face, etc.

[0018] In an embodiment, the plurality of facial landmarks may be identified by applying a landmark prediction model to the facial image. According to another aspect there is provided a method of training an Al model to generate caricatures by interactive editing of an input reference image of a subject, the method comprising: inputting image-text pairs of deformed facial images and deformed landmark information as conditions of an IdentityNet pipeline; and minimising a loss function and optimising the IdentityNet parameters while maintaining other parameters of a pre-trained diffusion model, a text encoder, a face encoder, and an adapter at constant values.

[0019] In an embodiment, the loss function may be the equation:

[0020]

[0021] where c is the text embeddings, cIDis the ID embeddings extracted from the face image prompt, and CTPS^ denotes the deformed landmark input of the IdentityNet.

[0022] In an embodiment, a landmark format comprises line-based representation of shape information of facial features.

[0023] According to another aspect there is provided a method of implementing an Al model trained to generate caricatures by interactive editing of an input reference image of a subject, wherein the method comprises applying one or more Bezier curves to fit source points of distinct facial features identified in the input reference image to polylines.

[0024] In an embodiment, the one or more Bezier curves may be defined according to the parametric equation:

[0025]

[0026] where each curve is defined by a set of control points Po, P±, Pn, where n is the order of the curve, and t is the curve parameter.

[0027] In an embodiment, for each facial feature, a number of control points of a polyline may be selected based on the number of source points identified for that feature to ensure a smooth fit.

[0028] In an embodiment, a single facial feature may be fitted by a plurality of Bezier curves.

[0029] In an embodiment, the control points of the Bezier curves and the landmark source points of the facial features may be aligned by minimising a further loss function.

[0030] In an embodiment, the further loss function may be defined according to the equation:

[0031]

[0032] where Lkdenotes the landmark source points, B(tk) denotes the control points of the Bezier curve, and tkare the different facial features identified by the landmarks.

[0033] In an embodiment, the method may comprise receiving one or more edit instructions from a user in the form of manipulations of the one or more control points of the one or more Bezier curves representing manipulations of the associated landmark source points. According to another aspect there is provided a device comprising one or more processors, the one or more processors configured to operate to implement the method according to claim 10.

[0034] BRIEF DESCRIPTION OF THE FIGURES

[0035] The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0036] Figure 1 shows an overview of the proposed process including a data set generation stage using TIP deformation within a training stage and an inference stage.

[0037] Figure 2 shows the training stage including training data generation stage in more detail.

[0038] Figure 3 shows an example of the proposed approach during inference.

[0039] Figure 4 shows a plurality of example input facial images, alongside their generated caricatures with additional realistic or cartoon effects added.

[0040] Figure 5 shows a flow diagram illustrating steps of a proposed method of generating a data set for training an Al model to generate caricatures by interactive editing of an input reference image of a subject.

[0041] Figure 6 shows a flow diagram illustrating steps of a proposed method of training an Al model to generate caricatures by interactive editing of an input reference image of a subject.

[0042] Figure 7 shows a flow diagram illustrating steps of a proposed method of implementing an Al model trained to generate caricatures by interactive editing of an input reference image of a subject.

[0043] Figure 8 shows an example of a device configured to implement the image processor to process images taken by an image sensor 1102 in the device 1101.

[0044] DETAILED DESCRIPTION OF THE INVENTION

[0045] The objective of the proposed approach is to streamline the caricature generating task and provide access to quality caricatures for everyone. Specifically, the objective is that with a portrait of yourself and a few dragging actions depicting how you envision your caricature, an automatically generated high fidelity and high-quality caricature can be achieved.

[0046] There is proposed a caricature generation approach for creative and high-quality caricature synthesis by interactive point-based landmark editing. The technical objective is to streamline the caricature creation process, while preserving the exaggeration, ID consistency, and creativity.

[0047] The proposed approach takes advantage of the flexibility and power of diffusion models. However, the proposed approach introduces dilated deformation into the process and addresses the time-consuming limitations of DemoCaricature. The proposed approach builds upon the existing techniques of InstantID by incorporating facial deformation information and line-based layout control to generate caricatures. As mentioned above, the challenge of caricature generation lies in the fact that it should exaggerate the facial feature while still maintaining the identity of the subject. This challenge is addressed by utilising the Thin Plate Spline (TPS) deformation algorithm and incorporating this deformation information into the existing identity-preservation of text-to-image (T2I) diffusion models. Additionally, the problem of editing caricature shapes is transformed into a landmark editing problem. This is then combined with the editing capabilities of Bezier curves to enable the generation of exaggerated caricatures with just a few dragging actions.

[0048] For solving the above problems, a Thin Plate Spline (TPS) deformation is applied for both face images and landmarks to simulate the distortion effects of caricatures. This avoids the problem of the expensive collection of caricatures and their manual labelling. Next, the landmark settings are extended from “point-based” to “line-based” for use within the ControlNet pipeline to support shape level editing instead of only location level editing. This provides a more subjective and fine-grained control over the face features, which is inherently more conducive to the generation of exaggerated faces.

[0049] Furthermore, Bezier curves are introduced for landmark fitting during the inference stage. This enables point-based editing via the intuitive controllability of the control points of Bezier curves. The utilisation of Bezier curves seamlessly combines together the “line-based” condition in training and the “dragging ability” by the control points in inference, thus providing a simpler and more direct caricature generation pipeline.

[0050] Figure 1 shows an overview of the whole process including a data set generation step 102 using TIP deformation within a training phase 104 and an inference phase 106. The training data generation step 102 comprises a facial image 108 with mapped landmarks 110 being deformed 112 via the TPS deformation algorithm and providing the resulting deformed landmarks 114. A text description 116 of the facial features is also generated and based on all of this information the pipeline is trained to generate a basic caricature 118.

[0051] The inference step 106 comprises inputting the original facial image 108, performing a Bezier fitting to the landmark points 120, dragging the points to edit the landmarks 122, and based on the textual and identification input in combination with the editing instructions by way of the deformed Bezier curves, generating a final caricature 124. Generating the caricature 124 may comprise applying other processes like style effects.

[0052] The general approach utilises the TPS deformation with existing ID-preserving generation approaches to address the delicate balance between ID preservation and exaggerated distortion. The landmark condition is extended such that the face editing problem is transformed into a landmark editing problem. Combined with the intuitive controllability of Bezier curves, this enables individuals to easily create personalised caricatures with several simple dragging actions. Both qualitative and quantitative results are achieved which deliver the best performance in terms of overall metrics such as generation speed, quality, and creativity.

[0053] Figure 2 shows the training phase 104 including training data generation step 102. Specifically, before training, a Thin Plate Spline (TPS) algorithm is utilised to pre-process 102 face images 108 and their landmarks 110 based on landmark information (e.g., enlarge eyes, distort mouths, etc.). The face embeddings 206 and text embeddings 208 are then input into a fixed U-Net 202 through cross-attention layers. Finally, the deformed face image 112 and deformed landmark 114 are set as the IdentityNet 204 condition. Note that only the IdentityNet 204 is trainable. That is, IdentityNet 204 is a trainable module, but the U-Net 206 is a frozen module. A Thin Plate Spline (TPS) deformation algorithm is leveraged to simulate the exaggerated distortions characteristic of caricatures without relying on any actual caricature data. TPS is a widely used non-rigid deformation technique that warps images based on a set of control points and their corresponding target locations. This allows for smooth and continuous deformations across the entire image.

[0054] To pre-process the facial images 108 and landmarks 110 for caricature synthesis, key facial landmarks are detected using a landmark prediction model. These landmarks represent important facial features such as the eyes, nose, mouth, and jawline. TPS deformation is then applied by defining the source points with reference to the original facial landmarks and the target points as a distorted version of these landmarks (e.g., enlarging eyes, stretching the nose bridge, sliming the whole face, etc.). This simulates the exaggerated proportions typically seen in caricatures.

[0055] TPS defines a smooth mapping function f(x, y) that minimises the bending energy of the transformation, ensuring smooth deformations. The mapping function may be defined as:

[0056]

[0057] where xt, ytare the control points (in this case of the different facial regions), a, . a2. a3are the affine coefficients, w;are the weights for each control point, and U(r) = r2log(r) is the radial basis function that governs the smoothness of the transformation.

[0058] Thus, there is provided a method of generating a data set for training an Al model to generate caricatures by interactive editing of an input reference image of a subject. The method comprises processing a facial image to identify a plurality of facial landmarks and defining the landmarks as a plurality of source points. The method then defines a plurality of target points corresponding to a location of the source points after deformation of the facial image. The source points are then related to landmark information comprising a text-based description of the facial feature in which the source points are located. Similarly, the target points are related to distorted landmark information comprising a text-based description of a distorted facial feature in which the source points were located.

[0059] The deformation of the facial image may be obtained by applying a mapping algorithm to the facial image. The mapping algorithm may be a thin plate spline algorithm.

[0060] The landmark information related to the source points may comprise any of the following facial features including eyes, nose, mouth, ears, jawline, etc. It should be understood that other facial features may be included as desired. The distorted landmark information related to the target points may comprise any of the following facial feature descriptions including enlarging eyes, stretching the nose bridge, sliming the whole face, etc. It should be understood that many other distorted facial features may be described in addition to those listed as desired.

[0061] The plurality of facial landmarks may be identified by applying a landmark prediction model to the facial image.

[0062] By adjusting the positions of the target points, various levels of distortion can be simulated to mimic the effects of caricature exaggeration. The resulting TPS transformation is applied both to the facial images and the corresponding landmarks, creating a warped version of human faces that exhibits exaggerated and stylised features for the training stage.

[0063] During training stage, only the parameters of the IdentityNet are optimised. The parameters of the pre-trained diffusion model, text encoder, face encoder, and an adapter are kept frozen. The entire pipeline is trained on image-text pairs featuring deformed human face subjects. The method employs a training objective akin to that used in the existing InstantID, according to the equation:

[0064]

[0065] The original setting of the landmark condition in InstantID is a five point facial representation. Therefore, it can only represent location information of facial features rather than shape information. The proposed approach extends this “point-based” landmark format to a “line-based” representation. A format of the landmarks therefore may comprise a line-based representation of shape information of facial features.

[0066] Thus, there is provided a method of training an Al model to generate caricatures by interactive editing of an input reference image of a subject. The method comprises inputting image-text pairs of deformed facial images and deformed landmark information as conditions of an IdentityNet pipeline. Next, a loss function is minimised while optimising the IdentityNet parameters and maintaining other parameters of a pre-trained diffusion model, a text encoder, a face encoder, and an adapter at constant values. The IdentityNet pipeline follows the ControlNet model’s design, taking the landmark and face embedding as input. It should be understood that ControlNet is widely used as an additional module of diffusion model for layout control. Similarly, the face encoder and adapter are taken from InstantID.

[0067] Figure 3 shows an example of the process during inference 106. During inference stage 106, the landmarks) may be extracted from any given reference image 108 and fitted with Bezier curves 120. By interactively dragging the control points, users are thus able to generate a series of caricatures with different shapes.

[0068] The proposed approach during inference for landmark editing uses Bezier curves to fit the polylines of the predicted facial landmarks. To start with a landmark prediction model is employed to extract the facial keypoints (for example, 68 keypoints as per the usual criteria), which correspond to various facial features 120. The facial features may be any such as the jawline, eyes, nose, and mouth. Next, multiple Bezier curves may be applied to fit the points of distinct facial regions 120.

[0069] Mathematically, a Bezier curve is defined by a set of control points Po, P±, ... , Pn, where n is the order of the curve. The parametric equation for a Bezier curve of order ‘n’ is given by:

[0070]

[0071] where t is the curve parameter. For each facial region, the appropriate number of control points are selected based on the number of the facial keypoints to ensure a smooth and natural fit. For instance, the jawline may be approximated using a single seventhorder Bezier curve, ensuring a smooth transition along the lower face contour and supporting more exaggerated morphing. In contrast, simpler contours like the eyes could be fit by lower-order curves. Therefore, each eye may be modelled with two separate fourth-order Bezier curves, allowing for accurate representation of the curved shape. Similar strategies can be applied to other facial regions to capture their unique geometry.

[0072]

[0073] The control points Po, P±, . .. , Pnare learned by minimising this loss function using least squares estimation. It should be understood that other minimising methods may be used as appropriate. As a result, the fitted Bezier curve smoothly passes through or closely approximates the predicted landmark points, providing a flexible yet accurate representation of the facial features.

[0074] During inference stage, users can edit the landmark by manipulating the control points of the curves 122. Small adjustments to these control points lead to changes in the shape and curvature of the corresponding facial features. It is noted that Bezier curves are computationally efficient, making this approach well-suited for intuitive and exaggerated edits to facial landmarks. This allows for easy creation of stylised or exaggerated caricatures, with the advantage of incorporating deformation information during the training process 104, as previously described.

[0075] Therefore, there is proposed a method of implementing an Al model trained to generate caricatures by interactive editing of an input reference image of a subject as described above. The method comprises applying one or more Bezier curves to fit source points of distinct facial features identified in the input reference image to polylines.

[0076] For each facial feature, a number of control points of a polyline may be selected based on the number of source points identified for that feature to ensure a smooth fit. A single facial feature may be fitted by a plurality of Bezier curves. The control points of the Bezier curves and the landmark source points of the facial features may be aligned by minimising a further loss function. The further loss function is different to the loss function described in relation to the IdentityNet pipeline.

[0077] The proposed approach may comprise receiving one or more edit instructions from a user in the form of manipulations of the one or more control points of the one or more Bezier curves representing manipulations of the associated landmark source points.

[0078] Thus, according to the above-described approach, there is proposed a method of data augmentation by application of TPS deformation to introduce exaggerated deformation face images and landmark pairs. The ControlNet condition of InstantID framework is extended to adapt to richer condition for point based landmark editing. That is, the IdentityNet model is adapted to richer conditional landmark inputs by fine-tuning the ControlNet. And a landmark dragging application is proposed to produce a variety of deformed landmarks by Bezier curve fitting and interactive control point editing.

[0079] Figure 4 shows a plurality of example input facial images 108, alongside their generated caricatures with additional realistic 402 or cartoon 404 effects added. The example output caricatures maintain the identity of the subjects while providing the distortion of facial features needed to create caricatures. The corresponding distorted Bezier curves 406a, b,c can be seen in the top right comers of the generated caricatures.

[0080] There is therefore proposed a device comprising one or more processors where the one or more processors are configured to operate to implement the method of generating a caricature according to the above description. Figure 5 shows a flow diagram illustrating steps of a method 500 of generating a data set for training an Al model to generate caricatures by interactive editing of an input reference image of a subject. The method 500 comprises a first step 502 of processing a facial image to identify a plurality of facial landmarks and defining the landmarks as a plurality of source points. The method comprises a further step 504 of defining a plurality of target points corresponding to a location of the source points after deformation of the facial image. The next step 506 comprises relating the source points to landmark information comprising a text-based description of the facial feature in which the source points are located. The method comprises a further step 508 of relating the target points to distorted landmark information comprising a text-based description of a distorted facial feature in which the source points were located.

[0081] Figure 6 shows a flow diagram illustrating steps of a method 600 of training an Al model to generate caricatures by interactive editing of an input reference image of a subject. The method 600 comprises a first step 602 of inputting image-text pairs of deformed facial images and deformed landmark information as conditions of an IdentityNet pipeline. The method comprises a further step 604 of minimising a loss function and optimising the IdentityNet parameters while maintaining other parameters of a pre-trained diffusion model, a text encoder, a face encoder, and an adapter at constant values.

[0082] Figure 7 shows a flow diagram illustrating steps of a method 700 of implementing an Al model trained to generate caricatures by interactive editing of an input reference image of a subject. The method comprises a step 702 of applying one or more Bezier curves to fit source points of distinct facial features identified in the input reference image to polylines.

[0083] As the method 700 can be implemented by the device 1101 or a server such as an entity 1111 located in the cloud, further features of the method 700 result directly from the functionality of the device as well as the different embodiments of the training described above and below.

[0084] Figure 8 shows an example of a device configured to implement the image processor to process images taken by an image sensor 1102 in the device 1101. The device may be a camera, a mobile phone, a laptop, a tablet, or any other device capable of capturing or processing image data. Such a device 1101 typically includes some onboard processing capability. This could be provided by the processor 1104. The processor 1104 could also be used for the essential functions of the device. The device typically also comprises memory a 1103.

[0085] The transceiver 1105 is capable of communicating over a network with other entities 1110, 1111. Those entities may be physically remote from the device 1101. The network may be a publicly accessible network such as the internet. The entities 1110, 1111 may be based in the cloud. In one example, entity 1110 is a computing entity and entity 1111 is a command and control entity. These entities are logical entities. In practice they may each be provided by one or more physical devices such as servers and datastores, and the functions of two or more of the entities may be provided by a single physical device. Each physical device implementing an entity comprises a processor and a memory. The devices may also comprise a transceiver for transmitting and receiving data to and from the transceiver 1105 of device 1101. The memory stores in a non-transient way code that is executable by the processor to implement the respective entity in the manner described herein.

[0086] The command and control entity 1111 may train the artificial intelligence models used in each module of the system. This is typically a computationally intensive task, even though the resulting model may be efficiently described, so it may be efficient for the development of the algorithm to be performed in the cloud, where it can be anticipated that significant energy and computing resource is available. It can be anticipated that this is more efficient than forming such a model at a typical device. In one implementation, once the deep learning algorithms have been developed in the cloud, the command and control entity can automatically form a corresponding model and cause it to be transmitted to the relevant device. In this example, the system is implemented at the device 1101 by processor 1104.

[0087] In another possible implementation, an image may be captured by a camera sensor 1102 and the image data may be sent by the transceiver 1105 to the cloud for processing in the system. The resulting target image could then be sent back to the device 1101, as shown at 1112 in Figure 8. The image data may alternatively be captured by a camera sensor remote to the device 1101, sent to the device 1101 via any appropriate data transfer mode (e.g. WiFi, Bluetooth, etc.), and then sent to the cloud via the transceiver 1105.

[0088] Therefore, the method may be deployed in multiple ways, for example in the cloud, on the device, or alternatively in dedicated hardware. As indicated above, the cloud facility could perform training to develop new algorithms or refine existing ones. Depending on the compute capability near to the data corpus, the training could either be undertaken close to the source data, or could be undertaken in the cloud, e.g. using an inference engine. The system may also be implemented at the device (e.g. a camera), in a dedicated piece of hardware, or in the cloud.

[0089] There is also provided herein according to the embodiments of the methods and apparatus described above, a computer program product comprising a computer readable storage medium for storing program code which causes a computer or a processor to perform the method described in reference to figure 5 or the method described in reference to figure 6 or the method described in reference to figure 7 when the program code is executed by the computer or the processor.

[0090] Therefore, according to the description herein, there is provided an end-to-end trainable neural network for caricature generation. There is also provided a generative Al which enables users to generate exaggerated caricature based on one reference image through a few dragging actions.

[0091] There is further provided Thin Plate Spline for involving deformation information into diffusion models. With this algorithm the expensive collection of caricature data can be avoided while ensuring that the model still learns the deformation information unique to caricatures.

[0092] There is further provided a line-based landmark condition in diffusion models, using line-based conditions instead of previous point-based conditions for further shape editing by dragging.

[0093] There is further provided a method of interactive editing by dragging based on Bezier curves. This is presented as a pipeline that fits the landmark by Bezier curves. Then users can drag the control points of Bezier curves to generate new landmarks for caricature generation.

[0094] Compared to existing caricature generation methods, the proposed method has fast inference and a high-quality appearance in the output caricature. Compared to existing face generation methods, the proposed method introduces deformation information by implementing a Thin Plate Spline algorithm, which makes the generation result more flexible. The proposed method also introduces Bezier curve landmark fitting to enable users easily create a variety of deformed landmarks.

[0095] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

CLAIMS1. A method of generating a data set for training an Al model to generate caricatures by interactive editing of an input reference image of a subject, the method comprising:processing a facial image to identify a plurality of facial landmarks and defining the landmarks as a plurality of source points;defining a plurality of target points corresponding to a location of the source points after deformation of the facial image;relating the source points to landmark information comprising a text-based description of the facial feature in which the source points are located; andrelating the target points to distorted landmark information comprising a text-based description of a distorted facial feature in which the source points were located.

2. The method according to claim 1 , wherein the deformation of the facial image is obtained by applying a mapping algorithm to the facial image.

3. The method according to claim 2, wherein the mapping algorithm is a thin plate spline algorithm.

4. The method according to any preceding claim, wherein the landmark information related to the source points comprises any of the following facial features including eyes, nose, mouth, ears, jawline, etc.

5. The method according to any preceding claim, wherein the distorted landmark information related to the target points comprises any of the following facial feature descriptions including enlarging eyes, stretching the nose bridge, sliming the whole face, etc.

6. The method according to any preceding claim, wherein the plurality of facial landmarks are identified by applying a landmark prediction model to the facial image.

7. A method of training an Al model to generate caricatures by interactive editing of an input reference image of a subject, the method comprising:inputting image-text pairs of deformed facial images and deformed landmark information as conditions of an IdentityNet pipeline; andminimising a loss function and optimising the IdentityNet parameters while maintaining other parameters of a pretrained diffusion model, a text encoder, a face encoder, and an adapter at constant values.

8. The method according to claim 7, wherein the loss function is the equation:where c is the text embeddings, cIDis the ID embeddings extracted from the face image prompt, and CTPS^ denotes the deformed landmark input of the IdentityNet.

9. The method according to claim 7 or 8, wherein a landmark format comprises line-based representation of shape information of facial features.

10. A method of implementing an Al model trained to generate caricatures by interactive editing of an input reference image of a subject, wherein the method comprises applying one or more Bezier curves to fit source points of distinct facial features identified in the input reference image to polylines.

11. The method according to claim 10, wherein the one or more Bezier curves are defined according to the parametric equation:where each curve is defined by a set of control points Po, P±, Pn, where n is the order of the curve, and t is the curve parameter.

12. The method according to claim 10 or 11, wherein for each facial feature, a number of control points of a polyline is selected based on the number of source points identified for that feature to ensure a smooth fit.

13. The method according to any of claims 10 to 12, wherein a single facial feature is fit by a plurality of Bezier curves.

14. The method according to any of claims 10 to 13, wherein the control points of the Bezier curves and the landmark source points of the facial features are aligned by minimising a further loss function.

15. The method according to claim 14, wherein the further loss function is defined according to the equation:where Lkdenotes the landmark source points, B(tk) denotes the control points of the Bezier curve, and tkare the different facial features identified by the landmarks.

16. The method according to any of claims 10 to 15, wherein the method comprises receiving one or more edit instructions from a user in the form of manipulations of the one or more control points of the one or more Bezier curves representing manipulations of the associated landmark source points.

17. A device comprising one or more processors, the one or more processors configured to operate to implement the method according to claim 10.

Images