Electronic device and method for generating face swap image

Abstract

An electronic device for generating a face swap image according to an embodiment of the present invention includes one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, wherein the one or more processors are configured to detect a human face in a source image, determine whether an image of the detected face is suitable for generating a face swap image, in case where it is determined that the image of the face is suitable for generating the face swap image, detect landmarks of the face, designate an area of the face based on the detected landmarks, parse the face to detect and segment the face by region, generate a modified image by removing an object that covers at least a part of the face from the image of the face using an image generation AI model, and generate a face swap image by replacing a face in a target image with the face in a modified image.

Description

ELECTRONIC DEVICE AND METHOD FOR GENERATING FACE SWAP IMAGE

[0001] The present invention relates to an electronic device and method for generating a face swap image, and more particularly, to an electronic device and method capable of generating a high-quality face swap image using an AI model.

[0002] Face swap technology is a technique for swapping the faces of two individuals in a digital image or video, and it has recently been gaining attention in various application fields. It is particularly utilized in entertainment, advertising, film production, and social media, and is evolving towards the generation of increasingly sophisticated images.

[0003] FIG. 1 is a diagram illustrating a face swap image generated by a conventional technique. Conventional face swap techniques commonly use directly inputting and replacing the facial area of a source image. Consequently, if a part of the face is covered by objects such as glasses, hats, or earrings, the face swap may not be performed smoothly. For example, as shown in area 1 of FIG. 1, the bangs from the source image may be directly reflected in the target image, distorting the main features of the swapped face and making the result look unnatural. Furthermore, if there is a difference in the facial areas between the source and target images, problems such as the swapped facial area becoming blurry or distorted, as shown in area 2 of FIG. 1, can occur, leading to a degradation in image quality.

[0004] Therefore, there is a need for a method that can remove noise elements while maintaining the main facial features and provide high-quality face swap images for various source images.

[0005] An object of the present invention is to provide an electronic device and method that can provide high-quality swap results even for various input images by removing noise elements such as glasses and hats while maintaining the main facial features, and can perform face swapping flexibly without degrading the quality of the source image.

[0006] Another object of the present invention is to provide an electronic device and method that can generate more natural and high-quality face swap images by performing face alignment and image quality improvement tasks.

[0007] An electronic device for generating a face swap image according to an embodiment of the present invention includes one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, wherein the one or more processors are configured to detect a human face in a source image, determine whether an image of the detected face is suitable for generating a face swap image, in case where it is determined that the image of the face is suitable for generating the face swap image, detect landmarks of the face, designate an area of the face based on the detected landmarks, parse the face to detect and segment the face by region, generate a modified image by removing an object that covers at least a part of the face from the image of the face using an image generation AI model, and generate a face swap image by replacing a face in a target image with the face in a modified image.

[0008] According to an embodiment, the detecting the human face in the source image may include, in case where a plurality of human faces are detected in the source image and a plurality of human faces are detected in the target image, detecting a face with a largest facial area among a plurality of faces in the source image, and detecting a face with a largest facial area among a plurality of faces in the target image.

[0009] According to an embodiment, the determining whether an image of the detected face is suitable for generating a face swap image may include detecting an object that covers at least a part of the face and segmenting an area where the object is detected, and in case where at least one of an eye, a nose, and a mouth in the face is covered by the object, determining that the image of the face is unsuitable for generating the face swap image.

[0010] According to an embodiment, wherein the source image may be received from an external device, and the one or more processors may be configured to transmit an unsuitability message to the external device in case where it is determined that the image of the detected face is unsuitable for generating the face swap image.

[0011] According to an embodiment, the parsing the face to detect and segment the face by region may include calculating a median value of a plurality of pixels detected as skin in the face and removing skin blemishes based on the median value.

[0012] According to an embodiment, the generating a modified image by removing an object that covers at least a part of the face from the image of the face using an image generation AI model may include, by using a human part segmentation model, detecting the object that covers at least a part of the face and segmenting an area where the object is detected, inputting information regarding the area where the object is detected and an object removal command to the image generation AI model, and by the image generation AI model, generating the modified image by removing the object from the image of the face based on the information regarding the area where the object is detected and the object removal command.

[0013] According to an embodiment, the image generation AI model may include an AI model that is a diffusion model trained using Low-Rank Adaptation (LoRA) with a first face image in which an object that covers at least a part of a face is present and text describing the first face image, and a second face image corresponding to the first face image in which an object that covers at least a part of the face is absent and text describing the second face image.

[0014] According to an embodiment, the generating a face swap image by replacing a face in a target image with a face in the modified image may include aligning the face of the modified image to correspond to the face of the target image based on the landmarks, and generating the face swap image by replacing the face in the target image with the aligned face of the modified image.

[0015] According to an embodiment, wherein the target image may be a predetermined image stored in the one or more memories.

[0016] According to an embodiment, wherein the one or more processors may be configured to improve a quality of the generated face swap image using a face restoration model and a super-resolution model.

[0017] A method performed by an electronic device including one or more processors and one or more memories storing at least one instruction executed by the one or more processors according to an embodiment of the present invention may include detecting a human face in a source image; determining whether an image of the detected face is suitable for generating a face swap image; in case where it is determined that the image of the face is suitable for generating the face swap image, detecting landmarks of the face; designating an area of the face based on the detected landmarks; parsing the face to detect and segment the face by region; generating a modified image by removing an object that covers at least a part of the face from the image of the face, using an image generation AI model; and generating a face swap image by replacing a face in a target image with a face in the modified image.

[0018] According to an embodiment, wherein the detecting the human face in the source image may include, in case where a plurality of human faces are detected in the source image and a plurality of human faces are detected in the target image, detecting a face with a largest facial area among a plurality of faces in the source image, and detecting a face with a largest facial area among a plurality of faces in the target image.

[0019] According to an embodiment, wherein the determining whether the image of the detected face is suitable for generating a face swap image may include detecting an object that covers at least a part of the face and segmenting an area where the object is detected; and in case where at least one of an eye, a nose, and a mouth in the face is covered by the object, determining that the image of the face is unsuitable for generating the face swap image.

[0020] According to an embodiment, wherein the source image is received from a user device, and the method may further include transmitting an unsuitability message to the user device in case where it is determined that the image of the detected face is unsuitable for generating the face swap image.

[0021] According to an embodiment, wherein the parsing the face to detect and segment the face by region may include calculating a median value of a plurality of pixels detected as skin in the face and removing skin blemishes based on the median value.

[0022] According to an embodiment, wherein the generating a modified image by removing an object that covers at least a part of the face from the image of the face using an image generation AI model may include: by using a human part segmentation model, detecting the object that covers at least a part of the face and segmenting an area where the object is detected; inputting information regarding the area where the object is detected and an object removal command to the image generation AI model; and by the image generation AI model, generating the modified image by removing the object from the image of the face based on the information regarding the area where the object is detected and the object removal command.

[0023] According to an embodiment, wherein the image generation AI model may include an AI model that is a diffusion model trained using LoRA with a first face image in which an object that covers at least a part of a face is present and text describing the first face image, and a second face image corresponding to the first face image in which an object that covers at least a part of the face is absent and text describing the second face image.

[0024] According to an embodiment, wherein the generating a face swap image by replacing a face in a target image with a face in the modified image may include aligning the face of the modified image to correspond to the face of the target image based on the landmarks, and generating the face swap image by replacing the face in the target image with the aligned face of the modified image.

[0025] According to an embodiment, wherein the target image may be a predetermined image stored in the one or more memories.

[0026] According to an embodiment, wherein the method may further include improving a quality of the generated face swap image using a face restoration model and a super-resolution model.

[0027] The electronic device and method for generating a face swap image according to the present invention can provide high-quality swap results even for various input images by removing noise elements such as glasses and hats while maintaining the main facial features, and can perform face swapping flexibly without degrading the quality of the source image.

[0028] Furthermore, the electronic device and method for generating a face swap image according to the present invention can generate more natural and high-quality face swap images by performing face alignment and image quality improvement tasks.

[0029] FIG. 1 is a diagram illustrating a face swap image generated by a conventional technique.

[0030] FIG. 2 is a diagram for explaining a system including a server device and a user device according to an embodiment of the present invention.

[0031] FIG. 3 is a block diagram of a server device according to an embodiment of the present invention.

[0032] FIG. 4 is a flowchart illustrating a method for generating a face swap image according to an embodiment of the present invention.

[0033] FIG. 5 is a diagram for explaining the face detection step in the flowchart of FIG. 4.

[0034] FIG. 6 is a diagram for explaining the image suitability determination step in the flowchart of FIG. 4.

[0035] FIG. 7 is a diagram for explaining the face landmark detection and face area designation step in the flowchart of FIG. 4.

[0036] FIG. 8 is a diagram for explaining the face parsing step in the flowchart of FIG. 4.

[0037] FIG. 9 is a diagram for explaining the step of removing noise from the face using an image generation AI model in the flowchart of FIG. 4.

[0038] FIG. 10 is a diagram for explaining a fine-tuning method of an image generation AI model according to an embodiment of the present invention.

[0039] FIG. 11 is a diagram for explaining the step of performing a face swap in the flowchart of FIG. 4.

[0040] FIG. 12 is a diagram for explaining the image quality improvement step in the flowchart of FIG. 4.

[0041] FIG. 13 and FIG. 14 are diagrams illustrating examples of face swap images generated according to an embodiment of the present invention.

[0042] Hereinafter, specific embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification, detailed descriptions of well-known functions or configurations will be omitted if they are deemed to unnecessarily complicate or obscure the gist of the invention.

[0043] The advantages, features, and methods of achieving them of the disclosed embodiments will be clearly understood by referring to the accompanying drawings and the embodiments described below. However, the present invention is not limited to these embodiments and may be implemented in various forms. These embodiments are provided only as examples to help a thorough understanding of the invention and are not intended to limit the scope of the invention.

[0044] In the accompanying drawings, the same or similar components are given the same reference numerals. In addition, when describing embodiments of the present invention, descriptions of the same or similar components may be omitted to avoid redundant descriptions. However, the omission of such a description is not intended to mean that the corresponding component is not included in a specific embodiment.

[0045] The terms used in this specification have been selected from currently commonly used terms that sufficiently reflect the functions of the invention, but they may be changed depending on the perspective of those skilled in the art or the development of new technologies. In addition, in a specific case, there may be a term arbitrarily selected by the applicant, and in this case, its meaning will be described in detail in the specification. Therefore, the terms in this specification should be interpreted based on their meaning and the overall content of the present invention, not just their simple names.

[0046] In this specification, a singular expression may include a plural meaning unless it is clearly limited to a singular expression. Conversely, a plural expression may be interpreted as a singular expression in context, unless it is clearly limited to a plural expression. Throughout the specification, when a specific part is said to include a certain component, it means that it may include an additional component, not excluding another component.

[0047] In this specification, the expression "each of a plurality of A" or "a plurality of A each" may refer to each of all elements included in the plurality of A, or may refer to each of some elements of the plurality of A.

[0048] In this specification, the expression "one or more A" may mean a set of one or more A, unless the context clearly indicates otherwise.

[0049] In this specification, the expression "configured to ~" may have meanings such as "set to ~", "having the ability to ~", "modified to ~", "made to ~", "able to ~", etc. depending on the context. This expression is not limited to "specifically designed in hardware," and for example, a processor configured to perform a specific operation may mean a general-purpose processor capable of performing the operation through software execution, or a special-purpose computer structured through programming to perform the specific operation.

[0050] FIG. 2 is a diagram for explaining a system including a server device and a user device according to an embodiment of the present invention.

[0051] The server device 100 may be a server device of a service provider. The service provider may be an operating entity that provides the face swap service according to the present invention. The server device 100 communicates with the user device 200 via a communication network 300 and can perform the method for generating a face swap image according to the present invention.

[0052] The server device 100 may include various types of servers. For example, the server device 100 may include various types of servers such as a centralized server, a cloud server, a distributed server, a virtual environment server, an edge server, a multi-tenant server, or a combination thereof, and each server may be implemented physically or logically. Additionally, the server device 100 may implement a dedicated server optimized for a specific function, a general-purpose server, or an integrated system thereof. However, this is merely an example, and the present invention is not limited thereto.

[0053] The user device 200 may be a device of a user who uses the aforementioned face swap image generation service. For example, the user device 200 may include a smartphone, a tablet computer, a personal computer (PC), a mobile phone, a personal digital assistant (PDA), a wearable device, etc., but the present invention is not limited thereto.

[0054] In this specification, when describing the configuration or operation of a device, the term "device (e.g., electronic device, server device, user device, etc.)" is used to refer to the device being described, and the term "external device" is used to refer to a device that exists externally from the perspective of the device being described. For example, when describing the server device 100 as the "device", the user device 200 may be referred to as an "external device" from the perspective of the server device 100. Also, for example, when describing the user device 200 as the "device", the server device 100 may be referred to as an "external device" from the perspective of the user device 200. That is, the server device 100 and the user device 200 can each be referred to as a "device (e.g., electronic device)" and an "external device" or an "external device" and a "device (e.g., electronic device)" respectively, depending on the perspective of the operating entity.

[0055] The communication network 300 may include a wired communication network or a wireless communication network. For example, the wired communication network may include communication networks based on methods such as Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), etc., and the wireless communication network may include communication networks based on methods such as enhanced Mobile Broadband (eMBB), Ultra Reliable Low-Latency Communications (URLLC), Massive Machine Type Communications (MMTC), Long-Term Evolution (LTE), Global System for Mobile communications (GSM), Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), Wireless Broadband (WiBro), Wireless Fidelity (WiFi), Bluetooth, Near Field Communication (NFC), Global Positioning System (GPS), etc., but the present invention is not limited thereto.

[0056] FIG. 3 is a block diagram of a server device according to an embodiment of the present invention.

[0057] The server device 100 may include a communication circuit 110, a processor 120, and a memory 130. At least one of the components of the server device 100 may be omitted, or another component may be added to the server device 100. Additionally or alternatively, some components may be integrated and implemented, or implemented as a single or multiple entities. At least some of the components inside or outside the server device 100 may be connected to each other via a bus, General Purpose Input / Output (GPIO), Serial Peripheral Interface (SPI), or Mobile Industry Processor Interface (MIPI), etc., to transmit or receive data or signals.

[0058] The communication circuit 110 may include one or more communication circuits. The communication circuit 110 may perform wired or wireless communication between the server device 100 and an external device (e.g., the user device 200 or an unillustrated electronic device). For example, the communication circuit 110 may perform wired communication based on the aforementioned methods such as USB, HDMI, etc., or wireless communication based on methods such as eMBB, URLLC, MMTC, etc., but the present invention is not limited thereto.

[0059] The processor 120 may include one or more processors. The processor 120 may drive software (e.g., instructions, programs, etc.) to control at least one component of the server device 100 connected to the processor 120. Furthermore, the processor 120 may perform various operations such as various computations, processing, data generation, or manipulation. Also, the processor 120 may read data from or write data to the memory 130.

[0060] The processor 120 may include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Application Processor (AP), a mobile AP, a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a Microcontroller Unit (MCU), a Field-Programmable Gate Array (FPGA), etc., but the present invention is not limited thereto.

[0061] The memory 130 may include one or more memories. The memory 130 may write various data or read data upon request from the processor 120, etc. The memory 130 may store instructions that, when executed by one or more processors, cause the one or more processors to perform operations. For example, the memory 130 may include Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Twin Transistor RAM (TTRAM), MRAM, Thyristor RAM (TRAM), Zero capacitor RAM (Z-RAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory, Magnetic RAM (MRAM), Spin-Transfer Torque MRAM (STT-MRAM), Conductive Bridging RAM (CBRAM), Ferroelectric RAM (FeRAM), Phase change RAM (PRAM), etc., but the present invention is not limited thereto.

[0062] In this specification, the expression "a set of instructions stored in the memory 130" or "a program stored in the memory 130" may be used to refer to an operating system for controlling the resources of the server device 100, an application, or middleware that provides various functions to the application so that the application can utilize the resources of the server device 100. In an embodiment, when the processor 120 performs a specific operation, the memory 130 may store the instructions performed by the processor 120 corresponding to that specific operation.

[0063] In this specification, the server device 100 may be referred to as an electronic device. In this specification, what is described as being performed by the server device 100 can be understood as being performed by at least one of the communication circuit 110, the processor 120, and the memory 130 of the server device 100.

[0064] FIG. 4 is a flowchart illustrating a method for generating a face swap image according to an embodiment of the present invention.

[0065] First, (although not shown) a user can access the face swap service through a webpage provided by the server device 100. The user can perform a face swap on a target image through the face swap service provided by the server device 100.

[0066] In step S110, the user may select a source image for generating a face swap image on the user device 200. For example, the source image may include a photograph containing a human face, and the number of faces in the source image may be singular or plural. For example, the server device 100 may require a specific format for the source image used for the face swap, such as the file type (jpg, png, etc.), photo size, and pixel count, or it may not require a specific format.

[0067] In step S115, the user device 200 may transmit the selected source image to the server device 100. The server device 100 may receive the source image from the user device 200.

[0068] In step S120, the server device 100 can display a waiting screen on the display of the user device 200 to inform the user of the processing status. For example, the waiting screen may include a spinner alert, through which the server device 100 can visually communicate to the user that the system is processing a task. The waiting screen may automatically disappear once the task is complete.

[0069] In step S125, the server device 100 can detect a human face in the received source image. This will be explained with reference to FIG. 5.

[0070] FIG. 5 is a diagram for explaining the face detection step in the flowchart of FIG. 4.

[0071] The server device 100 can detect a human face in the received source image using a Convolutional Neural Network (CNN)-based face detection model. The server device 100 can detect a human face in the source image (FIG. 5(a)) and extract the facial area (FIG. 5(b)). In FIG. 5(b), the facial area is extracted in the form of a box, but this is an example, and the area can be extracted in various shapes such as an ellipse or a circle. The server device 100 can perform steps S130 to S170 based on the detected facial area.

[0072] According to an embodiment, in case where a plurality of human faces are detected in the source image and a plurality of human faces are detected in the target image, the server device 100 may detect a face with the largest facial area among a plurality of faces in the source image, and detect a face with the largest facial area among a plurality of faces in the target image. In this case, the server device 100 may perform steps S130 to S170 based on the faces (the largest faces) detected in the source image and the target image, respectively. However, this is an example, and if a plurality of faces are detected in the source image and the target image respectively, the server device 100 may list the detected faces in order of size and perform steps S130 to S170 on a pair of faces selected according to rules set by the server device 100 or the user device 200. For example, a face swap may be performed on face pairs whose sizes match within a certain error range (e.g., 10%), and the rules for selecting face pairs can be set in various ways.

[0073] Returning to the flowchart of FIG. 4, in step S130, the server device 100 may determine whether the image of the detected face is suitable for generating a face swap image. This will be explained with reference to FIG. 6.

[0074] FIG. 6 is a diagram for explaining the image suitability determination step in the flowchart of FIG. 4.

[0075] First, the server device 100 can detect an object that covers at least a part of the face in the detected face using a CNN-based image segmentation model. If there is an object covering at least a part of the face, the server device 100 can segment the area where the object is detected through the image segmentation model.

[0076] FIG. 6(a) shows an example image where a camera is covering the mouth, nose, left eye, and part of the right eye, and the server device 100 can segment the area occupied by the camera, distinguishing it from other parts of the face. FIG. 6(b) shows an example image where a mask is covering the mouth and nose, and the server device 100 can segment the area occupied by the mask, distinguishing it from other parts of the face. FIG. 6(c) shows an example image where sunglasses and bangs are covering parts of the face, and the server device 100 can segment the area occupied by the sunglasses covering parts of the face and the area occupied by the bangs, distinguishing them from other parts of the face.

[0077] The server device 100 can determine whether the image of the face is suitable for generating a face swap image based on the segmented object area and the remaining area of the face. The criteria for determining suitability can vary.

[0078] According to an embodiment, the server device 100 may determine that the image of the face is unsuitable for generating a face swap image if at least one of the eyes, nose, and mouth is covered by a specific object. The object may include accessories such as sunglasses, hats, or body parts like hair or hands. For example, if at least one of the body parts including the eyes, nose, and mouth is covered by an object by a predetermined ratio or more (e.g., 30%, 40%, etc.), the server device 100 may determine that the image of the face is unsuitable for generating a face swap image. Examples of such unsuitable images are shown in FIG. 6(a) and FIG. 6(b). In FIG. 6(a), the mouth, nose, and left eye are all covered by the camera, and part of the right eye is also covered by the camera. In FIG. 6(b), the mouth and nose are both covered by a mask. Therefore, the server device 100 can determine that the images in both FIG. 6(a) and FIG. 6(b) are unsuitable for generating a face swap image.

[0079] According to an embodiment, if there is no human face in the source image or the detected face is a character, the server device 100 may determine that the image is unsuitable for generating a face swap image.

[0080] According to an embodiment, the server device 100 may infer the age of the person from the face detected in the source image, and if the inferred age is below a certain age (e.g., 19 years old), it may determine it as unsuitable for generating a face swap image.

[0081] According to an embodiment, even if parts of the eyes, nose, and mouth are covered, if the overall shape of the covered body part can be inferred, the server device 100 may determine that the image is suitable for generating a face swap image. For example, in the case of FIG. 6(c), although the sunglasses cover part of the nose, the overall shape of the nose can be inferred, so it does not interfere with image integration. Therefore, the server device 100 can determine it as a suitable image for generating a face swap image. Alternatively, since regular glasses do not obscure body parts including the eyes through their lenses, the server device 100 can determine it as a suitable image for generating a face swap image.

[0082] The unsuitable and suitable images described above are examples, and there can be various other examples of unsuitable and suitable images, to which the present invention is not limited. For example, cases where the face is excessively distorted, where major facial parts are not clearly visible due to a specific angle, or where face recognition is difficult due to poor lighting, may correspond to unsuitable images. On the other hand, even if a part of the face is covered, if the main features are clearly visible and the image quality is sufficient, it can be determined as a suitable image.

[0083] Through the process described above, it can be determined whether the image of the face detected in the source image is suitable for generating a face swap image, and the determined images can be used as training data for an AI model to improve its determination accuracy. Specifically, the AI model can be trained on suitability based on various conditions such as the degree of facial occlusion, lighting conditions, facial angles, and the presence of accessories (e.g., glasses, hats, etc.). Through this, the AI model learns the detailed features of the image and the determination criteria, allowing it to determine suitability with higher accuracy during face swap image generation and provide optimized results to the user.

[0084] Returning to the flowchart of FIG. 4, if the server device 100 determines that the image of the detected face is unsuitable for generating a face swap image (No path from step S130), it may transmit an unsuitability message to the user device 200 (step S135). The unsuitability message may include content to help the user's understanding, for example, a guide message such as "The selected image is not suitable for face swap. Please select another image." may be displayed. The message may appear as a pop-up on the screen of the user device 200, and the user can select another image or retry on the user device 200.

[0085] If the server device 100 determines that the image of the face is suitable for generating a face swap image (Yes path from step S130), it can detect the landmarks of the face in the source image and designate an area of the face based on the detected landmarks (step S140). This will be explained with reference to FIG. 7.

[0086] FIG. 7 is a diagram for explaining the face landmark detection and face area designation step in the flowchart of FIG. 4.

[0087] The server device 100 can detect the landmarks of the face for the image of the face determined to be suitable for generating a face swap image.

[0088] Landmarks are the main feature points of the face represented as coordinate values and can be set based on important parts of the face such as the eyes, nose, mouth, and jawline. These landmarks are automatically extracted through an AI-based face recognition algorithm. For example, the AI model for determining landmarks can utilize a deep learning-based CNN (Convolutional Neural Network) model, and the AI model can be pre-trained with a vast face dataset to be trained to determine facial landmarks despite various facial angles, lighting, and expression changes.

[0089] For example, a plurality of landmark points can be used in the landmark detection process, and these points can correspond to the outer and inner corners of the eyes, the tip of the nose, the boundary of the lips, and various positions on the jawline. Landmark points play an important role in forming the contour of the face, and each landmark reflects the proportions and shape of the face in detail, so recognition is possible even when the face is rotated or tilted.

[0090] The server device 100 can set the landmarks by calculating the landmark points (coordinate values) of major parts such as the eyes, nose, and mouth. An example where facial landmarks are set in this manner is shown in FIG. 7(a). The detected landmarks represent the main features of the face, and the server device 100 can estimate the facial contour through the coordinate values of the landmarks. In FIG. 7(a), the results show the eyes, nose, mouth, and facial contour of the face marked with landmark points.

[0091] The server device 100 can designate a facial area based on the detected landmarks. For example, the server device 100 can create a closed-curve-shaped area that includes the facial contour using the landmark coordinate values. In FIG. 7(b), the generated facial area is shown in highlighted region. The designated facial area sets the precise facial range to be used for the face swap, preventing the face from exceeding its boundaries or abnormal synthesis from occurring during the face swap process.

[0092] Returning to the flowchart of FIG. 4, in step S145, the server device 100 can parse the face detected in the source image to detect and segment it by region. This will be explained with reference to FIG. 8.

[0093] FIG. 8 is a diagram for explaining the face parsing step in the flowchart of FIG. 4.

[0094] Face parsing refers to the process of individually recognizing and segmenting each component that constitutes a face within an image. Through parsing, the server device 100 can detect the detected face by detailed regions and segment each area into detailed facial parts such as skin, hair, eyes, nose, and lips.

[0095] FIG. 8 shows the result of the face detected in FIG. 5 being parsed, with each part distinguished by a different color. For example, hair is shown in red (R1 region), eyes in yellow (R2 region), eyebrows in light green (R3 region), nose in orange (R4 region), lips in pink (R5 region), and skin in blue (R6 region).

[0096] According to an embodiment, while or after face parsing is performed, an additional task of calculating the median value of a plurality of pixels detected as skin on the face and removing skin blemishes based on this median value may be performed. For example, areas that deviate from the median value by a predetermined amount (e.g., areas that differ by 30% or more, or less) may be recognized as blemishes, and the corresponding area can be corrected to the median value. This blemish removal operation can be performed automatically by the server device 100 or directly by the user through the user device 200.

[0097] According to an embodiment, while or after face parsing is performed, correction work may be performed on major parts such as the eyes, nose, and mouth, or additional corrections such as skin tone adjustment may be performed. This work can also be performed automatically by the server device 100 or directly by the user through the user device 200.

[0098] Returning to the flowchart of FIG. 4, in step S150, the server device 100 can generate a modified image by removing an object that covers at least a part of the face from the image of the face, using an image generation AI model. This will be explained with reference to FIG. 9.

[0099] FIG. 9 is a diagram for explaining the step of removing noise from the face in the flowchart of FIG. 4.

[0100] First, the server device 100 can detect an object that covers at least a part of the face and segment the area where the object is detected, using a human part segmentation model.

[0101] Specifically, the server device 100 can use a CNN-based human part segmentation model to detect an area including the face and an object covering the face (e.g., accessories such as glasses, hats, earrings, or body parts like hair) and segment it from the facial area. For example, in object detection, the server device 100 can more clearly detect the area of the object covering the face by additionally using the facial landmark points and facial area information extracted in step S140.

[0102] Next, the server device 100 can input information regarding the area where the object was detected and a text-based object removal command to the image generation AI model. The server device 100 can use the image generation AI model to remove the object from the image of the face and generate a modified image, based on the information regarding the area where the image object was detected and the object removal command. For example, the image generation AI model may include a diffusion model such as a Text-to-Image Inpainting model. The Text-to-Image Inpainting model can interpret input text commands (e.g., "remove glasses," "remove hat," etc.) to naturally remove the specified object. The image generation AI model can correct the area where the object was removed with pixels that harmonize with the surrounding facial area, thereby generating a natural modified image without a sense of incongruity with the source image.

[0103] The server device 100 according to the present invention can more accurately remove an object by using the information regarding the detected area along with the object removal command as input for the image generation AI model.

[0104] For example, if hair covers part of the face, when the server device 100 inputs the area information segmented as hair and a hair removal command (e.g., "remove hair") to the image generation AI model, the image generation AI model can more accurately target that hair area to remove the hair at that location and naturally restore the original appearance of the face. In this case, inpainting is applied only to that hair area, so only the hair can be removed without affecting other areas of the face.

[0105] Similarly, accessories like glasses or hats are also automatically recognized and removed by the image generation AI model based on the area information where the object was detected, enabling accurate and efficient processing without requiring separate input or area designation from the user.

[0106] In the example of FIG. 9, first, sunglasses and bangs are detected using a human part segmentation model. Then, information regarding the areas segmented as sunglasses and bangs, respectively, along with a noise removal command (e.g., "remove sunglasses and bangs"), can be input to the image generation AI model. FIG. 9 shows an example where the sunglasses and bangs have been removed through this process. The server device 100 can generate a modified image of the face with the sunglasses and bangs naturally removed, based on the facial area detected in FIG. 5.

[0107] The automated face noise removal operation described above, where the server device 100 performs the entire process from object detection to removal, can provide a cleaner and more natural face image without user intervention, achieving high completeness in face swap image generation.

[0108] The server device 100 can perform fine-tuning to increase the object removal accuracy of the image generation AI model. This will be explained with reference to FIG. 10.

[0109] FIG. 10 is a diagram for explaining a fine-tuning method of an image generation AI model according to an embodiment of the present invention.

[0110] In the server device 100, additional training can be applied to the image generation AI model to increase its object removal accuracy. For example, a fine-tuning technique such as Low-Rank Adaptation (LoRA) can be used to reflect a specific style or feature in the image generation AI model. LoRA is a technique that adds a low-rank (low-dimensional) transformation to the existing model parameters, allowing the model to be fine-tuned for specific characteristics without updating all the parameters of a large-scale model. This can increase the efficiency and speed of learning while reflecting desired detailed functions in the model.

[0111] For example, as training data for the image generation AI model, pairs of a first face image in which an object that covers at least a part of the face is present and text describing the first face image, and a second face image corresponding to the first face image in which the object covers at least a part of the face is absent and text describing the second face image can be used, and the image generation AI model may include an AI model that is a diffusion model trained using LoRA with this training data.

[0112] For example, if the object is sunglasses, a plurality of pairs can be prepared, consisting of a first face image where sunglasses cover part of the nose and cheeks and text describing the first face image (e.g., "an image where parts of the nose and cheeks are covered by sunglasses"), and a second face image of the same first face image but without sunglasses and text describing the second face image (e.g., "an image with the sunglasses covering parts of the nose and cheeks removed"). These pairs can be input into the image generation AI model and trained using LoRA.

[0113] Similarly, if the object is bangs, a plurality of pairs can be prepared, consisting of a first face image where bangs cover the forehead and text describing the first face image (e.g., "an image where part of the forehead is covered by hair"), and a second face image of the same first face image but without bangs and text describing the second face image (e.g., "an image with the hair covering part of the forehead removed"). These can be used as training data for fine-tuning the image generation AI model with LoRA.

[0114] If there are multiple objects covering the face, descriptions for each object can be included in the text. FIG. 10(a) shows a first face image where not only sunglasses but also bangs cover part of the face, and text describing the first face image (e.g., "an image where part of the forehead is covered by bangs, and parts of the cheeks and nose are covered by sunglasses"). FIG. 10(b) shows a second face image of the same first face image but without the bangs and sunglasses, and text describing the second face image (e.g., "an image with the bangs covering part of the forehead and the sunglasses covering parts of the cheeks and nose removed"). These images and texts can also be used as training data for fine-tuning the image generation AI model with LoRA.

[0115] The sunglasses and hair described above are just examples of objects, and by creating similar image and text pairs for various objects that cover the face and performing fine-tuning of the image generation AI model, the object removal accuracy can be further increased.

[0116] Returning to the flowchart of FIG. 4, in step S155, the server device 100 can generate a face swap image by replacing the face in the target image with the face from the modified image generated in step S150. This will be explained with reference to FIG. 11.

[0117] FIG. 11 is a diagram for explaining the step of performing a face swap in the flowchart of FIG. 4.

[0118] First, the server device 100 can align the face of the modified image to precisely overlap with the face of the target image based on the detected facial landmarks, that is, so that the major parts of the two faces, such as the eyes, nose, and mouth, correspond.

[0119] For example, if the position or tilt of the modified image's face is misaligned with the target image's face, the landmark information can be used to fine-tune the modified image's face to match the angle of the target image's face, allowing it to be smoothly combined with the target face.

[0120] As another example, even if the modified image's face is facing forward and the target image's face is in profile, alignment can be performed based on the landmarks to match the position and angle of the two faces. Through this landmark-based alignment, natural swapping is possible even between faces at various angles and orientations, and it can respond to subtle differences in expression. Therefore, face swapping corresponding to various facial directions and poses becomes possible, providing users with more flexible images.

[0121] Next, the server device 100 can generate a face swap image by replacing the face in the target image with the aligned face of the modified image. For example, the server device can use a Generative Adversarial Network (GAN)-based face swap model to naturally combine the face of the modified image with the target image, thereby generating a high-quality face swap image.

[0122] Compared to the face swap result in FIG. 1, the face swap image in FIG. 11, generated according to an embodiment of the present invention, shows significantly improved results in areas 1 and 2. For example, in area 1, the bangs from the source image are removed, and the target image's bangs are displayed naturally, showing a harmonious combination with the source image's face. This is because the bangs area was naturally removed from the source image. Furthermore, in the case of area 2, the faces of the source image and the target image are accurately overlapped through the face alignment process, and the swapped face is applied naturally to the target image with relatively improved image quality. Through such alignment, the final face swap result can be expressed more naturally and consistently.

[0123] According to an embodiment, the target image may include a predetermined image stored in one or more memories 130 of the server device 100. For example, the face swap image generation method of the present invention can be used for the purpose of swapping the face from an image provided by the user device 200 onto a predetermined target image, such as a movie poster. In such a case, the target image is set to a predetermined image stored on the server device 100 (e.g., a poster for video content such as a movie or drama), while the source image input from the user device 200 can be provided in various angles and compositions. Since the target image is fixed, the server device 100 can pre-analyze and store the landmark information of the target image to increase the efficiency of alignment with the source image. In this way, the server device 100 can perform the alignment between various source images provided by the user device 200 and the target image more quickly and accurately.

[0124] In another embodiment, the server device 100 can receive the target image from the user device 200.

[0125] Returning to the flowchart of FIG. 4, in step S160, the server device 100 can perform an operation to improve the quality of the generated face swap image. This will be explained with reference to FIG. 12.

[0126] FIG. 12 is a diagram for explaining the image quality improvement step in the flowchart of FIG. 4.

[0127] The server device 100 can further improve the quality of the face swap image generated in step S155 using a face restoration model and a super-resolution model. FIG. 12 shows the result of applying the face restoration and super-resolution models to the face swap image to enhance its completeness.

[0128] The face restoration model can improve the image quality by restoring the detailed features of the face. The face restoration model is commonly used to restore low-resolution or noisy face images to high resolution, and it can provide clearer and more realistic results by correcting major landmarks such as the eyes, nose, mouth, and skin texture.

[0129] The super-resolution model can be used to generate an overall high-resolution image by upscaling the image's resolution. The super-resolution model can supplement the pixels of the existing image and add fine details, minimizing quality loss while enlarging the image. The super-resolution model can correct the image generated by the face swap to make it appear sharper and clearer.

[0130] Through the image quality improvement operation, the server device 100 can provide the user with a high-quality face swap image with improved clarity and detail, rather than simply providing a face swap image where faces are merely combined.

[0131] Returning to the flowchart of FIG. 4, in step S165, the server device 100 can provide the generated (quality-improved) face swap image to the user device 200.

[0132] In step S170, the user device 200 can display the face swap image in a pop-up format on the display. The pop-up shows the face swap image to the user in real-time and may include functions to save the image or share it on social media.

[0133] According to an embodiment, the server device 100 may provide the user device 200 with options to share the face swap image via a messaging app or set it as a wallpaper. This allows users to utilize their own movie poster-style images for various purposes.

[0134] According to an embodiment, the server device 100 may transmit the face swap image to the user device 200 as a file or provide a link to download the face swap image. However, this is merely an example, and the method by which the server device 100 provides the face swap image to the user device 200 can vary, and the present invention is not limited thereto.

[0135] FIG. 13 and FIG. 14 are diagrams illustrating examples of face swap images generated according to an embodiment of the present invention.

[0136] FIG. 13 and FIG. 14 show images generated according to the face swap image generation method of the present invention, illustrating that face swaps can be naturally applied under various conditions.

[0137] For example, even when the face in the source image of FIG. 13(a) is facing forward and the target image of FIG. 13(b) is in profile, the server device 100 precisely aligns the face of the source image to the side profile of the target image, showing an example of a face-swapped image. By the face swap image generation method according to the present invention, the face of the source image is well-adjusted to the angle of the target image, resulting in a natural swap.

[0138] As another example, even when the face in the source image of FIG. 14(a) is partially covered by hair and sunglasses, it shows an example of a naturally face-swapped image onto the face of the target image of FIG. 14(b). Through the object removal and face restoration models of the present invention, the covered parts of the source image are restored and consistently applied to the target image, resulting in a natural swap.

[0139] The electronic device and method for swapping a face from a source image onto a single target image have been described above. Since a video is a result created by continuously combining a plurality of pictures (frames), a natural face swap video can be created by applying the aforementioned face swap image generation method to each frame.

[0140] According to an embodiment, the server device 100 may align the face of the source image to the target frame for each frame of a video, perform object removal and face restoration if necessary, and then proceed with the face swap operation. After performing the face swap for each frame in this way, the frames can be connected to provide a continuous face swap video. Through this, the server device 100 can provide users with a more dynamic and entertaining face swap experience, and users can enjoy an immersive experience as if their own face is naturally appearing in a movie or advertisement scene.

[0141] The methods according to the present invention may be computer-implemented methods. In this disclosure, although each operation of the methods has been shown and described in a certain order, each operation may be performed in an order that can be arbitrarily combined according to the present disclosure, in addition to being performed sequentially. In one embodiment, at least some of the operations may be performed in parallel, repeatedly, or heuristically. The present disclosure does not exclude changes or modifications to the methods. In one embodiment, at least some of the operations may be omitted, or another operation may be added.

[0142] The various embodiments of the present invention may be implemented as software recorded on a machine-readable recording medium. The software may be software for implementing the various embodiments of the present disclosure described above. The software may be inferred from the various embodiments of the present disclosure by programmers in the technical field to which the present disclosure belongs. For example, the software may be a machine-readable instruction (e.g., code or code segment) or a program. The machine may be a device capable of operating according to instructions called from a recording medium, for example, a computer. In one embodiment, the machine may be an electronic device according to the embodiments of the present disclosure. In one embodiment, a processor of the machine may execute the called instructions to cause the components of the machine to perform functions corresponding to the instructions. In one embodiment, the processor may be a processor of an electronic device according to the embodiments of the present disclosure. The recording medium may mean any kind of recording medium on which data is stored and which can be read by a machine. The recording medium may include, for example, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. In one embodiment, the recording medium may be a memory. In one embodiment, the recording medium may be implemented in a distributed form in a computer system, etc. connected by a network. The software may be stored and executed in a distributed manner in a computer system, etc. The recording medium may be a non-transitory computer-readable recording medium. The non-transitory computer-readable recording medium means a tangible medium regardless of whether data is stored semi-permanently or temporarily, and does not include a signal that is transmitted temporarily.

[0143] As described above, those skilled in the technical field of the present invention will be able to recognize that the present invention can be implemented in various forms without changing its technical principles or core features. Therefore, it should be understood that the above embodiments are illustrative and not restrictive of the scope of the present invention. The scope of the present invention is defined by the following claims rather than the detailed description, and all changes or modifications according to the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

[0144] The features and advantages described in this specification are only some of them, and more additional features and advantages will become clear to those skilled in the art with reference to the drawings, specification, and claims. In addition, it should be noted that the language used in this specification was selected for readability and explanation, and was not necessarily selected for the purpose of limiting or describing the subject matter of the present invention.

[0145] The descriptions of the above embodiments are presented for illustrative purposes, and are not intended to limit the scope of the present invention to the precise forms. Those skilled in the art will be able to understand that various modifications and variations are possible through the disclosure of the present invention.

[0146] Therefore, the scope of the present invention is not limited by the detailed description, but is defined by the claims of this specification. Therefore, the embodiments of the present invention are exemplary and do not limit the scope of the present invention described in the following claims.

Claims

1.An electronic device for generating a face swap image, comprising:one or more processors; andone or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations,wherein the one or more processors are configured to:detect a human face in a source image,determine whether an image of the detected face is suitable for generating a face swap image,in case where it is determined that the image of the face is suitable for generating the face swap image, detect landmarks of the face,designate an area of the face based on the detected landmarks,parse the face to detect and segment the face by region,generate a modified image by removing an object that covers at least a part of the face from the image of the face, using an image generation AI model, andgenerate a face swap image by replacing a face in a target image with a face in the modified image.2.The electronic device of claim 1, wherein the detecting the human face in the source image includes:in case where a plurality of human faces are detected in the source image and a plurality of human faces are detected in the target image, detecting a face with a largest facial area among a plurality of faces in the source image, and detecting a face with a largest facial area among a plurality of faces in the target image.3.The electronic device of claim 1, wherein the determining whether an image of the detected face is suitable for generating a face swap image includes:detecting an object that covers at least a part of the face and segmenting an area where the object is detected, andin case where at least one of an eye, a nose, and a mouth in the face is covered by the object, determining that the image of the face is unsuitable for generating the face swap image.4.The electronic device of claim 1, wherein the source image is received from an external device, andthe one or more processors are configured to transmit an unsuitability message to the external device in case where it is determined that the image of the detected face is unsuitable for generating the face swap image.5.The electronic device of claim 1, wherein the parsing the face to detect and segment the face by region includes, calculating a median value of a plurality of pixels detected as skin in the face and removing skin blemishes based on the median value.6.The electronic device of claim 1, wherein the generating a modified image by removing an object that covers at least a part of the face from the image of the face using an image generation AI model includes:by using a human part segmentation model, detecting the object that covers at least a part of the face and segmenting an area where the object is detected,inputting information regarding the area where the object is detected and an object removal command to the image generation AI model, andby the image generation AI model, generating the modified image by removing the object from the image of the face based on the information regarding the area where the object is detected and the object removal command.7.The electronic device of claim 6, wherein the image generation AI model includes an AI model that is a diffusion model trained using Low-Rank Adaptation (LoRA) with a first face image in which an object that covers at least a part of a face is present and text describing the first face image, and a second face image corresponding to the first face image in which an object that covers at least a part of the face is absent and text describing the second face image.8.The electronic device of claim 1, wherein the generating a face swap image by replacing a face in a target image with a face in the modified image includes:aligning the face of the modified image to correspond to the face of the target image based on the landmarks, and generating the face swap image by replacing the face in the target image with the aligned face of the modified image.9.The electronic device of claim 1, wherein the target image is a predetermined image stored in the one or more memories.10.The electronic device of claim 1, wherein the one or more processors are configured to improve a quality of the generated face swap image using a face restoration model and a super-resolution model.11.A method performed by an electronic device including one or more processors and one or more memories storing at least one instruction executed by the one or more processors,the method comprising:detecting a human face in a source image;determining whether an image of the detected face is suitable for generating a face swap image;in case where it is determined that the image of the face is suitable for generating the face swap image, detecting landmarks of the face;designating an area of the face based on the detected landmarks;parsing the face to detect and segment the face by region;generating a modified image by removing an object that covers at least a part of the face from the image of the face, using an image generation AI model; andgenerating a face swap image by replacing a face in a target image with a face in the modified image.12.The method of claim 11, wherein the detecting the human face in the source image includes:in case where a plurality of human faces are detected in the source image and a plurality of human faces are detected in the target image, detecting a face with a largest facial area among a plurality of faces in the source image, and detecting a face with a largest facial area among a plurality of faces in the target image.13.The method of claim 11, wherein the determining whether an image of the detected face is suitable for generating a face swap image includes:detecting an object that covers at least a part of the face and segmenting an area where the object is detected; andin case where at least one of an eye, a nose, and a mouth in the face is covered by the object, determining that the image of the face is unsuitable for generating the face swap image.14.The method of claim 11, wherein the source image is received from a user device, andthe method further comprising:transmitting an unsuitability message to the user device in case where it is determined that the image of the detected face is unsuitable for generating the face swap image.15.The method of claim 11, wherein the parsing the face to detect and segment the face by region includes, calculating a median value of a plurality of pixels detected as skin in the face and removing skin blemishes based on the median value.16.The method of claim 11, wherein the generating a modified image by removing an object that covers at least a part of the face from the image of the face, using an image generation AI model includes:by using a human part segmentation model, detecting the object that covers at least a part of the face and segmenting an area where the object is detected;inputting information regarding the area where the object is detected and an object removal command to the image generation AI model; andby the image generation AI model, generating the modified image by removing the object from the image of the face based on the information regarding the area where the object is detected and the object removal command.17.The method of claim 16, wherein the image generation AI model includes an AI model that is a diffusion model trained using LoRA with a first face image in which an object that covers at least a part of a face is present and text describing the first face image, and a second face image corresponding to the first face image in which an object that covers at least a part of the face is absent and text describing the second face image.18.The method of claim 11, wherein the generating a face swap image by replacing a face in a target image with a face in the modified image includes:aligning the face of the modified image to correspond to the face of the target image based on the landmarks, and generating the face swap image by replacing the face in the target image with the aligned face of the modified image.19.The method of claim 11, wherein the target image is a predetermined image stored in the one or more memories.20.The method of claim 11, further comprising improving a quality of the generated face swap image using a face restoration model and a super-resolution model.

Images