Image editing method based on semantic mask expansion and electronic device for performing method

Abstract

Provided is a method in which an electronic device edits an image on the basis of generative AI. The method may comprise the steps of: identifying an object in an image on the basis of a user input; generating a mask representing a region of the object; performing noise prediction using a noise processing network on the basis of the image and the mask; detecting a semantic region associated with the object on the basis of the result of the noise prediction; generating an expanded mask including the semantic region; and displaying an edited image via a generative model on the basis of the image and the expanded mask.

Description

Semantic mask expansion-based image editing method and electronic device performing the method

[0001] The present disclosure relates to a method for expanding a mask by detecting a region semantically associated with an object, a method for editing an image using the expanded mask, an electronic device for performing the method, and a server.

[0002] Generative AI is a technology that learns the structure and patterns of large-scale data and generates new synthetic data based on input data. It produces human-level results in various tasks involving text, images, voice, video, music, and more. For example, editing an image using a generative model generates a new image by editing the original image based on given data (e.g., images, masks, etc.).

[0003] However, when masks are used for image editing, areas associated with the object, such as light reflections and shadows, are sometimes not included in the mask, and this incomplete mask results in unnatural image editing outcomes.

[0004] According to one aspect of the present disclosure, a method for an electronic device to edit an image based on generative AI may be provided. The method may include the step of identifying an object within an image based on user input. The method may include the step of generating a mask representing the object region. The method may include the step of performing noise prediction using a noise processing network based on the image and the mask. The method may include the step of detecting a semantic region associated with the object based on the noise prediction result. The method may include the step of generating an extended mask including the semantic region. The method may include the step of displaying an edited image through a generative model based on the image and the extended mask.

[0005] According to one aspect of the present disclosure, an electronic device for editing an image based on generative AI may be provided. The electronic device may include at least one processor, a memory for storing instructions, and a display. By executing the instructions by the at least one processor, the electronic device may identify an object within an image based on user input. By executing the instructions by the at least one processor, the electronic device may generate a mask representing the object region. By executing the instructions by the at least one processor, the electronic device may perform noise prediction using a noise processing network based on the image and the mask. By executing the instructions by the at least one processor, the electronic device may detect a semantic region associated with the object based on the noise prediction result. By executing the instructions by the at least one processor, the electronic device may generate an extended mask including the semantic region. By executing the above instructions by the at least one processor, the electronic device can control the display to display an image edited through a generation model based on the image and the extended mask.

[0006] According to one aspect of the present disclosure, a computer-readable recording medium may be provided having a program recorded thereon for executing any one of the methods described above and below, wherein an electronic device and / or server generates an extended mask and generates and provides an edited image based on the extended mask.

[0007] FIG. 1 is a drawing for exemplarily illustrating an image editing operation performed by an electronic device according to one embodiment of the present disclosure.

[0008] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0009] FIG. 3 is a flowchart for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0010] FIG. 4 is a drawing illustrating the result of an electronic device according to one embodiment of the present disclosure editing an image using an extended mask.

[0011] FIG. 5 is a diagram illustrating the operation of an electronic device calculating a delta score according to one embodiment of the present disclosure.

[0012] FIG. 6 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure to generate an extended mask based on a delta score.

[0013] FIG. 7 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure to generate an edited image using an extended mask.

[0014] FIG. 8 is a drawing for illustrating an example of an electronic device according to one embodiment of the present disclosure editing an image using a generation model.

[0015] FIG. 9 is a diagram illustrating part of the process of an electronic device according to one embodiment of the present disclosure processing an image using a generation model.

[0016] FIG. 10 is a drawing for illustrating a semantic region associated with an object that is included in an extended mask according to one embodiment of the present disclosure.

[0017] FIG. 11 is a drawing for explaining the difference between a method of an electronic device according to one embodiment of the present disclosure and a conventional method.

[0018] FIG. 12a is a drawing for explaining the operation of changing an extended mask in an electronic device according to one embodiment of the present disclosure.

[0019] FIG. 12b is a drawing for illustrating examples of an electronic device according to one embodiment of the present disclosure changing an extended mask.

[0020] FIG. 13 is a drawing for illustrating an exemplary operation of an electronic device editing an image according to one embodiment of the present disclosure.

[0021] FIG. 14 is a drawing for illustrating an exemplary operation of an electronic device editing an image according to one embodiment of the present disclosure.

[0022] FIG. 15a is a drawing for illustrating an exemplary operation of an electronic device editing an image according to one embodiment of the present disclosure.

[0023] FIG. 15b is a drawing for illustrating an exemplary operation in which an electronic device according to one embodiment of the present disclosure provides an edited image.

[0024] FIG. 16 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0025] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail. In the present disclosure, the expression "at least one of a, b, or c" may refer to "a," "b," "c," "a and b," "a and c," "b and c," "all of a, b, and c," or variations thereof.

[0026] The terms used in this disclosure have been selected to be as widely used and general as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant explanatory sections. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0027] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art as described in this specification. Additionally, terms including ordinal numbers, such as "first" or "second," used in this specification may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another.

[0028] When a part of a specification is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "part" or "module" as used in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware or software, or as a combination of hardware and software.

[0029] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, parts unrelated to the description have been omitted from the drawings to clearly explain the present disclosure. Additionally, for convenience of explanation, different reference numbers have been used throughout the specification even for identical components.

[0030] The present disclosure will be described below with reference to the attached drawings.

[0031] FIG. 1 is a drawing for exemplarily illustrating an image editing operation performed by an electronic device according to one embodiment of the present disclosure.

[0032] In one embodiment, the electronic device may provide a function to edit an image using a generative model. The image editing function may be provided through a program or application running on the electronic device. The electronic device may edit an image using the generative model and provide it to the user.

[0033] A generative model can be an artificial intelligence model that utilizes a diffusion process. A generative model can be trained through a forward diffusion process that progressively adds noise and a back-diffusion process that predicts and removes noise; the trained generative model can generate new images by creating initial noise and using a back-diffusion process that predicts and removes noise from the initial noise.

[0034] For example, as illustrated in Fig. 1, the electronic device can perform an editing operation to remove objects within an image using a generative model.

[0035] In a general object removal operation, an input image (10) and a mask (20) are input into a generating model, and the result of removing an object from the input image (10) is output from the generating model. In this case, the generating model infers that new pixels can be generated in the area inside the mask (20) (e.g., the object area), while maintaining the identity of the input image (10) in the area outside the mask (20) (e.g., the area outside the object). However, since the mask (20) represents only the area corresponding to the object, if there is an area associated with the object within the image, e.g., a shadow, the image editing result of the generating model may result in the shadow still remaining or the object being regenerated based on the shadow.

[0036] The electronic device may use an extended mask (30) to remove objects more effectively. The extended mask (30) complements the mask (20) and may additionally include semantic regions associated with the object (e.g., shadow regions). The electronic device may generate the extended mask (30) using the input image (10) and the mask (20). The electronic device may generate an edited image (40) by inputting the input image (10) and the extended mask (30) into a generation model. When the electronic device uses the extended mask (30), a natural object removal result may be obtained in the edited image (40), in which the object and the semantic regions associated with the object (e.g., shadows) are removed together.

[0037] In FIG. 1, the image editing operation is described as an example of an object removal operation, but the image editing operation is not limited to this. The image editing operation may include a background change operation. For example, if an electronic device intends to change the background of an image while retaining an object, a natural background change result can be obtained in which not only is the object retained within the image with the changed background, but the area associated with the object (e.g., shadow) is also retained. In other words, the image editing operation of the electronic device may include all possible editing situations that can be utilized by using an extended mask (30) that includes a semantic area associated with an object in image editing operations such as naturally deleting or retaining an object.

[0038] The electronic device may be implemented as various types of devices that provide image editing functions. For example, the electronic device may include devices capable of displaying images through a display, such as smart TVs, smartphones, tablet PCs, laptop PCs, smart signage, head-mounted displays, and glasses-type displays, but is not limited thereto. As another example, the electronic device may be implemented as various types and forms of electronic devices capable of wired or wireless connection to a display. For example, the electronic device may include devices capable of displaying images through a connected display by wired or wireless connection, such as set-top boxes, desktop PCs, and servers, but is not limited thereto.

[0039] The specific operations of the electronic device editing an image will be described in more detail through the drawings and explanations described below.

[0040] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0041] Referring to FIG. 2, an electronic device (100) according to one embodiment may include a processor (110), memory (120), a display (130), and a communication interface (140).

[0042] The processor (110) can control the overall operations of the electronic device (100). The processor (110) can control the overall operations of the electronic device (100) by executing one or more instructions of a program stored in the memory (120) of the electronic device (100), thereby enabling the electronic device (100) to generate a semantically expanded mask and to edit and / or create an image using the expanded mask. For example, the processor (110) can handle various tasks including processing arithmetic and logical operations of processes running in the electronic device (100), managing main memory (122), and transferring data to storage (124).

[0043] The processor (110) may include a processing circuit. The processing circuit may include an operation unit that performs arithmetic and logical operations, a control unit that interprets commands and controls execution, a register that stores data, and a cache memory, but is not limited thereto.

[0044] The processor (110) may be composed of at least one of, for example, a Central Processing Unit (CPU), a Microprocessor, a Graphic Processing Unit (GPU), ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), an Application Processor (AP), a Neural Processing Unit (NPU), or an AI-dedicated processor designed with a hardware structure specialized for processing AI models, but is not limited thereto.

[0045] In one embodiment, there may be one or more processors (110). If there is one or more processors (110), the operations of the present disclosure may be performed by one or more processors by executing instructions and / or programs stored in memory (120) individually or collectively. If the method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one processor (110) or by a plurality of processors (110).

[0046] For example, when a first operation, a second operation, and a third operation are performed in an electronic device (100) by a method according to one embodiment, all of the first operation, the second operation, and the third operation may be performed by a first processor, or some of the first to third operations may be performed by a first processor (e.g., a general-purpose processor) and the remaining operations may be performed by a second processor (e.g., an artificial intelligence dedicated processor). Here, operations for training / inference of an artificial intelligence model may be performed by an artificial intelligence dedicated processor, which is an example of a second processor. However, the embodiments of the present disclosure are not limited thereto.

[0047] One or more processors (110) according to the present disclosure may be implemented as a single-core processor or as a multi-core processor. When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single core or by a plurality of cores included in one or more processors (110).

[0048] The detailed operations of the electronic device (100) performed by the processor (110) will be described in detail with reference to the drawings below.

[0049] The memory (120) can store data processed by the electronic device (100).

[0050] The memory (120) may include a main memory (122) that stores data currently being processed in the electronic device (100). The main memory (122) may store programs and data currently being executed by the processor (110) to allow the processor (110) to access the data quickly. The main memory (122) may include volatile memory such as, for example, RAM (Random Access Memory) or SRAM (Static Random Access Memory), but is not limited thereto.

[0051] The memory (120) may include storage (124) for permanently storing large amounts of data (e.g., programs, applications, system files, media files, etc.). The storage (124) may include non-volatile memory including at least one of, for example, a hard disk drive (HDD), a solid-state drive (SSD), an optical drive (e.g., a CD), a flash drive, a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), and a PROM (Programmable Read-Only Memory), but is not limited thereto.

[0052] The display (130) can output a video signal to the screen of the electronic device (100) under the control of the processor (110). For example, the display (130) can output to the screen a video signal processed during the process of the electronic device (100) providing an edited image, such as an image list for selecting an input image (original), image search results, selected images, mask and extended mask creation results, a graphic interface for verifying / modifying the extended mask, and image creation / editing results. The display (130) may include a touch panel. The touch panel may include one or more touch sensors that detect touch input. In one embodiment, user input related to an image editing operation may be obtained through the touch panel.

[0053] The communication interface (140) can perform data communication with another electronic device (e.g., a server) under the control of the processor (110). For example, the electronic device (100) can download an application for image editing from an online store. For example, the electronic device (100) can download a generative model trained for image editing from a server. For example, the electronic device (100) may perform image editing using a cloud-based generative model that transmits input data (e.g., an image and an extended mask) for image editing to a server and obtains an edited image from the server. Alternatively, the electronic device (100) may receive an original image for image editing from an external electronic device.

[0054] The communication interface (140) can perform data communication between an electronic device (100) and another electronic device (e.g., a server, an external electronic device, etc.) by using at least one of data communication methods including, for example, a wired LAN (e.g., Ethernet), a wireless LAN (e.g., Wi-Fi), a cellular network (e.g., 4G, 5G, etc.), Bluetooth, BLE (Bluetooth Low Energy), ZigBee, infrared communication (IrDA, infrared Data Association), NFC (Near Field Communication), RF communication, and various other types of known wireless / wired communication technologies. The communication interface (140) may include a communication circuit designed to use the aforementioned communication methods.

[0055] FIG. 3 is a flowchart for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0056] In operation 310, the electronic device (100) can identify an object within an image based on user input.

[0057] In one embodiment, the electronic device (100) can acquire an image. For example, the electronic device (100) may receive an image from an external device (e.g., another electronic device, server, etc.), capture an image using a camera included in the electronic device (100), or load an image stored in the storage of the electronic device (100). The electronic device (100) may display an image list for image selection for image loading and may provide an image search function. The electronic device (100) may display the acquired image. The acquired image may refer to the original image on which an image editing operation is to be performed and may be used as input data for a generation model for image editing.

[0058] The electronic device (100) may receive user input selecting an area within an image displayed on a screen. The area within the image may be an area corresponding to an object. User input may include, but is not limited to, touch-based input (e.g., short touch, long touch, drag, etc.), input device-based input (e.g., mouse, keyboard, etc.), voice-based input, etc. In one embodiment, the electronic device (100) may include a natural language processing engine to process the user's voice command.

[0059] The electronic device (100) can identify objects within an image based on user input. The electronic device (100) can analyze the area within the image where user input is received, convert voice commands into text, and identify the area within the image corresponding to the voice command using a model capable of text-image linkage processing (e.g., Contrastive Language-Image Pretraining; CLIP).

[0060] For example, an electronic device (100) can identify objects within an image based on a segmentation algorithm. The segmentation algorithm may be, for example, at least one of semantic segmentation, instance segmentation, and panoptic segmentation, but is not limited thereto.

[0061] For example, an electronic device (100) can identify objects within an image based on an object detection algorithm. Object detection can be performed using, for example, a Convolutional Neural Network (CNN), a Region-based CNN (R-CNN), a Transformer-based model, etc., but is not limited thereto.

[0062] In one embodiment, the electronic device (100) can automatically identify objects within an image. For example, the electronic device (100) can receive user input to execute an image editing function and select an image. When the image corresponding to the user input is identified, the electronic device (100) can detect objects within the image using the object identification method described above.

[0063] In operation 320, the electronic device (100) can generate a mask representing an object region.

[0064] In one embodiment, the electronic device (100) may generate a mask representing an area corresponding to an object identified within an image. The mask may be a binary mask. The electronic device (100) may generate a mask for the identified object area by setting the pixel values ​​of the area corresponding to the object to 1 and the pixel values ​​of the remaining area to 0. The electronic device (100) may apply certain processing techniques to generate the mask. For example, the electronic device (100) may improve the quality of the mask by applying techniques such as morphological operations, Gaussian blur, and thresholding.

[0065] In operation 330, the electronic device (100) can perform noise prediction using a Noise Processing Network based on an image and a mask.

[0066] A noise processing network may be a neural network that performs the role of predicting noise or removing noise to restore or generate data. A noise processing network that performs noise prediction in operation 330 may be a noise processing network included in a generative model trained for object removal. A generative model trained for object removal may be a diffusion model that generates images based on a diffusion process.

[0067] A noise processing network can be referred to as a Noise Prediction Network in that it predicts noise, or as a Denoising Network in that it removes noise. Additionally, it may be referred to as a Diffusion Network in that it handles the reverse diffusion process used to predict and remove noise. A noise processing network can be implemented using U-Net, a neural network architecture capable of performing noise prediction and removal tasks, but it is not limited to this.

[0068] In one embodiment, the electronic device (100) can perform noise prediction with or without a mask on the image.

[0069] For example, the electronic device (100) can perform unconditional prediction that predicts noise using only the image without mask conditions. In unconditional prediction, the noise processing network can predict noise in the entire image data using only the noised image. Since the noise processing network is a noise processing network of a generative model trained for object removal, the result of the unconditional prediction may mean predicting noise mixed in the image from the noised image to remove objects from the image.

[0070] For example, the electronic device (100) can perform conditional prediction by applying a mask to an image as a condition to predict noise. In conditional prediction, a noise processing network can predict noise using the image with added noise and the mask. Since the noise processing network is a noise processing network of a generative model trained for object removal, the result of the conditional prediction may mean predicting noise mixed in the mask area of ​​the image from the image with added noise to remove the object specified by the mask from the image. For example, the object area of ​​the mask (M=1) may undergo noise prediction to remove the object through noise prediction and removal, while the non-object area of ​​the mask (M=0) may retain data without noise prediction and removal. Alternatively, when performing conditional prediction, the prediction strength may be applied relatively higher to the object area of ​​the mask than to the non-object area of ​​the mask.

[0071] Generally, the inverse diffusion process of a noise processing network performed in the inference task of a generative model is carried out by iterating noise prediction and removal tasks through multiple time steps. For example, from the initial time step t=T to t=0, noise prediction and removal tasks are repeated at each time step, and noise is gradually removed. In other words, by iterating the noise prediction and removal tasks over the time steps, a state where noise is finally removed can be reached at t=0.

[0072] In one embodiment, noise prediction may be performed in a single step. For example, the electronic device (100) may perform only noise prediction corresponding to the initial time step t=T. In one embodiment, noise prediction may be performed fewer times than noise prediction for the entire time step. For example, the electronic device (100) may perform noise prediction (and noise removal) corresponding to the intermediate time step t=n (n>0) from the initial time step t=T.

[0073] In other words, since operation 330 is intended to obtain data that the electronic device (100) uses to extend the mask in subsequent operations 340 and 350, it is not necessary to perform all noise prediction and removal operations for the entire time step to predict the final noise. The electronic device (100) is characterized by performing noise prediction and removal operations for only some of the time steps among the entire time steps.

[0074] In operation 340, the electronic device (100) can detect a semantic region associated with an object based on the noise prediction result.

[0075] In one embodiment, the electronic device (100) may calculate a delta score based on a conditional prediction result and an unconditional prediction result, which are different types of noise prediction. The delta score is an indicator representing the difference between two values ​​and may represent the difference between the conditional prediction and the unconditional prediction.

[0076] The electronic device (100) can detect semantic regions associated with an object based on a delta score. Regions where a difference value exists based on the delta score may include object regions and semantic regions associated with the object.

[0077] In the present disclosure, a semantic region associated with an object may refer to an area within a semantic range that is affected by the object within an image. For example, semantic elements corresponding to a semantic region may include incidental effects caused by the object (e.g., shadows, light reflections, etc.), elements within the image that are connected to or dependent on the object (e.g., ski poles when the object is a skier, etc.), and areas that are actually included in the object but are unintentionally missing during object detection. However, elements that are independent of the original object, such as another object next to the object, may not be included in the semantic region. Nevertheless, the semantic region associated with an object is not defined by the examples described above. In the present disclosure, the semantic region associated with an object may refer to an area detected by a delta score.

[0078] In other words, the semantic region associated with an object may represent the result of a generative model trained for object removal (hereinafter referred to as the object removal model) receiving a mask as input and predicting noise in the object region and the surrounding region associated with the object (e.g., shadows) to remove the object corresponding to the mask. To remove the (selected) object within an image, the object removal model is trained to perform an inpainting task in which pixel data is generated and filled by repeatedly predicting and removing noise at time steps, treating the region where the object existed as a missing region. Through this process, the object removal model can predict noise in the object and the surrounding region associated with the object.

[0079] The electronic device (100) can perform conditional and unconditional predictions with a mask as a condition using a noise processing network of a generative model trained for object removal, and calculate a delta score by comparing each prediction result. Consequently, the region detected by the delta score may include an object region and a semantic region associated with the object.

[0080] In operation 350, the electronic device (100) may generate an extended mask that includes a semantic region. The extended mask may mean that the mask region is extended compared to the mask representing the object region obtained in operation 320. The extended mask may include an object region and a semantic region associated with the object.

[0081] In one embodiment, the electronic device (100) may apply one or more post-processing techniques to a delta score representing the difference between a conditional prediction and an unconditional prediction. The one or more post-processing techniques may include, for example, morphological operations (e.g., dilation operations), Gaussian blur, thresholding, etc., but are not limited thereto.

[0082] In operation 360, the electronic device (100) can display an image edited through a generated model based on an image and an extended mask.

[0083] In one embodiment, the electronic device (100) may input an image and an extended mask into a generation model to obtain an edited image. The image editing operation may be, for example, an object removal operation or a background editing operation, but is not limited thereto. Specifically, the object removal operation may be to naturally remove an object within the image based on an area corresponding to the extended mask, and the background editing operation may be to change the background while naturally maintaining an object within the image based on an area corresponding to the extended mask.

[0084] The electronic device (100) can determine the type of operation to edit the image. For example, the electronic device (100) can determine the type of operation for image editing based on user input selecting the type of operation. The electronic device (100) can select a generative model corresponding to the type of image editing operation. For example, if an object removal operation is selected, the electronic device (100) can select an object removal model. In this case, the object removal model may be an object removal model including a noise processing network mentioned in the aforementioned operation 330. For example, if a background editing operation is selected, the electronic device (100) can select a background editing model. In one embodiment, the generative model may be a general-purpose model trained to handle multiple operations (e.g., a multi-task model). In this case, an operation corresponding to the type of operation selected by the multi-task model (e.g., object removal, background editing, etc.) may be performed to obtain an edited image.

[0085] Generative models (e.g., object removal models, background editing models, multitasking models, etc.) may be diffusion models that utilize a diffusion process for image editing and generation tasks. Generative models may be implemented through variations or extensions of neural networks for processing diffusion processes. Generative models may include, for example, U-nets, but are not limited thereto.

[0086] The electronic device (100) can display the edited image on a screen. Alternatively, the electronic device (100) may transmit the edited image to another electronic device.

[0087] FIG. 4 is a drawing illustrating the result of an electronic device according to one embodiment of the present disclosure editing an image using an extended mask.

[0088] Referring to FIG. 4, the results of editing the original image (400) by the electronic device (100) in two ways are described. Specifically, the difference between a mask-based edited image (415), in which objects within the original image (400) are removed based on the original image (400) and a mask (410), and an extended mask-based edited image (440), in which a delta score (420) and an extended mask (430) are generated based on the original image (400) and objects within the original image are removed based on the original image (400) and the extended mask (430).

[0089] The electronic device (100) can acquire an original image (400). In the example of FIG. 4, an object and a shadow of the object exist within the original image (400).

[0090] A mask (410) is used for the task of removing an object from an original image (400) in the conventional manner. An electronic device (100) can identify an object within the original image (400) and generate a mask (410) representing the object region. Since the mask (410) is based on the object identification result, the shadow region of the object may not be included in the mask (410). The electronic device (100) can input the original image (400) and the mask (410) into a generation model to obtain a mask-based edited image (415). In this case, the generation model performs an inference task to remove an object from the original image (400) by referencing the mask (410). Consequently, an unnatural result may appear in the mask-based edited image (415) obtained using the mask (410), in which the object has been removed but the object's shadow still exists.

[0091] In one embodiment, the electronic device (100) can calculate a delta score (420) for generating an extended mask (430) based on an original image (400) and a mask (410). The delta score (420) may represent the difference between an unconditional prediction, in which noise prediction is performed using a noise processing network with only the original image (400), and a conditional prediction, in which noise prediction is performed using a noise processing network with the mask (410) applied as a condition to the original image (400). The electronic device (100) can obtain the delta score (420) by calculating the difference between the conditional prediction and the unconditional prediction.

[0092] The electronic device (100) can generate an extended mask (430) by applying one or more post-processing steps to a delta score (420). The extended mask (430) includes masking information for a wider area than the mask (410), and in the example of FIG. 4, may include an object area and a shadow area of ​​the object. When the extended mask (430) is obtained, the electronic device (100) can input the original image (400) and the extended mask (430) into a generation model to obtain an extended mask-based edited image (440). In this case, the generation model performs an inference operation to remove the object and associated area within the original image (400) by referencing the extended mask (430). As a result, the object and the object's shadow are removed from the extended mask-based edited image (440) obtained using the extended mask (430), so a natural result can be obtained.

[0093] That is, when the electronic device (100) edits an image using an extended mask (430), a more effective and natural image editing result can be obtained. Specific operations such as the electronic device (100) calculating a delta score (420) based on the original image (400) and the mask (410), generating an extended mask (430) based on the delta score (420), and generating an extended mask-based edited image (440) based on the extended mask (430) are further described with reference to FIGS. 5 to 8, which will be described later.

[0094] FIG. 5 is a diagram illustrating the operation of an electronic device calculating a delta score according to one embodiment of the present disclosure.

[0095] In one embodiment, the electronic device (100) can perform different types of noise prediction based on an image and a mask. Noise prediction can be performed using a noise processing network (500). The noise processing network (500) used for calculating the delta score may be a noise processing network of a generative model trained for object removal. In other words, the electronic device (100) does not need to separately train the noise processing network (500) to perform noise prediction for calculating the delta score, and the electronic device (100) may use a noise processing network of a generative model already trained to perform a defined task (e.g., an object removal task).

[0096] When a generative model is trained to perform an object removal task, the parameters of the noise processing network (500) are optimized so that the process of predicting and removing noise from an image is repeated, and finally, the object can be removed from the image. In this case, the generative model may use mask conditioning, which applies a mask as a condition to remove the object, and classifier-free guidance (CFG) methods. The CFG method uses a combination of conditional prediction and unconditional prediction, where conditional prediction predicts noise under a condition (e.g., a mask), and unconditional prediction predicts noise without a condition. By using the CFG method, the parameters of the noise processing network (500) can be optimized during the training process of the generative model for object removal, by adjusting the trade-off between the conditional likelihood, which is the probability of an image being generated under a condition, and the unconditional likelihood, which is the probability of an image being generated without a condition.

[0097] In other words, if the noise processing network (500) is a noise processing network of a generative model trained for object removal, when the electronic device (100) performs noise prediction using the noise processing network (500), the predicted noise is output to remove objects within the image. The electronic device (100) can perform unconditional prediction and conditional prediction, respectively, using the noise processing network (500).

[0098] For example, an electronic device (100) can input an unconditional input (510) into a noise processing network (500) to obtain an unconditional prediction (515). The electronic device (100) can obtain a noised image by adding initial noise to an image. The initial noise may be Gaussian noise following a standard normal distribution (0, 1). In the unconditional prediction, the unconditional input (510) may only include the noised image.

[0099] For example, an electronic device (100) can input a conditional input (520) into a noise processing network (500) to obtain a conditional prediction (525). In the conditional prediction, the conditional input (520) may additionally include a conditional mask in the image with added noise.

[0100] The electronic device (100) can calculate a delta score (530) representing the difference between an unconditional prediction (515) and a conditional prediction (525). In one embodiment, the electronic device (100) can obtain the delta score (530) by taking the absolute value of the difference between the unconditional prediction (515) and the conditional prediction (525).

[0101] In one embodiment, when predicting noise using a noise processing network (500), conditional prediction and unconditional prediction can be performed fewer times than the noise prediction over all time steps. In an inference operation using a generative model, the noise prediction and removal operations for all time steps are repeated to produce a final result. For example, from the initial time step t=T to t=0, the noise prediction and removal operations are repeated at each time step to gradually remove noise, and at t=0, the noise is finally removed. However, noise prediction for calculating the delta score (530) does not need to be performed over all time steps. The electronic device (100) can obtain data for mask expansion with only a small amount of computation by performing conditional prediction and unconditional prediction fewer times than the noise prediction over all time steps. In one embodiment, the electronic device (100) can perform conditional prediction and unconditional prediction as a single-step noise prediction. If conditional prediction and unconditional noise prediction are each performed only once for a single time step, the amount of computation can be minimized.

[0102] The electronic device (100) can generate an extended mask by applying one or more post-processing steps to the delta score (530). The operation of generating the extended mask will be further described with reference to FIG. 6.

[0103] FIG. 6 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure to generate an extended mask based on a delta score.

[0104] In one embodiment, the electronic device (100) can obtain an extended mask (620) based on a delta score (610). The electronic device (100) can generate an extended mask (620) including an object region and a semantic region associated with the object by applying one or more post-processing steps to the delta score (610).

[0105] For example, the electronic device (100) may apply morphological operations to the delta score (610). Morphological operations may include dilation, erosion, opening, and closing operations. Specifically, the electronic device (100) may apply a dilation operation to the delta score (610) to expand the region where the value of the delta score (610) exists outward, or to fill in holes or gaps between regions. However, it is not limited to the examples described above, and the electronic device (100) may apply other operations of morphological operations to obtain an expanded mask (620).

[0106] For example, the electronic device (100) can apply a Gaussian blur to the delta score (610). The Gaussian blur smooths the boundary areas represented by the values ​​of the delta score (610).

[0107] For example, the electronic device (100) may apply threshold processing to the delta score (610). Since the delta score (610) may have various values, the electronic device (100) may binarize the pixel values ​​of the delta score (610) to 0 or 1 based on a defined threshold. Binarization may, for example, binarize the pixel values ​​to 0 or 1 based on a threshold of 0.2, but is not limited thereto.

[0108] Meanwhile, the processing order of one or more of the aforementioned post-processing techniques may be adjusted to obtain an optimal expanded mask (620). For example, the electronic device (100) may remove noise through a dilation operation, generate a smooth boundary through a Gaussian blur, and then obtain an expanded mask (620) through threshold processing. Alternatively, for example, the electronic device (100) may first apply binarization to the delta score (610), remove noise through a dilation operation, and then apply a Gaussian blur to smooth the boundary.

[0109] FIG. 7 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure to generate an edited image using an extended mask.

[0110] In one embodiment, the electronic device (100) may perform an image editing operation based on an image (710) and an extended mask (720) using a generative model including a noise processing network (700). The noise processing network (700) performing the image editing operation of FIG. 7 may be the same as or different from the noise processing network (500) used to calculate the delta score (530) of FIG. 5. For example, if the type of image editing operation is 'object removal' and a generative model trained for object removal is used, the noise processing network (700) of the generative model may be the same as the noise processing network (500) of FIG. 5. However, even in this case, the noise processing network (500) of FIG. 5 and the noise processing network (700) of the generative model may be implemented differently. For example, if the type of image editing operation is 'background change' and a generative model trained for background change is used, the noise processing network (700) of the generative model may be different from the noise processing network (500) of FIG. 5. Or, for example, if the generating model is a general-purpose model capable of multitasking image editing such as 'object removal' and 'background change', the noise processing network (700) of the generating model and the noise processing network (500) of FIG. 5 may be implemented identically or differently.

[0111] The noise processing network (700) can receive an image (710) and an extended mask (720) as input and repeat the process of predicting and removing noise. The process of repeating noise prediction and removal can be performed from the entire time step t=T to t=0. In the initial time step, the noise processing network (700) can generate initial noise and add it to the image (710), and predict and remove noise from the image (730) with added noise using the extended mask (720) as a condition. As the time step progresses, noise is gradually removed by predicting and removing noise while referring to the condition at each time step, and finally, an edited image (740) can be obtained. The edited image (740) can represent the result of removing objects within the image (710).

[0112] The overall operation of the generative model including the noise processing network (700) is described further with reference to FIG. 8.

[0113] FIG. 8 is a drawing for illustrating an example of an electronic device according to one embodiment of the present disclosure editing an image using a generation model.

[0114] In explaining Fig. 8, the generative model is described as a diffusion model that utilizes a diffusion process. A diffusion model can process data in latent space by compressing the original data into a lower dimension. However, this is merely an example for the convenience of explanation, and a diffusion model can also process data in pixel space without transforming the data into latent space.

[0115] The inference process of the generative model is a process of generating a new image using an input (e.g., an image (800)) and a condition (e.g., an extended mask (802)). The generative model can obtain a final image through a back-diffusion process that generates initial noise (e.g., initial latent data), predicts noise starting from the sampled initial noise at time steps, and repeats the process of removing the predicted noise.

[0116] In one embodiment, the electronic device (100) may acquire an image (800). The electronic device (100) may apply a predetermined preprocessing (e.g., resizing) to the image (800) to use it as input data for a diffusion model. The electronic device (100) may identify an object within the image (800) to generate a mask (801) representing the object region, and may generate an extended mask (802) based on the image (800) and the mask (801). The extended mask (802) may be generated based on a delta score representing the difference between a conditional prediction based on the mask (801) and an unconditional prediction. Since the specific operation of the electronic device (100) generating the extended mask (802) has been described above in the description of the previous drawings, a repetitive description is omitted.

[0117] The encoder (810) can convert the image (800) to generate latent data (803). The encoder (810) can convert the image (800) into latent data that can be processed in a latent space and add noise. The initial noise at time step t=T can be obtained by sampling from a standard normal distribution N(0,1).

[0118] The backdiffusing process, which is the inference process of a generative model, is a process of gradually removing noise from the time step t=T to t=0. In other words, the initial latent data And, the latent data at any time step t between t=T and t=0 am.

[0119] The generative model can predict noise at each time step. At each time step, one diffusion process (backward diffusion process) can be performed. Noise prediction can be performed by a noise processing network (820). The noise processing network (820) can be implemented based on a U-Net architecture for processing noise prediction tasks, but is not limited thereto.

[0120] At each time step, the noise (804) predicted through noise prediction can be obtained. Noise prediction is a conditional prediction and unconditional prediction It may include. Conditional prediction is predicting noise under a condition (e.g., an extended mask (802)), and unconditional prediction is predicting noise without a condition. The condition integration module (830) can pass the extended mask (802) to the noise processing network (820). In the noise prediction process, guidance without a classifier (CFG) that utilizes a combination of conditional and unconditional prediction may be used. This can be expressed as follows.

[0121]

[0122] In the above formula, is the combined predicted noise (804), and is the predicted noise given condition c (briefly, ) and, is predicted noise in the absence of conditions (briefly, ) and, is a guidance scale indicating the degree of condition reflection. And, the condition image corresponding to condition c is an extended mask (802).

[0123] When the predicted noise (804) is obtained at each time step t, the generative model can obtain the noise-removed potential data (805) by removing the predicted noise (804) from the current potential data (803). The noise-removed potential data (805) can be used again in the noise prediction process at the next time step t-1. The process of adding, predicting, and removing noise at each time step can be controlled by the scheduler of the generative model.

[0124] The generation model can obtain final potential data by repeating noise prediction and removal from t=T to t=0. The final potential data can be converted into an edited image (806), which is the final image, by a decoder (840).

[0125] FIG. 9 is a diagram illustrating part of the process of an electronic device according to one embodiment of the present disclosure processing an image using a generation model.

[0126] In one embodiment, the electronic device (100) may perform conditional prediction and unconditional prediction fewer times than the noise prediction of the entire time steps of the generative model to generate an extended mask. For example, the electronic device (100) may perform conditional prediction and unconditional prediction as a single-step noise prediction. This utilizes the characteristics of the generative model trained for object removal (hereinafter, object removal model).

[0127] A trained object removal model is trained to perform the task of removing objects. For example, the object removal model may be a model prepared to perform the task of removing objects after pre-training and / or fine-tuning training has been performed and performance validation has been conducted.

[0128] In the inference process of the trained object removal model, the object removal model receives an image (900) and a mask (910) corresponding to the object as input and performs the task of removing the object within the image (900). In this case, a back-diffusion process may be used. The back-diffusion process is a process of repeating noise prediction and removal over all time steps, and is a process of gradually removing noise from time step t=T to t=0. This process requires a sufficient number of iterations, and by repeating each time step, a result in which the object is removed is output. Here, a sufficient number of iterations may mean, for example, 100 times (T=100), 1000 times (T=1000), etc., but is not limited thereto. However, the task of the electronic device (100) expanding the mask (910) using the trained object removal model does not require performing all time steps included in the inference process of the object removal model.

[0129] For example, iteration 2 (920) visualizes the result of repeating the noise prediction and removal of the time step twice. And iteration 3 (930) shows the noise prediction and removal of the time step three times, and iteration 4 (940) shows the noise prediction and removal of the time step four times, representing only some of the initial time steps among the total time steps T of the inference process.

[0130] Here, the output image represents the result of restoring the image after repeating noise prediction and removal for 2, 3, and 4 time steps. The mask augmentation result represents the result of augmenting the mask (910) based on the difference between the conditional prediction and the unconditional prediction at the corresponding time step. In this case, since the repetition of the time steps is insufficient, the output image may show an incomplete object removal result. For example, the output image may show a result where the object and the object's shadow are not completely removed. However, the mask augmentation result may show a result where the object region and the semantic region associated with the object are detected.

[0131] This means that the trained object removal model has the ability to infer areas that need to be modified to remove objects within the image (900) (e.g., object areas and shadow areas). Therefore, if the time steps repeated in the inference process of the trained object removal model are insufficient, the object removal result is incomplete, but it is possible to supplement the mask in the initial time steps of the inference process based on the result of predicting noise for object removal. In other words, since the electronic device (100) aims to generate an extended mask that includes semantic areas associated with objects, it can obtain an extended mask with a small amount of computation using only the initial time steps without needing to perform all operations of the entire time steps.

[0132] FIG. 10 is a drawing for illustrating a semantic region associated with an object that is included in an extended mask according to one embodiment of the present disclosure.

[0133] In one embodiment, the electronic device (100) may generate an extended mask to perform object-based image editing operations using a generative model. The extended mask may include a semantic region associated with an object. A semantic region associated with an object may refer to a region within a semantic range that is affected by the object within the image.

[0134] For example, semantic elements corresponding to a semantic domain may include incidental effects caused by an object, elements within the image that are connected to or dependent on the object, and regions that are actually included in the object but are unintentionally missing during object detection. However, elements that are independent of the original object, such as another object next to the object, may not be included in the semantic domain.

[0135] For example, referring to the first image, an object within the first image may be a skier (1000). The electronic device (100) may detect the skier (1000) by performing an object detection operation on the first image, or may detect the skier (1000) based on user input selecting an intended area within the first image.

[0136] As an example of the result of an electronic device (100) detecting an object in a first image, a skier (1000) may be detected in the first image, and consequently, the object refers to the skier (1000). In this case, the left pole (1002), the right pole (1004), and the shadow (1006) may represent semantic regions associated with the object. Specifically, the left pole (1002) and the right pole (1004) represent elements that are connected to the skier (1000) or are in a subordinate relationship with respect to the skier (1000). And, the shadow (1006) represents an incidental effect caused by the skier (1000).

[0137] As another example of the result of the electronic device (100) detecting an object in the first image, a skier (1000), a left pole (1002), and a right pole (1004) in the first image may be detected as a single object. In this case, a shadow (1006), which is an area representing a side effect caused by the object, may represent a semantic area associated with the object.

[0138] As another example of the result of an electronic device (100) detecting an object in a first image, a skier (1000) may be detected in the first image, and only one of the left pole (1002) or the right pole (1004) may be detected. In this case, the remaining pole, which is an area unintentionally missing during object detection, and the shadow (1006), which is an area representing a side effect caused by the object, may represent semantic areas associated with the object.

[0139] In other words, the semantic region associated with the object is not defined based on specific criteria, but rather by a delta score representing the difference between the unconditional prediction and the conditional prediction using a noise processing network based on a mask corresponding to the object. Since the electronic device (100) uses a noise network of a generative model trained for object removal when generating an extended mask, calculating the difference between the unconditional prediction and the conditional prediction of the mask corresponding to the object region results in the detection of not only the object but also the semantic regions associated with the object. Consequently, the extended mask may include regions corresponding to the left pole (1002), the right pole (1004), and the shadow (1006), which are semantic regions associated with the object.

[0140] For example, referring to the second image, the objects within the second image may be a person (1010) and a dog (1020). When there are multiple objects within an image, such as in the second image, the electronic device (100) may identify the objects based on user input. For example, the electronic device (100) may detect the person (1010) based on user input and set it as a reference object. In this case, the person's shadow (1012) may represent a semantic region associated with the object. Furthermore, the dog (1020) and the dog's shadow (1022), which are elements independent of the reference object (1010), may not be included in the semantic region associated with the object.

[0141] In other words, the semantic domain associated with an object refers to a domain that extends based on a reference object, and categories, rules, etc., for the semantic domain are not defined. Specifically, a person shadow (1012) and a dog shadow (1022) are both shadows, but being a shadow does not necessarily make it a semantic domain; rather, the domain that extends in association with the object based on the reference object, the person (1010), is defined as the semantic domain.

[0142] Meanwhile, although a person (1010) was described as the reference object in the second image, the reference object does not necessarily have to be a single object. For example, multiple objects such as a person (1010) and a dog (1020) may be detected and set as reference objects. In this case, the semantic region associated with the object may include a person shadow (1012) and a dog shadow (1022).

[0143] FIG. 11 is a drawing for explaining the difference between a method of an electronic device according to one embodiment of the present disclosure and a conventional method.

[0144] Referring to FIG. 11, an example of a conventional method (1110) is illustrated. The conventional method (1110) is based on an anomaly detection method and may involve inputting an input image (1111) into a reconstruction model to obtain an output image (1112). The reconstruction model may be a model that removes objects within the input image (1111) and performs an inpainting operation to fill in pixels in the loss area where the objects were removed, but is not limited thereto. The reconstruction model may be a model based on a diffusion model utilizing a diffusion process and may be a model based on a Variational Autoencoder (VAE), but is not limited thereto. The reconstruction model may be trained based on a ground truth mask representing the area where object removal is required.

[0145] In the existing method (1110), when detecting objects and semantic regions associated with objects, regions requiring object removal are detected based on the difference between the input image (1111) and the output image (1112). For example, an anomaly detection-based mask (1113) indicating regions requiring object removal may be obtained. In this case, the restoration model requires a relatively large amount of computation because it undergoes the entire image generation process of generating an output image (1112) from an input image (1111). Furthermore, since the existing method (1110) is an End-to-End method that receives an input image (1111) and generates a result output image (1112), calculation errors may occur in the background region as processes such as noise prediction for image generation are performed for the background region in addition to the object region within the image.

[0146] In contrast, according to the method (1120) of the present disclosure, the electronic device (100) can calculate an unconditional prediction (1123) and a conditional prediction (1124) through a noise processing network based on an input image (1121) and a mask (1122), and generate an extended mask (1125) based on the difference between the conditional prediction (1124) and the unconditional prediction (1123). In this case, the extended mask (1125) can be generated with less computational power than the existing method (1110). The electronic device (100) can generate the extended mask (1125) with a reduced computational cost through the method (1120) of the present disclosure, with fewer NFEs (Number of Function Evaluations). For example, if only a single-time step inverse diffusion process (NFE: 1) is performed when calculating the unconditional prediction (1123) and only a single-time step inverse diffusion process (NFE: 1) is performed when calculating the conditional prediction (1124), the NFE for the method (1120) of the present disclosure may be 2. And, since the method (1120) of the present disclosure performs conditional / unconditional prediction based on the mask (1122), noise prediction is mainly performed on the region where the condition is applied or not applied. Therefore, the occurrence of calculation error can be suppressed for the background region excluding the object region and the semantic region associated with the object.

[0147] FIG. 12a is a drawing for explaining the operation of changing an extended mask in an electronic device according to one embodiment of the present disclosure.

[0148] In operation 1202, the electronic device (100) can display an image and an extended mask. Operation 1202 can be performed after operation 350, which generates the extended mask of FIG. 3, has been performed.

[0149] The electronic device (100) may display an expanded mask on the screen of the electronic device (100) so that a user can view the expanded mask. For example, the electronic device (100) may display an image and the expanded mask together, display the expanded mask overlaid on the image, or display only the expanded mask. In one embodiment, when displaying the expanded mask, the electronic device (100) may display only the expanded area in addition to the object area by visually processing it differently.

[0150] The electronic device (100) may display an extended mask so that the user can verify the extended mask and may ask the user whether to approve the extended mask. For example, the electronic device (100) may output elements such as text, buttons, or popups on the screen, visually or audibly, asking whether the extended mask area is correct, or output a sound such as voice guidance through a speaker asking whether the extended mask area is correct. For example, in the case of object removal, the electronic device (100) may ask, "Would you like to delete this area?" and in the case of object movement, it may ask, "Would you like to move this area?". If the user approves the extended mask through user input such as touch input or voice input, the electronic device (100) may perform operation 1206. If the user does not approve the extended mask, the electronic device (100) may perform operation 1204.

[0151] In operation 1204, the electronic device (100) can change the mask based on user input. Changing the mask may include adding or deleting mask regions. Changing the mask may include creating a mask for a new region different from the previously created mask region.

[0152] In one embodiment, the electronic device (100) may add a mask region based on user input. For example, the electronic device (100) may regenerate an expanded mask to include a new object and a semantic region associated with the new object based on user input selecting a new object. Or, for example, the electronic device (100) may regenerate an expanded mask to include a selected region based on user input selecting a region associated with an existing object. However, the cases in which the electronic device (100) adds an expanded mask region are not limited to the examples described above.

[0153] In one embodiment, the electronic device (100) may delete a portion of an extended mask based on user input. For example, the electronic device (100) may regenerate an extended mask such that a selected portion is excluded based on user input selecting a portion of the extended mask. Alternatively, for example, the electronic device (100) may regenerate an extended mask such that a selected object and a semantic area associated with the selected object are excluded based on user input selecting one of a plurality of objects. However, the cases in which the electronic device (100) deletes an extended mask area are not limited to the examples described above.

[0154] In one embodiment, the electronic device (100) may change the extended mask area based on user input. For example, the electronic device (100) may remove the extended mask corresponding to the existing object and create an extended mask corresponding to the other object based on user input selecting an object other than the object represented by the extended mask. However, the cases in which the electronic device (100) changes the extended mask area are not limited to the examples described above.

[0155] In operation 1206, the electronic device (100) can display an image edited through a generated model based on an image and an extended mask. Since operation 1206 corresponds to operation 360 of FIG. 3, a repetitive description is omitted for brevity.

[0156] FIG. 12b is a drawing for illustrating examples of an electronic device according to one embodiment of the present disclosure changing an extended mask.

[0157] Referring to FIG. 12b, the image (1210) contains objects 'vase' and 'cube'. In describing FIG. 12b, the extended mask (1212) is described as including the object 'vase' region and the semantic region 'vase shadow' associated with the object among the objects in the image (1210).

[0158] In one embodiment, the electronic device (100) may display an image (1210) and an extended mask (1212). For example, the electronic device (100) may display the image (1210) and the extended mask (1212) generated for the image (1210) together. Or, for example, the electronic device (100) may display an image (1214) with the extended mask (1212) overlaid on the image (1210) so that the object detected within the image and the semantic region associated with the object can be visually identified. Or, for example, the electronic device (100) may display only the extended mask (1216). In the examples described above, the electronic device (100) may also display an extended mask (1218) that distinguishes the semantic region (1219) associated with the object.

[0159] In one embodiment, the electronic device (100) may receive user input to change an extended mask. The electronic device (100) may edit an image using a generation model based on whether the extended mask has been changed or not.

[0160] For example, if the extended mask is not changed, the electronic device (100) can input the image (1220) and the extended mask (1222) into a generating model. The generating model edits the image based on the image (1220) and the extended mask (1222). As a result, an edited image (1224) can be obtained in which the object corresponding to the extended mask (1222) and the semantic region associated with the object are removed. As illustrated in the example, the edited image (1224) may show the result of removing the vase and the vase shadow within the image (1220).

[0161] For example, when the extended mask is changed, the electronic device (100) may input the image (1230) and the changed extended mask (1232) into a generating model. The generating model edits the image based on the image (1230) and the changed extended mask (1232). As a result, an edited image (1234) may be obtained in which objects corresponding to the changed extended mask (1232) and semantic regions associated with the objects are removed. As illustrated in the example, the edited image (1234) may show a result in which the vase, vase shadow, cube, and cube shadow within the image (1230) are removed.

[0162] FIG. 13 is a drawing for illustrating an exemplary operation of an electronic device editing an image according to one embodiment of the present disclosure.

[0163] In one embodiment, the electronic device (100) may receive user input selecting an object within an image (1310). The electronic device (100) may identify the object corresponding to the user input and generate a mask (1320) corresponding to the object area.

[0164] The electronic device (100) can generate an expanded mask (1330) through semantic mask expansion based on an image (1310) and a mask (1320). Semantic mask expansion may include a process of calculating the difference between a conditional prediction and an unconditional prediction that predicts noise based on the input image (1310) and the mask (1320), and as a result, the expanded mask (1330) may include a shadow region, which is a semantic region associated with an object.

[0165] The electronic device (100) may display an extended mask (1330). For example, the electronic device (100) may display an image (1310) and the extended mask (1330) together, display an image with the extended mask (1330) overlaid on the image (1310), or display only the extended mask (1330). Additionally, in the examples described above, the electronic device (100) may display the semantic region associated with an object within the extended mask (1330) differently so that it can be distinguished visually. The electronic device (100) may display the extended mask (1330) and receive user input to approve or change the extended mask (1330). For example, the electronic device (100) may provide a popup such as “Would you like to select this region?” to allow the user to approve a region of the extended mask (1330). If the user does not approve the area of ​​the extended mask (1330) (e.g., select No), the electronic device (100) may receive additional user input to change the area of ​​the extended mask (1330).

[0166] The electronic device (100) can perform image editing operations using an extended mask (1330). The electronic device (100) can determine the type of image editing operation to be performed to edit the image (1310) based on the area corresponding to the extended mask (1330). For example, the electronic device (100) can receive user input selecting the image editing operation type. The electronic device (100) can select a generative model trained for image editing corresponding to the operation type.

[0167] For example, the electronic device (100) can change the style of the background using a generative model. Changing the background style may be a task of changing the style of color, texture, season, weather, etc. while maintaining the basic elements that constitute the image (e.g., composition, structure, identity, etc.). In one embodiment, the electronic device (100) may select a general-purpose generative model to perform the background style change task. In one embodiment, the electronic device (100) may select a generative model trained for image style change to perform the background style change task.

[0168] The electronic device (100) can input an image (1310) and an extended mask (1330) into the generation model. Additionally, the electronic device (100) can input an edit prompt into the generation model to change the style of the image. In this case, the generation model can use the edit prompt as a text condition, which is a condition granted for image generation. The edit prompt may be text representing instructions or commands for image editing. The edit prompt may be obtained based on user input. Alternatively, the edit prompt may be selected from a set of edit prompts stored in the electronic device (100). For example, any edit prompt may be selected from the set of edit prompts, a predetermined edit prompt may be selected, or any one of the edit prompts may be selected by user input.

[0169] For example, the edit prompt may be text such as the description of image editing, “Make it snowy.” The generation model may refer to the edit prompt when generating an edited image (1340) based on the input image (1310) and the extended mask (1330). As a result, an edited image (1340) can be obtained that represents a style editing result in which the weather is changed to snowy weather, while the background and composition of the objects in the image (1310) are maintained and the objects and semantic regions associated with the objects are naturally maintained based on the extended mask (1330). The electronic device (100) may display the edited image (1340) on a screen.

[0170] In one embodiment, the process of generating an expanded mask (1330) may utilize a noise processing network of a generative model trained for object removal, and the process of editing the background may utilize a generative model trained for style change. The generative model (e.g., an object removal model, a style change model, etc.) may be a diffusion model that utilizes a diffusion process for image editing and generation tasks. The generative model may be implemented through a variation or extension of a neural network for processing the diffusion process. The generative model may include, for example, a U-net, but is not limited thereto.

[0171] In one embodiment, generative models processing different types of image editing tasks may use a common network structure while having training data and conditions set differently to perform specific task types. For example, an object removal model may be trained to remove objects with an image and an extended mask as conditions, and a style change model may be trained to change the image style with an image, an extended mask, and an editing prompt as conditions. Alternatively, for example, a background synthesis model may be trained to synthesize an image background with an image, an extended mask, and a reference background image as conditions.

[0172] In one embodiment, the generative model may be a multitask model trained to handle multiple tasks. In this case, the task of the multitask model may be selected depending on what the input data is. For example, if an image and an extended mask are input, an object removal task may be selected, and if text / image is additionally input to the image and the extended mask, a background change task may be selected. An edited image may be obtained by performing a task corresponding to the type of task selected by the multitask model (e.g., object removal, background change, etc.).

[0173] FIG. 14 is a drawing for illustrating an exemplary operation of an electronic device editing an image according to one embodiment of the present disclosure.

[0174] In one embodiment, the electronic device (100) may receive user input selecting an object within an image (1410). The electronic device (100) may identify the object corresponding to the user input and generate a mask (1420) corresponding to the object area.

[0175] The electronic device (100) can generate an expanded mask (1430) through semantic mask expansion based on an image (1410) and a mask (1420). The expanded mask (1430) may include a shadow area, which is a semantic area associated with an object. The electronic device (100) can display the expanded mask (1430) and finally determine the area of ​​the expanded mask (1430) based on user input that approves or changes the expanded mask (1430).

[0176] The electronic device (100) can determine the type of image editing operation to edit the image (1410) based on the area corresponding to the extended mask (1430). For example, the electronic device (100) can receive user input selecting the image editing operation type. The electronic device (100) can select a generative model trained for image editing corresponding to the operation type.

[0177] For example, a background can be synthesized using a generation model of an electronic device (100). Background synthesis may be a process of moving an object within an image and compositing it onto a new background. In one embodiment, the electronic device (100) may select a general-purpose generation model to perform the background synthesis. In one embodiment, the electronic device (100) may select a generation model trained for background synthesis to perform the background synthesis.

[0178] The electronic device (100) can input an image (1410) and an extended mask (1430) into the generation model. Additionally, the electronic device (100) can input an image to be referenced for background synthesis into the generation model. In this case, the generation model can use the reference image as an image condition, which is a condition applied for image generation.

[0179] For example, the reference image may be a beach image. The generating model may refer to the reference image when generating an edited image (1440) based on the input image (1410) and the extended mask (1430). As a result, an edited image (1440) may be obtained that represents a background composite result in which the background is changed to a beach while the object and the semantic region associated with the object are naturally maintained based on the extended mask (1430). The electronic device (100) may display the edited image (1440) on a screen.

[0180] In one embodiment, the process of generating an extended mask (1430) may utilize a noise processing network of a generative model trained for object removal, and the process of editing the background may utilize a generative model trained for background synthesis. The generative model (e.g., an object removal model, a background synthesis model, etc.) may be a diffusion model that utilizes a diffusion process for image editing and generation tasks. The generative model may be implemented through a variation or extension of a neural network for processing the diffusion process. The generative model may include, for example, a U-net, but is not limited thereto.

[0181] In one embodiment, generative models processing different types of image editing tasks may use a common network structure while having training data and conditions set differently to perform specific task types. For example, an object removal model may be trained to remove objects with an image and an extended mask as conditions, and a style change model may be trained to change the image style with an image, an extended mask, and an editing prompt as conditions. Alternatively, for example, a background synthesis model may be trained to synthesize an image background with an image, an extended mask, and a reference background image as conditions.

[0182] In one embodiment, the generative model may be a general-purpose model trained to handle multiple tasks (e.g., a multitask model). In this case, the task of the multitask model may be selected depending on what the input data is. For example, if an image and an extended mask are input, an object removal task may be selected, and if text / image is additionally input to the image and the extended mask, a background change task may be selected. An edited image may be obtained by performing a task corresponding to the type of task selected by the multitask model (e.g., object removal, background change, etc.).

[0183] FIG. 15a is a drawing for illustrating an exemplary operation of an electronic device editing an image according to one embodiment of the present disclosure.

[0184] In one embodiment, the electronic device (100) may receive user input selecting an object within an image (1510). The electronic device (100) may identify the object corresponding to the user input and generate a mask (1520) corresponding to the object area. In the example of FIG. 15a, the object 'hand' is selected by user input and a mask corresponding to the hand is generated, which is described as an example.

[0185] The electronic device (100) can generate an expanded mask (1535) through semantic mask expansion based on an image (1510) and a mask (1520). Semantic mask expansion may include a process of calculating the difference between a conditional prediction and an unconditional prediction that predicts noise based on the input image (1510) and the mask (1520). By referring to the semantic mask expansion result (1530), semantic regions associated with the object (hand), such as shadow regions and light reflection regions, may be detected.

[0186] The electronic device (100) can generate an expanded mask (1535) through semantic mask expansion based on an image (1510) and a mask (1520). The expanded mask (1535) may include a shadow area and a light reflection area, which are semantic regions associated with an object. The electronic device (100) can display the expanded mask (1535) and finally determine the area of ​​the expanded mask (1535) based on user input that approves or changes the expanded mask (1535).

[0187] The electronic device (100) can perform image editing operations using an extended mask (1535). The electronic device (100) can determine the type of image editing operation to be performed to edit the image (1510) based on the area corresponding to the extended mask (1535). For example, the electronic device (100) can receive user input selecting the image editing operation type. The electronic device (100) can select a generative model trained for image editing corresponding to the operation type.

[0188] For example, an object can be removed using a generation model of the electronic device (100). In one embodiment, the electronic device (100) may select a general-purpose generation model to perform the object removal operation. In one embodiment, the electronic device (100) may select a generation model trained for object removal to perform the object removal operation.

[0189] The electronic device (100) can input an image (1510) and an extended mask (1535) into a generative model. Using the generative model, the electronic device (100) can obtain an edited image (1540) representing a natural object removal result in which the object and the semantic regions associated with the object (e.g., light reflections, shadows, etc.) are removed together.

[0190] FIG. 15b is a drawing for illustrating an exemplary operation in which an electronic device according to one embodiment of the present disclosure provides an edited image.

[0191] In one embodiment, the electronic device (100) can display an edited image (1540). The electronic device (100) can be implemented as various types of devices including a display. For example, the electronic device (100) can be implemented as a smart appliance such as a refrigerator (1500).

[0192] The electronic device (100) can acquire an image of the food ingredient (e.g., the image (1510) of FIG. 15a) when the user places the food ingredient inside the refrigerator (1500). The electronic device (100) can generate a mask (e.g., the mask (1520) of FIG. 15a) corresponding to an object to be removed (e.g., the user's hand) based on user input, and can generate an expanded mask (e.g., the expanded mask (1535) of FIG. 15a) by expanding the mask to remove the object. The electronic device (100) can display an edited image (1540) based on the expanded mask. The edited image (1540) may show a result in which the user's hand is naturally removed. The electronic device (100) can generate an edited image (1540) such that only the whole food ingredient is included in the image and display it in the list of stored items of the refrigerator (1500).

[0193] In one embodiment, the electronic device (100) can transmit the edited image (1540) to another electronic device (e.g., a smart appliance). For example, the electronic device (100) may be an electronic device separate from the refrigerator (1500) (e.g., a smartphone, etc.). The electronic device (100) can transmit the edited image (1540) to another electronic device so that the edited image (1540) can be displayed on the other electronic device. Specifically, the electronic device (100) can transmit the edited image (1540) to the refrigerator (1500).

[0194] In one embodiment, the electronic device (100) can transmit an edited image (1540) to another electronic device so that the edited image (1540) can be displayed on the other electronic device, and the electronic device (100) can check information related to the other electronic device. For example, the electronic device (100) can receive a list of stored items and images of stored items from a refrigerator (1500), and display the edited image (1540) so that it is included in the list of stored items, thereby allowing a user to check the food ingredients stored in the refrigerator (1500) from a distance.

[0195] Meanwhile, FIG. 15b describes an example in which the electronic device (100) is implemented as a refrigerator (1500) or the electronic device (100) is implemented as a smartphone and interacts with the refrigerator (1500), but the types of the electronic device (100) are not limited thereto. The electronic device (100) may include, for example, smart home appliances (e.g., smart TV, smart refrigerator, etc.), smartphones, tablet PCs, laptop PCs, glasses-type displays, head-mounted displays, etc., which are devices capable of displaying images through a display. In addition to the examples described above, the electronic device (100) may be applied to all types of electronic devices and all kinds of embodiments capable of generating an extended mask representing an object and a semantic region associated with the object, and providing an edited image using the extended mask.

[0196] FIG. 16 is a block diagram illustrating the configuration of a server according to one embodiment of the present disclosure.

[0197] In one embodiment, the electronic device (100) may operate using a cloud-based AI method that generates an on-device extended mask and receives an edited image from a generative model operated on a server (2000) without performing image editing operations using a generative model. Accordingly, all or at least some of the operations of the aforementioned electronic device (100) may be performed by the server (2000).

[0198] For example, an electronic device (100) may generate an extended mask based on an image and a mask, and transmit the image and the extended mask to a server (2000). The server (2000) may generate an edited image using a generation model based on the image and the extended mask, and transmit it to the electronic device (100). Alternatively, the electronic device (100) may transmit the image to the server (2000), and after the mask generation and image editing operations are performed at the server (2000), the edited image may be transmitted to the electronic device (100).

[0199] The server (2000) may be a computing device composed of hardware elements having higher performance specifications than the electronic device (100), capable of processing complex computations and tasks using large-scale data, such as training, inference, management, distribution, and operation of the generative model.

[0200] The processor (2100) may include a processing circuit. The processing circuit may include an operation unit that performs arithmetic and logical operations, a control unit that interprets commands and controls execution, a register that stores data, and a cache memory, but is not limited thereto.

[0201] The processor (2100) may be composed of at least one of, for example, a Central Processing Unit (CPU), a Microprocessor, a Graphic Processing Unit (GPU), ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), an Application Processor (AP), a Neural Processing Unit (NPU), or an AI-dedicated processor designed with a hardware structure specialized for processing AI models, but is not limited thereto.

[0202] The memory (2200) may include a main memory (2210) that stores data currently being processed by the server (2000). The main memory (2210) may store programs and data currently being executed by the processor (2100) to allow the processor (2100) to access the data quickly. The main memory (2210) may include volatile memory such as, for example, RAM (Random Access Memory) or SRAM (Static Random Access Memory), but is not limited thereto.

[0203] The memory (2200) may include storage (2220) that permanently stores large amounts of data (e.g., programs, applications, system files, media files, etc.). The storage (2220) may include non-volatile memory including at least one of, for example, a hard disk drive (HDD), a solid-state drive (SSD), an optical drive (e.g., a CD), a flash drive, ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), and PROM (Programmable Read-Only Memory), but is not limited thereto.

[0204] The communication interface (2230) can perform data communication between a server (2000) and another electronic device (e.g., a server, an external electronic device, etc.) by using at least one of data communication methods including, for example, a wired LAN (e.g., Ethernet), a wireless LAN (e.g., Wi-Fi), a cellular network (e.g., 4G, 5G, etc.), Bluetooth, BLE (Bluetooth Low Energy), ZigBee, infrared communication (IrDA, infrared Data Association), NFC (Near Field Communication), RF communication, and various other types of known wireless / wired communication technologies. The communication interface (2230) may include a communication circuit designed to use the aforementioned communication methods.

[0205] The functions of the processor (2100), memory (2200), and communication interface (2230) of the server (2000) can correspond to the functions described for each component of the electronic device (100) of FIG. 2. Therefore, for brevity, repetitive descriptions are omitted.

[0206] The present disclosure relates to a method, electronic device, and server for expanding a mask to include a semantic region associated with an object. It also relates to a method, electronic device, and server for editing an image using a generative model based on the expanded mask. The technical problems to be solved by the present disclosure are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description in this specification.

[0207] According to one aspect of the present disclosure, a method for an electronic device to edit an image based on generative AI may be provided.

[0208] The above method may include a step of identifying an object within an image based on user input.

[0209] The above method may include the step of generating a mask representing the object region.

[0210] The above method may include the step of performing noise prediction using a noise processing network based on the image and the mask.

[0211] The above method may include the step of detecting a semantic region associated with the object based on the noise prediction result.

[0212] The above method may include the step of generating an extended mask including the semantic region.

[0213] The above method may include the step of displaying an image edited through a generation model based on the image and the extended mask.

[0214] The step of performing the above noise prediction may include the step of adding noise to the image.

[0215] The step of performing the noise prediction above may include the step of performing conditional prediction and unconditional prediction on the image with the added noise, using the mask as a condition.

[0216] The step of detecting the semantic region may include the step of detecting the semantic region based on the difference between the conditional prediction and the unconditional prediction.

[0217] The step of performing the noise prediction above may involve performing the conditional prediction and the unconditional prediction using a noise processing network of a generative model trained for object removal.

[0218] The above semantic region may represent the result of a generative model trained for object removal receiving the mask as a condition and predicting noise for the object region and surrounding regions associated with the object to remove the object corresponding to the mask.

[0219] The step of detecting the semantic region may include applying one or more post-processing steps, including a dilation operation, to the difference between the conditional prediction and the unconditional prediction.

[0220] The step of detecting the semantic region above may include a step of thresholding the result of applying one or more post-processing steps.

[0221] The step of performing the above conditional prediction and the above unconditional prediction may be to perform the above conditional prediction and the above unconditional prediction as a single step of noise prediction.

[0222] The step of performing the above conditional prediction and the above unconditional prediction may be to perform the above conditional prediction and the above unconditional prediction fewer times than the noise prediction of the total time steps.

[0223] The above method may include the step of displaying the image and the extended mask together.

[0224] The above method may include the step of changing the area of ​​the expanded mask based on user input regarding the expanded mask.

[0225] The above method may include a step of determining the type of operation to edit the image based on the area corresponding to the expanded mask.

[0226] The step of displaying an image edited through the above-mentioned generation model may involve selecting a generation model corresponding to the above-mentioned work type and displaying the image edited through the selected generation model.

[0227] The type of operation to edit the above image may be either an object removal operation or a background change operation.

[0228] According to one aspect of the present disclosure, an electronic device for editing images based on generative AI may be provided.

[0229] The electronic device may include at least one processor, a memory for storing instructions, and a display.

[0230] By executing the above instructions by the at least one processor, the electronic device can identify an object within an image based on user input.

[0231] By executing the above instructions by the at least one processor, the electronic device can generate a mask representing the object region.

[0232] By executing the above instructions by the at least one processor, the electronic device can perform noise prediction using a noise processing network based on the image and the mask.

[0233] By executing the above instructions by the at least one processor, the electronic device can detect a semantic region associated with the object based on the noise prediction result.

[0234] By executing the above instructions by the at least one processor, the electronic device can generate an extended mask including the semantic region.

[0235] By executing the above instructions by the at least one processor, the electronic device can control the display to display an image edited through a generation model based on the image and the extended mask.

[0236] By executing the above instructions by the at least one processor, the electronic device can add noise to the image.

[0237] By executing the above instructions by the at least one processor, the electronic device can perform conditional prediction and unconditional prediction on the noise-added image with the mask as a condition.

[0238] By executing the above instructions by the at least one processor, the electronic device can detect the semantic region based on the difference between the conditional prediction and the unconditional prediction.

[0239] By executing the above instructions by the at least one processor, the electronic device can perform the conditional prediction and the unconditional prediction using a noise processing network of a generative model trained for object removal.

[0240] The above semantic region may represent the result of a generative model trained for object removal receiving the mask as a condition and predicting noise for the object region and surrounding regions associated with the object to remove the object corresponding to the mask.

[0241] By executing the above instructions by the at least one processor, the electronic device can apply one or more post-processing operations, including an expansion operation, to the difference between the conditional prediction and the unconditional prediction.

[0242] By executing the above instructions by the at least one processor, the electronic device can critically process the result of applying the one or more post-processing steps.

[0243] By executing the above instructions by the at least one processor, the electronic device can perform the conditional prediction and the unconditional prediction as a single-stage noise prediction.

[0244] By executing the above instructions by the at least one processor, the electronic device can perform the conditional prediction and the unconditional prediction fewer times than the noise prediction of the entire time steps.

[0245] By executing the above instructions by the at least one processor, the electronic device can control the display to display the image and the extended mask together.

[0246] By executing the above instructions by the at least one processor, the electronic device can change the area of ​​the extended mask based on user input regarding the extended mask.

[0247] By executing the above instructions by the at least one processor, the electronic device can determine the type of operation to edit the image based on the area corresponding to the expanded mask.

[0248] By executing the above instructions by the at least one processor, the electronic device can control the display to select a generation model corresponding to the type of operation and display an image edited through the selected generation model.

[0249] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include computer storage media and communication media. Computer storage media include both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include other data of modulated data signals, such as computer-readable instructions, data structures, or program modules.

[0250] Additionally, computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory storage media' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, 'non-transitory storage media' may include a buffer in which data is stored temporarily.

[0251] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0252] The foregoing description of the present disclosure is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0253] The scope of the present disclosure is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present disclosure.

Claims

1. In a method for an electronic device to edit images based on generative AI, A step (310) of identifying objects within an image based on user input; Step (320) of generating a mask representing the object area above; A step (330) of performing noise prediction using a noise processing network based on the above image and the above mask; A step (340) of detecting a semantic region associated with the object based on the noise prediction result above; Step (350) of generating an extended mask including the above semantic region; and A method comprising the step (360) of displaying an image edited through a generation model based on the above image and the above extended mask.

2. In Paragraph 1, The step of performing the above noise prediction is, Step of adding noise to the above image; and The method includes the step of performing conditional prediction and unconditional prediction on the above-mentioned noise-added image using the above-mentioned mask as a condition, and The step of detecting the above semantic region is, A method comprising the step of detecting the semantic region based on the difference between the conditional prediction and the unconditional prediction.

3. In Paragraph 2, The step of performing the noise prediction above involves performing the conditional prediction and the unconditional prediction using a noise processing network of a generative model trained for object removal. A method in which the semantic region above represents the result of a generative model trained for object removal receiving the mask as a condition input and predicting noise for the object region and the surrounding region associated with the object to remove the object corresponding to the mask.

4. In Paragraph 2, The step of detecting the above semantic region is, A step of applying one or more post-processing steps, including a dilation operation, to the difference between the above conditional prediction and the above unconditional prediction; and A method comprising the step of thresholding the result of applying one or more of the above-mentioned post-processing steps.

5. In any one of paragraphs 2 through 4, The step of performing the above conditional prediction and the above unconditional prediction is, A method of performing the above conditional prediction and the above unconditional prediction as a single-step noise prediction.

6. In Paragraph 1, The above method further comprises the step of displaying the image and the extended mask together.

7. In Paragraph 6, The above method further comprises the step of changing the area of ​​the expanded mask based on user input regarding the expanded mask.

8. In an electronic device (100) that edits images based on generative AI, At least one processor (110); Memory (120) for storing instructions; and Includes a display (130), By executing the above instructions by the at least one processor (110), the electronic device (100) Identify objects within an image based on user input, and Create a mask representing the object area above, and Noise prediction is performed using a noise processing network based on the above image and the above mask, and Based on the noise prediction results above, detect the semantic region associated with the object, and Generate an extended mask including the above semantic region, and An electronic device (100) that controls the display (130) to display an image edited through a generation model based on the above image and the above extended mask.

9. In Paragraph 8, By executing the above instructions by the at least one processor, the electronic device, Add noise to the above image, and Conditional prediction and unconditional prediction are performed on the above-mentioned noise-added image using the above-mentioned mask as a condition, and An electronic device that detects the semantic region based on the difference between the above conditional prediction and the above unconditional prediction.

10. In Paragraph 9, By executing the above instructions by the at least one processor, the electronic device performs the conditional prediction and the unconditional prediction using a noise processing network of a generative model trained for object removal, and An electronic device in which the semantic region above represents the result of a generative model trained for object removal receiving the mask as a condition input and predicting noise for the object region and the surrounding region associated with the object to remove the object corresponding to the mask.

11. In Paragraph 9, By executing the above instructions by the at least one processor, the electronic device, One or more post-processing operations including dilation are applied to the difference between the above conditional prediction and the above unconditional prediction, and An electronic device that critically processes the result of applying one or more of the above post-processing methods.

12. In any one of paragraphs 9 through 11, By executing the above instructions by the at least one processor, the electronic device, An electronic device that performs the above conditional prediction and the above unconditional prediction as a single-step noise prediction.

13. In Paragraph 8, An electronic device that controls a display to display the image and the extended mask together, by executing the above instructions by the at least one processor.

14. In Paragraph 8, An electronic device that changes the area of ​​the extended mask based on user input for the extended mask, by executing the above instructions by the above at least one processor.

15. A computer-readable recording medium having a program for executing the method of any one of paragraphs 1 through 7 on a computer.

Images