Electronic device, method, and non-transitory computer-readable storage medium for determining objects in image to be removed according to user input

Abstract

An electronic device according to an embodiment may identify a plurality of objects in a source image. The electronic device may display, on the basis of identifying the plurality of objects, a UI object for removing at least one object from among the plurality of objects. On the basis of receiving a first user input related to the UI object, the electronic device may visually emphasize and display an object of a first group among the plurality of objects relative to an object of a second group. The distance between a reference point of the source image and an object in the second group may be smaller than the distance between the reference point of the source image and an object in the first group. On the basis of receiving a second user input for generating an edited image, the electronic device may acquire the edited image in which the object of the first group is removed according to inpainting, and the object of the second group is maintained.

Description

Electronic device, method, and non-transient computer-readable storage medium for determining an object within an image to be removed according to user input

[0001] The present disclosure relates to an electronic device, a method, and a non-transient computer-readable storage medium for determining an object within an image to be removed according to user input.

[0002] Technologies for processing photos and / or videos using artificial intelligence are being developed. For example, technologies for recognizing one or more characters (or strings) associated with photos and / or videos are being developed. For example, technologies for classifying subjects captured by photos and / or videos (e.g., objects including people, animals, and / or vehicles) are being developed.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. None of the foregoing shall be claimed as prior art related to the present disclosure, nor shall it be used to determine prior art.

[0004] According to one embodiment, an electronic device may include a display, a memory comprising one or more storage media for storing instructions, and at least one processor comprising a processing circuit. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause a source image to be displayed through the display. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause a plurality of objects within the source image to be identified. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause a user interface (UI) object to be displayed for removing at least one of the plurality of objects based on identifying the plurality of objects within the source image. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause, based on receiving a first user input related to the UI object, to visually highlight a first group of objects among the plurality of objects with respect to a second group of objects. The distance between the reference point of the source image and the objects in the second group may be smaller than the distance between the reference point of the source image and the objects in the first group. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause, based on receiving a second user input to create an edited image while the objects in the first group are visually highlighted, to obtain an edited image in which the objects in the first group are removed according to inpainting and the objects in the second group are retained.

[0005] In one embodiment, a method of an electronic device including a display may be provided. The method may include an operation of displaying a source image through the display. The method may include an operation of identifying a plurality of objects within the source image. Based on identifying the plurality of objects within the source image, the method may include an operation of displaying a UI object for removing at least one of the plurality of objects. Based on receiving a first user input related to the UI object, the method may include an operation of visually highlighting a first group of objects among the plurality of objects with respect to a second group of objects. The distance between a reference point of the source image and an object in the second group may be smaller than the distance between the reference point of the source image and an object in the first group. Based on receiving a second user input for creating an edited image while the object in the first group is visually highlighted, the method may include an operation of obtaining an edited image in which the object in the first group is removed according to inpainting and the object in the second group is retained.

[0006] In one embodiment, a non-transient computer-readable storage medium comprising instructions may be provided. When executed by an electronic device including a display, the instructions may cause the electronic device to display a source image through the display. When executed by the electronic device, the instructions may cause the electronic device to receive a first user input for removing at least one object from the source image. When executed by the electronic device, the instructions may cause the electronic device to display a visual object associated with a first object among a plurality of objects included in the source image in a superposition on the source image based on the first user input. When executed by the electronic device, the instructions may cause the electronic device to obtain an edited image in which the first object among the plurality of objects is removed and the second object among the plurality of objects is retained, based on receiving a second user input for removing the first object associated with the visual object. When the above instructions are executed by the electronic device, the electronic device may cause the edited image to be displayed through the display.

[0007] According to one embodiment, an electronic device comprises a display, a memory including one or more storage media for storing instructions, and at least one processor including a processing circuit, wherein the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to display a source image through the display. The instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to receive a first user input for removing at least one object from the source image. The instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to superimpose a visual object associated with a first object among a plurality of objects included in the source image onto the source image based on the first user input. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the first object among the plurality of objects to be removed and the second object among the plurality of objects to be retained, based on receiving a second user input for removing the first object associated with the visual object. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the edited image to be displayed through the display.

[0008] FIG. 1 illustrates exemplary states of an electronic device according to one embodiment.

[0009] FIG. 2 illustrates a schematic block diagram of an electronic device according to one embodiment.

[0010] FIG. 3 illustrates a flowchart for explaining operations performed by an electronic device according to one embodiment.

[0011] FIG. 4 illustrates an exemplary operation of an electronic device that acquires first map information for a source image using a trained model.

[0012] FIG. 5 illustrates an exemplary operation of an electronic device that acquires second map information for a source image using a trained model.

[0013] FIG. 6 illustrates an exemplary operation of an electronic device that classifies objects included in a source image using first map information for the source image.

[0014] FIG. 7 illustrates an exemplary operation of an electronic device for determining information for classifying objects included in a source image.

[0015] FIG. 8 illustrates an exemplary operation of an electronic device for determining at least one object to be removed from a source image using map information representing the result of recognizing objects included in a source image.

[0016] FIG. 9 illustrates exemplary states of an electronic device receiving user input for editing a source image.

[0017] FIG. 10 illustrates an exemplary operation of an electronic device that receives user input for editing a source image.

[0018] FIG. 11 illustrates a flowchart for explaining operations performed by an electronic device according to one embodiment.

[0019] FIG. 12 is a block diagram of an electronic device in a network environment according to various embodiments.

[0020] Hereinafter, various embodiments of this document will be described with reference to the attached drawings.

[0021] The various embodiments of this document and the terms used therein are not intended to limit the technology described in this document to specific embodiments and should be understood to include various modifications, equivalents, and / or substitutions of such embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar components. A singular expression may include a plural expression unless the context clearly indicates otherwise. In this document, expressions such as “A or B,” “at least one of A and / or B,” “A, B or C,” or “at least one of A, B and / or C” may include all possible combinations of items listed together. Expressions such as “first,” “second,” “first,” or “second” may modify the components, regardless of order or importance, and are used only to distinguish one component from another and do not limit the components. When it is mentioned that a certain (e.g., 1st) component is “(functionally or telecommunicationally) connected” or “connected” to another (e.g., 2nd) component, said certain component may be directly connected to said other component or connected through another component (e.g., 3rd component).

[0022] FIG. 1 illustrates exemplary states (191, 192, 193, 194) of an electronic device (101) according to one embodiment. The electronic device (101) of the present disclosure may be referred to as a mobile device, user equipment (UE) (or user terminal), multifunction device, portable communication device, and / or portable device. Exemplary hardware component(s) included in the electronic device (101) are described with reference to FIG. 2 and / or FIG. 12.

[0023] According to one embodiment, an electronic device (101) may be configured to create, edit, and / or remove an image. In the present disclosure, an image may refer to a file converted into a JPEG (joint photographic experts group) (or JPG), PNG (portable network graphics), HEIF (high efficiency image format), bitmap, or other suitable format, and / or information stored in said file. In the present disclosure, the term “image” may be used interchangeably with terms such as image file, media content, content, or photograph.

[0024] Referring to FIG. 1, an exemplary state (191) of an electronic device (101) displaying a source image (130) is illustrated. For example, the electronic device (101) may display the source image (130) through a display (110). In the present disclosure, the source image (130) may be referred to as the original image. An exemplary state (191) of the electronic device (101) may be provided based on the execution of a software application for viewing images stored in the electronic device (101), referred to as a gallery application. According to one embodiment, the electronic device (101) may enhance the user experience and / or convenience related to editing the source image (130) based on the image editing function of the gallery application.

[0025] Referring to FIG. 1, within a state (191), a screen (e.g., execution screen) displayed through the display (110) of the electronic device (101) may include parts (121, 122, 123, 124, 125, 126). For example, part (121) may be referred to as the state bar and may display state information of the electronic device (101), including the current time and / or state of charge (SOC). For example, part (122) may include icons for switching the screen being displayed within the state (191) or changing the layout of the screen. For example, through part (123), the electronic device (101) may display a source image (130). Based on a touch gesture (e.g., pinch-to-zoom gesture, and / or swipe gesture) performed on part (123), the electronic device (101) may enlarge or reduce the source image (130) displayed through part (123). For example, within part (124), a plurality of images stored in the electronic device (101) containing the source image (130) may be listed. The plurality of images listed in part (124) may be arranged to be scrolled along one direction (e.g., the horizontal direction of the display (110)). For example, part (125) may include icons for executing functions related to the source image (130) displayed through part (123). For example, the part (126) may be referred to as a navigation bar and may include icons (e.g., a back button, a home button, and / or a recent apps button) for changing the screen displayed through the display (110).

[0026] Referring to the exemplary state (191) of FIG. 1, the electronic device (101) may display an icon (125-1) for editing a source image (130) through a part (125). The icon (125-1) may correspond to a function referred to as photo assist. The icon (125-1) may be displayed to execute a function for editing the source image (130) using information obtained from the source image (130) based on deep learning. The icon (125-1) may be displayed to execute a function for changing the source image (130) at least partially (using an artificial intelligence model), referred to as inpainting.

[0027] According to one embodiment, an electronic device (101) may be configured to determine an object within an image (e.g., source image (130)) to be removed based on user input. Referring to FIG. 1, in state (191), the electronic device (101) may switch to state (192) based on identifying user input indicating the selection of an icon (125-1) (e.g., a tap gesture on a part of the display (110) where the icon (125-1) is displayed). In state (192), the electronic device (101) may include the source image (130) and display a screen for editing the source image (130). For example, the screen displayed in state (192) may include a part (126) referred to as a navigation bar. For example, a screen displayed within a state (192) may include a portion (141) containing icons for undoing or redoing the editing of the source image (130). For example, a screen displayed within a state (192) may include a portion (142) for receiving a setting value (e.g., at least one parameter) related to the editing of the source image (130).

[0028] In the state (192) of FIG. 1, the electronic device (101) can identify at least one object in the source image (130). For example, based on an input indicating the selection of an icon (125-1), the electronic device (101) can identify at least one object in the source image (130). At least one program executed by the electronic device (101) to identify at least one object is described with reference to FIG. 2. Referring to FIG. 1, the source image (130) may include a plurality of objects, including an object (131) corresponding to a person who is the main subject. Based on identifying the plurality of objects in the source image (130), the electronic device (101) can display a UI object (141-1) for removing at least one of the plurality of objects. An exemplary state (192) in which the electronic device (101) displays a UI object (141-1) on part (141) is illustrated, but the embodiments are not limited thereto. An exemplary UI object (141-1) including an eraser-shaped icon and / or image is illustrated, but the embodiments are not limited thereto. In one embodiment, the UI object (141-1) may be displayed based on identifying an object to be removed from the source image (130).

[0029] In the state (192) of FIG. 1, the electronic device (101) may transition to the state (193) based on identifying user input related to the UI object (141-1) (e.g., a tap gesture on a portion of the display (110) on which the UI object (141-1) is displayed). For example, in the state (193), the electronic device (101) may display at least one object among a plurality of objects identified from the source image (130) with respect to other objects. Referring to the exemplary state (193) of FIG. 1, the electronic device (101) may display object (132) among a plurality of objects with respect to object (131) with respect to object (131). For example, in the state (193), the electronic device (101) may recommend the remaining objects (e.g., object (132)) that are different from object (131) as objects to be removed from the source image (130). For example, a first group of objects including the remaining objects mentioned above can be visually highlighted with respect to a second group of remaining objects.

[0030] Object(s) (e.g., object (132)) in a source image (130) that are visually highlighted by the electronic device (101) within state (193) may be other object(s) that are different from the main subject (e.g., object (131)) identified from the source image (130). Referring to the exemplary source image (130) of FIG. 1, since object (131) represents a person who is the main subject, other objects (e.g., object (132)) representing other people may be recommended as objects to be removed from the source image (130). An exemplary operation of the electronic device (101) for recommending at least one object to be removed from the source image (130) is described with reference to FIGS. 3 through 8 and / or FIG. 11.

[0031] In the exemplary state (193) of FIG. 1, the electronic device (101) may display a UI object (142-1) for creating an edited image (160) from a source image (130). A UI object (142-1) having the form of a button containing text such as “Create Image” is shown, but embodiments are not limited thereto. In the exemplary state (193) of FIG. 1, while visually highlighting at least one object to be removed from the source image (130), the electronic device (101) may display a popup menu (150) for editing the source image (130). For example, the electronic device (101) may display the popup menu (150) superimposed on the source image (130) based on receiving user input related to the UI object (141-1). The popup menu (150) may be displayed at a location adjacent to the at least one object that is visually highlighted (e.g., a location adjacent to the top of said at least one object). A popup menu (150) may be displayed superimposed on a source image (130). The popup menu (150) may include a UI object (152) for removing at least one object from the source image (130). The popup menu (150) may include a UI object (151) for moving at least one object within the source image (130). The UI objects included in the popup menu (150) (e.g., UI objects (151, 152)) may be referred to as options for editing the source image (130). An exemplary operation of an electronic device (101) associated with the popup menu (150) is described with reference to FIG. 9 and / or FIG. 10.

[0032] In the exemplary state (193) of FIG. 1, the electronic device (101) that identifies user input related to a UI object (152) (e.g., a tap gesture on a portion of the display (110) on which the UI object (152) is displayed) may transition to state (194). Similarly, based on user input related to a UI object (142-1), the electronic device (101) may transition to state (194). For example, based on receiving user input (e.g., user input related to the UI object (152)) for creating an edited image while at least one object in the source image (130) is visually highlighted, the electronic device (101) may obtain or create an edited image (160) in which the at least one object (e.g., object (132)) is removed (e.g., removal based on inpainting) and another object (e.g., object (131)) is retained. In the present disclosure, the edited image (160) may be referred to as a synthetic image and / or an AI (artificial intelligence) generated image.

[0033] In the exemplary state (194) of FIG. 1, the electronic device (101) may display an edited image (160) through the display (110). The edited image (160) may be displayed as a replacement for the source image (130) that was displayed in a state prior to state (194) (e.g., states (192, 193)). For example, in state (194), the electronic device (101) may display a UI object (142-2) (e.g., a UI object in the form of a button containing text such as “view original”) to display the source image (130) that was displayed prior to the edited image (160). For example, within state (194), the electronic device (101) may display a UI object (142-3) for saving an edited image (160) that is being displayed through the display (110) (e.g., a UI object in the form of a button containing text such as “save copy”). The embodiment is not limited thereto, and the electronic device (101) may (further) display a UI object for sharing the edited image (160).

[0034] In the exemplary state (194) of FIG. 1, based on user input related to the UI object (142-2), the electronic device (101) may display the source image (130) through the display (110). For example, the electronic device (101) may stop displaying the edited image (160) through the display (110) and may display the source image (130) again. Based on user input related to the UI object (142-2), the electronic device (101) may discard the edited image (160). For example, the electronic device (101) may remove the edited image (160) from the memory of the electronic device (101).

[0035] In the exemplary state (194) of FIG. 1, based on user input associated with a UI object (142-3), the electronic device (101) may store an edited image (160) displayed through the display (110). For example, the edited image (160) may be stored in the memory of the electronic device (101) and / or stored on a server (e.g., a cloud server) connected to the electronic device (101) (e.g., upload). For example, the edited image (160) may be overwritten on the source image (130). For example, the edited image (160) may be stored as a file different from the source image (130). For example, the edited image (160) may be stored with information for restoring the source image (130) from the edited image (160). In the above example, information for restoring the source image (130) may be stored in the metadata of the edited image (160) (e.g., metadata based on EXIF ​​(exchangeable image file format)).

[0036] As described above, according to one embodiment, an electronic device (101) can perform a function to remove at least one object (e.g., objects (131, 132)) from a source image (130). For example, the electronic device (101) can obtain or generate information for segmenting at least one object from the source image (130) using an artificial intelligence model. Using said information, the electronic device (101) can identify a main subject within the source image (130) (e.g., a subject located at the center of the source image (130), such as object (131), and / or a subject occupying a relatively large size in the source image (130)). To support a function to remove the remaining subjects that are different from the main subject, the electronic device (101) can recommend and / or highlight said remaining subjects. Based on user input (e.g., user input for a UI object (152)), the electronic device (101) can generate or composite an edited image (160) in which the remaining subjects (e.g., object (132)) are removed and the main subject (e.g., object (131)) is retained.

[0037] Below, with reference to FIG. 2, an exemplary hardware structure included in the electronic device (101) of FIG. 1 is described.

[0038] FIG. 2 illustrates a schematic block diagram of an electronic device (101) according to one embodiment. The electronic device (101) of FIG. 2 may include the electronic device (101) of FIG. 1. The electronic device (101) may have a form factor of a bar-type smartphone (291), as well as a laptop PC (personal computer) (294), a foldable-type smartphone (292), a sliderable (or rollable)-type smartphone, a head-mounted display (HMD) device (293) (or head-wearable electronic device), a watch, an electronic control unit (ECU) in a vehicle (e.g., an electric vehicle, EV), a tablet PC, and / or other suitable computing device (e.g., a desktop PC).

[0039] Referring to FIG. 2, the electronic device (101) may include a processor (210), a memory (220), and / or a display (110). The processor (210) may be electrically and / or operatively coupled to the memory (220) and / or the display (110). Electrical coupling of the electronic components may include a state in which a wired signal path (or a connection for wireless communication) for transmitting a signal is established between the electronic components. Operatory coupling of the electronic components may include a state in which the electronic components are directly coupled (or indirectly coupled) so that one of the electronic components controls another electronic component. Referring to FIG. 2, an electrical connection between the processor (210), the memory (220), and the display (110), based on the electronic components and referred to as a communication bus (202), is schematically illustrated. Through the communication bus (202), the processor (210), memory (220), and display (110) can be communicated together.

[0040] Referring to FIG. 2, the processor (210) of the electronic device (101) may include circuits (e.g., processing circuits and / or cores) for performing operations on data (e.g., arithmetic operations and / or logical operations). A binary code (e.g., instructions) representing said operations may be input to the processor (210). The processor (210) may include a central processing unit (CPU), a graphic processing unit (GPU), and / or a neural processing unit (NPU). The processor (210) may be referred to as an application processor (AP) and / or a system on a chip (SoC). The processor (210) may have a structure for loading (or fetching) and / or executing multiple instructions simultaneously (e.g., a multi-core structure based on a combination of multiple core circuits such as a dual core, quad core, hexa core, or octa core). In an electronic device (101) comprising at least one processor including a processor (210), the at least one processor may perform the operations of the present disclosure individually or collectively. For example, the at least one processor may perform the operations of FIG. 3 and / or FIG. 11 individually and / or collectively by executing instructions stored in memory (215).

[0041] The memory (220) of FIG. 2 may include a circuit for storing data (or instructions) that are input to or output from the processor (210). The memory (220) may include volatile memory, such as random-access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM). Non-volatile memory may be referred to as storage. Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, solid state drive (SSD), and embedded multimedia card (eMMC). The memory (220) may include one or more storage media (e.g., the volatile memory and / or non-volatile memory described above) distributedly located in the electronic device (101). The processor (210) of the electronic device (101) may execute instructions in the memory (220) within the electronic device (101) to perform functions and / or operations (e.g., the operations of FIG. 3 and / or FIG. 11) indicated by said instructions.

[0042] A display (110) of an electronic device (101) may include a circuit for visualizing information provided from a processor (210). The display (110) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). Embodiments are not limited thereto, and the display (110) may include electronic paper. A display area (or active area) of the display (110) may include a light-emitting area formed by pixels (e.g., active pixels) of the display (110). The display (110) may include a sensor (e.g., a touch sensor) for detecting an external object (e.g., a user's finger) on the display (110). The sensor may be included in the display (110) in the form of a panel (e.g., a touch sensor panel).

[0043] Referring to FIG. 2, the processor (210) can execute an image editing application (231), which is a program stored in memory (220). The exemplary states (191, 192, 193, 194) of FIG. 1 may be provided based on the execution of the image editing application (231). For example, a set of instructions (e.g., a package) referred to as the image editing application (231) may be stored in memory (220). The processor (210) that executes the image editing application (231) may perform the operation of the present disclosure, including FIG. 1. Referring to FIG. 2, the blocks included in the image editing application (231) (e.g., individual object segmentation model (232), person object segmentation model (233), clipper map generator (234), object recommender (235), and inpainting model (236)) can each correspond to the functions of the image editing application (231).

[0044] In the present disclosure, the term “model” may be used interchangeably with terms such as trained model, artificial intelligence model, neural network, and / or deep learning model. A trained model may include a computational model defined to simulate the neural activity of an organism, instructions representing said computational model, and / or a set of parameters used to define said computational model. For example, a trained model may include nodes referred to as artificial neurons and / or perceptrons, and weights assigned to connections between said nodes. For example, nodes of a trained model based on a feedforward neural network (FNN) may be included in an input layer, one or more hidden layers, and an output layer, respectively. The weights may define a weighted sum between nodes included in each of the layers, such as the input layer, hidden layer, and / or output layer. A set of said weights may be stored in memory (220). The nodes of the trained model can be arranged according to structures such as CNN (convolutional neural network), RNN (recurrent neural network), LSTM (long-short term memory), and / or Transformer.

[0045] According to one embodiment, a processor (210) that executes an object recommender (235) and / or an image editing application (231) can identify or segment object(s) that can be removed from an image (e.g., source image (130) of FIG. 1). The object recommender (235) can be executed in response to user input indicating a selection of a UI object (141-1) of FIG. 1. For example, the processor (210) can input the image into an individual object segmentation model (232) and / or a person object segmentation model (233). For example, information output from the individual object segmentation model (232) and / or the person object segmentation model (233) into which the image is input may indicate the result of recognizing at least one object within the image. Information output from the individual object segmentation model (232) is described with reference to FIG. 4. Information output from the person object segmentation model (233) is described with reference to FIG. 5.

[0046] According to one embodiment, a processor (210) that executes a clipper map generator (234) and / or an image editing application (231) can identify or determine an object corresponding to a main subject (e.g., an object (131) in one embodiment of FIG. 1) among the objects recognized from an image by using information output from an individual object segmentation model (232) and / or a person object segmentation model (233). For example, the main subject may be a person. The information generated using the clipper map generator (234) may indicate the location and / or size of the object corresponding to the main subject within the image. The operation of the processor (210) that executes the clipper map generator (234) is described with reference to FIG. 6 and / or FIG. 7. An object corresponding to the main subject identified using the clipper map generator (234) may not be recommended as an object to be removed from the image.

[0047] According to one embodiment, a processor (210) that executes an object recommender (235) and / or an image editing application (231) can generate or obtain information (e.g., recommendation information and / or recommendation area) indicating an object to be removed from an image by using a person (e.g., a person corresponding to the main subject) identified from information of a clipper map generator (234), an object corresponding to another person, and an object corresponding to a non-person object. The object(s) that are visually highlighted within the state (193) of FIG. 1 may be associated with the information generated by the object recommender (235).

[0048] According to one embodiment, an inpainting model (236) may be executed by a processor (210) for editing (e.g., inpainting) an image. Based on user input for selecting an option of a UI object (142-1) and / or a popup menu (150) of FIG. 1 (e.g., an option corresponding to either one of the UI objects (151, 152)), the processor (210) may execute the inpainting model (236). In the exemplary state (193) of FIG. 1, the processor (210), having identified user input related to a UI object (152), may execute the inpainting model (236) to infer the appearance of the source image (130) from which visually highlighted objects among the objects in the source image (130) have been removed. In the above example, from the information output from the inpainting model (236), the processor (210) can identify the edited image (160) of FIG. 1. In the view that the individual object segmentation model (232), the person object segmentation model (233), and / or the inpainting model (236) are installed on an electronic device (101) owned by the user, the individual object segmentation model (232), the person object segmentation model (233), and / or the inpainting model (236) may be referred to as an on-device model.

[0049] Below, with reference to FIG. 3, the operation of the processor (210) that executed the image editing application (231) is described.

[0050] FIG. 3 illustrates a flowchart for explaining operations performed by an electronic device according to one embodiment. The operations of FIG. 3 may be performed by the electronic device (101) of FIG. 1 and 2, and / or the processor (210) of FIG. 2. The order in which the operations of FIG. 3 are performed is not limited to the order shown in FIG. 3. For example, the electronic device (101) and / or the processor (210) of FIG. 2 may perform the operations of FIG. 3 in an order different from the order shown in FIG. 3. For example, the electronic device (101) and / or the processor (210) of FIG. 2 may perform at least two of the operations of FIG. 3 substantially simultaneously based on multitasking and / or multithreading. The operations of FIG. 3 may be performed by the processor (210) that runs the image editing application (231) of FIG. 2.

[0051] Referring to FIG. 3, within operation (310), an electronic device according to one embodiment can identify a source image (e.g., source image (130) of FIG. 1). The electronic device can display the source image through a display (e.g., display (110) of FIG. 2). Within the states (191, 192) of FIG. 1, the electronic device can perform operation (310).

[0052] Referring to FIG. 3, in operation (320), an electronic device according to one embodiment can identify a removable object within a source image. Based on user input selecting an icon (125-1) of FIG. 1, the electronic device can perform operation (320). In operation (320), the electronic device can execute an image editing application (231) of FIG. 2, an individual object segmentation model (232), a person object segmentation model (233), a clipper map generator (234), and / or an object recommender (235). An operation of the electronic device described with reference to FIG. 4 through 7 may be associated with operation (320).

[0053] The removable object of operation (320) may include object(s) that are different from the main subject among the objects included in the source image. For example, in the source image (160) of FIG. 1, a person referred to as object (131) may occupy a larger area than another person (e.g., a person referred to as object (132)). Also, the person referred to as object (131) may be positioned closer to the center of the source image (160) than the other person. In the above example, the electronic device may determine the remaining objects (e.g., object (132)) that are different from object (131) as removable objects. For example, the distance between a removable object and a reference point of the source image (e.g., a reference point spaced apart from the edges of the source image, such as a center point) may be longer than the distance between another object (e.g., an object that is not removable and / or the main subject) and said reference point.

[0054] Referring to FIG. 3, in operation (330), an electronic device according to one embodiment may identify or determine whether one or more removable objects have been identified in a source image. If one or more removable objects have been identified in the source image (330-Yes), the electronic device may perform operation (340). If no removable objects have been identified in the source image (330-No), the electronic device may perform operation (350) (without performing operation (340). For example, if no removable objects have been identified in the source image, the electronic device may bypass, refrain from, or forgo performing operation (340).

[0055] Referring to FIG. 3, in operation (340), an electronic device according to one embodiment may recommend removable objects within a source image. In the exemplary state (192) of FIG. 1, an electronic device that has identified one or more removable objects may display a UI object (141-1). Based on user input indicating the selection of the UI object (141-1) (e.g., user input for removing at least one object from the source image), the electronic device may recommend or visualize removable objects within the source image (130), as in the state (193) of FIG. 1. The UI object (152) of FIG. 1 (or a popup menu (150) containing the UI object (152)) may be displayed based on the user input.

[0056] Based on operation (340), the electronic device can classify objects that are removable among the objects included in the source image into a first group. The remaining objects(s) for the objects classified into the first group can be classified into a second group.

[0057] In operation (340), the electronic device may display a visual object associated with a removable object superimposed on the source image. For example, the visual object may represent the outline of the removable object. If the removable objects identified from the source image are connected to each other and / or superimposed, the visual object may represent the outline of a combination of the removable objects. The embodiments are not limited thereto, but the visual object displayed to recommend a removable object in operation (340) may include an icon superimposed on the source image (e.g., a checkbox and / or an icon in the form of “-”, “x”).

[0058] Referring to FIG. 3, within operation (350), an electronic device according to one embodiment may determine an object to be removed from a source image based on user input. The user input of operation (350) may include user input related to a visual object displayed to recommend an object that can be removed by operation (340). In a state where an object that can be removed is recommended based on operation (340) (e.g., state (193) of FIG. 1), the electronic device may receive user input to add an object that can be removed or to exclude at least one of the recommended object(s). If operation (340) is not performed (e.g., if no object that can be removed is identified), the electronic device may receive user input within operation (350) to select at least one object within the source image. Based on said user input, the electronic device may determine an object to be removed from the source image.

[0059] Referring to FIG. 3, within operation (360), an electronic device according to one embodiment may generate an edited image (e.g., an edited image (160) of FIG. 1) which is the result of removing an object determined based on operation (350) from a source image. Operation (360) may be performed in response to user input for selecting a UI object (152) of FIG. 1. The electronic device may generate or obtain an edited image of operation (360) by executing the inpainting model (236) of FIG. 2. The electronic device may generate or composite an edited image having an appearance in which an object(s) determined based on operations (340, 350) have been removed from a source image.

[0060] Referring to FIG. 3, in operation (370), an electronic device according to one embodiment may display an edited image. The electronic device may display the edited image through at least a portion of the display, such as in the state (194) of FIG. 1. Along with the edited image, the electronic device may display a visual object for storing the edited image (e.g., UI object (142-3) of FIG. 1). (Based on user input associated with the visual object) the electronic device may store the edited image in memory (e.g., memory (220) of FIG. 2).

[0061] Below, with reference to FIG. 4, exemplary information generated by an electronic device performing the operation (320) is described.

[0062] FIG. 4 illustrates an exemplary operation of an electronic device (101) that obtains first map information (e.g., map information (420)) for a source image (410) using a trained model (e.g., individual object segmentation model (232)). The electronic device (101) of FIG. 1 and 2 and / or the processor (210) of FIG. 2 may perform the operation of the electronic device (101) described with reference to FIG. 4. The operation of the electronic device (101) described with reference to FIG. 4 may be related to the individual object segmentation model (232) of FIG. 2 and / or an image editing application (231). The operation of the electronic device (101) described with reference to FIG. 4 may be related to the operation (320) of FIG. 3.

[0063] Referring to FIG. 4, an exemplary state of an electronic device (101) that has identified an exemplary source image (410) is illustrated. The source image (410) may be stored in the memory (220) of the electronic device (101) or / or transmitted from an external electronic device to the electronic device (101) via a network such as the Internet. The electronic device (101) that has identified the source image (410) may input (or apply) the source image (410) to an individual object segmentation model (232) to obtain map information (420) corresponding to the source image (410). In this disclosure, the term “map information” may be used interchangeably with terms such as map and / or confidence map. The operation of an electronic device (101) inputting a source image (410) into an individual object segmentation model (232) may include the operation of acquiring information that corresponds to the source image (410) and has a format readable by the individual object segmentation model (232). The information may include feature information, feature vectors, and / or embedding information of the source image (410). For example, the processor (210) of the electronic device (101) may acquire map information (420) by performing calculations defined by the individual object segmentation model (232) using the information. The map information (420) may be referred to as an individual object map in terms of the information acquired from the individual object segmentation model (232).

[0064] Referring to FIG. 4, map information (420) obtained using an individual object segmentation model (232) may represent the result of recognizing objects from a source image (410). The map information (420) may include pixels corresponding to regions of the source image (410). The map information (420) may be two-dimensional information mapped to the source image (410). Each pixel of the map information (420) may include a value representing an object located in a corresponding region of the source image (410). For example, if the size of the map information (420) matches the size of the source image (410) (e.g., width and / or height), the value of any pixel of the map information (420) may represent an object corresponding to a corresponding pixel within the source image (410). The value may include the ID of the object (e.g., if a person is identified, the person's name, index, and / or ID). Referring to Fig. 4, a corresponding person's ID can be assigned to each of the areas within the map information (420) corresponding to 11 people.

[0065] The objects segmented from the source image (410) using the individual object segmentation model (232) are not limited to people. The individual object segmentation model (232) can identify areas of the source image (410) corresponding to objects (e.g., umbrella, bag, paper, stairs, building, shadow, and / or mirror) as well as people. Map information (420) output from the individual object segmentation model (232) may include the result of recognizing (individual) people and / or objects from the source image (410) (e.g., a value representing the class and / or category of said object).

[0066] Referring to the source image (410) and map information (420) of FIG. 4, the map information (420) obtained from the individual object segmentation model (232) may include the result of segmenting objects having a relatively large size within the source image (410). According to one embodiment, the electronic device (101) can identify or segment areas where people(s) are captured in the source image (410) using a person object segmentation model (233).

[0067] FIG. 5 illustrates an exemplary operation of an electronic device (101) that obtains second map information (e.g., map information (530)) for a source image (410) using a trained model (e.g., a human object segmentation model (232)). The electronic device (101) of FIG. 1 and 2 and / or the processor (210) of FIG. 2 may perform the operation of the electronic device (101) described with reference to FIG. 5. The operation of the electronic device (101) described with reference to FIG. 5 may be related to the human object segmentation model (233) and / or image editing application (231) of FIG. 2. The operation of the electronic device (101) described with reference to FIG. 5 may be related to the operation (320) of FIG. 3.

[0068] Referring to FIG. 5, an electronic device (101) that has identified a source image (410) can input (or apply) the source image (410) to a person object segmentation model (233) to obtain map information (530) corresponding to the source image (410). The operation of obtaining map information (420) using an individual object segmentation model (232) and the operation of obtaining map information (530) using a person object segmentation model (233), as described with reference to FIG. 4, can be performed substantially simultaneously based on identifying the source image (410). The operation of the electronic device (101) inputting the source image (410) into the person object segmentation model (233) may include the operation of inputting feature information, feature vectors, and / or embedding information corresponding to the source image (410) into the person object segmentation model (233). The map information (530) of FIG. 5 can be referred to as a person object map in terms of information obtained from the person object segmentation model (233).

[0069] Referring to FIG. 5, map information (530) obtained using a person object segmentation model (233) may represent the result of recognizing a region corresponding to a person in a source image (410). Map information (530) may be two-dimensional information mapped to the source image (410). Each pixel of the map information (530) may include a value representing an object located in a corresponding region of the source image (410). For example, if the size of the map information (530) matches the size (e.g., width and / or height) of the source image (410), the value of any pixel of the map information (530) may indicate whether the corresponding pixel in the source image (410) represents a person. For example, if the value of a pixel of the map information (530) is 255 (or a first specified value), the pixel may indicate that a person is located in a corresponding region within the source image (410). For example, if the value of a pixel of map information (530) is 0 (or a second specified value), the pixel may indicate that no person is located in the corresponding area within the source image (410).

[0070] Referring to FIG. 5, map information (420) illustrated in FIG. 4 (e.g., map information obtained from an individual object segmentation model (232)) is shown together with a source image (410) and map information (530). When comparing the map information (530) obtained from the person object segmentation model (233) and the map information (420) obtained from the individual object segmentation model (232), the map information (530) may include results of recognizing almost all people within the source image (410), and the map information (420) may include results of recognizing people with relatively large sizes. According to one embodiment, the electronic device (101) can use map information (530) obtained from a person object segmentation model (233) and map information (420) obtained from an individual object segmentation model (232) to recommend other person(s) different from the main subject person(s) (5 people located close to the camera in the exemplary source image (410) of FIG. 4 and / or FIG. 5) as object(s) to be removed from the source image (410).

[0071] Below, with reference to FIG. 6, an exemplary operation of an electronic device (101) that determines a removable object from a source image (410) using map information (420, 530) is described.

[0072] FIG. 6 illustrates an exemplary operation of an electronic device for classifying objects included in a source image (410) using first map information (e.g., map information (420)) for the source image (410). The electronic device (101) of FIG. 1 and 2 and / or the processor (210) of FIG. 2 may perform the operation of the electronic device described with reference to FIG. 6. The operation of the electronic device described with reference to FIG. 6 may be related to the clipper map generator (234) of FIG. 2. The operation of the electronic device described with reference to FIG. 6 may be related to the operation (320) of FIG. 3.

[0073] According to one embodiment, an electronic device may obtain map information (e.g., relative size-based map information (610) and / or absolute size-based map information (620)) including pixels corresponding to regions of a source image (410). Each of the pixels of the map information may have a value based on the size of an object located in a corresponding region of the source image (410). The pixels of the absolute size-based map information (620) may represent the size of an object located in a corresponding region of the source image (410) (e.g., the number of pixels in the source image (410) corresponding to the object and / or the size of a portion of the source image (410) corresponding to the object). The pixels of the relative size-based map information (610) may represent the normalized size of an object located in a corresponding region of the source image (410) (e.g., the ratio between the object and the largest object in the source image (410)).

[0074] The electronic device may obtain a combination (e.g., a weighted sum based on weights corresponding to each of the individual object map and the person object map) of the map information (420) (e.g., individual object map) obtained by executing the individual object segmentation model (232) of FIG. 2 and the person object segmentation model (233) of FIG. 2. The electronic device may obtain relative size-based map information (610) and absolute size-based map information (620), and map information (530) obtained by executing the individual object segmentation model (233) of FIG. 2. By normalizing the map information based on the combination according to the size of the largest object in the source image (410), the electronic device may obtain or generate relative size-based map information (610). By normalizing the map information based on the combination according to the sizes of objects in the source image (410) (e.g., absolute size), the electronic device may obtain or generate absolute size-based map information (620).

[0075] According to one embodiment, an electronic device may perform a convolution operation based on a Gaussian filter (615) on relative size-based map information (610). Referring to FIG. 6, the Gaussian filter (615) may be a two-dimensional Gaussian filter having a peak value at a center point. Based on the convolution operation, the values ​​of the pixels of the relative size-based map information (610) may be mixed (e.g., blurred). When changing the value of a specific pixel using the Gaussian filter (615), the value of the specific pixel may be changed less if the values ​​of other pixels adjacent to the specific pixel are equal. For example, among the objects of the source image (410), the values ​​corresponding to objects having a relatively small size may be reduced (relatively much) by the Gaussian filter (615). For example, among the objects of the source image (410), values ​​corresponding to objects having a relatively large size can be reduced (relatively less) by the Gaussian filter (615). The map information (630) of FIG. 6 may represent the result of performing the convolution operation based on the Gaussian filter (615) on the relative size-based map information (610).

[0076] According to one embodiment, the electronic device may increase or decrease the values ​​of the pixels of the map information (630) using the location of the corresponding object in the source image (410). For example, the electronic device may obtain the map information (640) by applying a two-dimensional Gaussian filter (635) to the map information (630) (e.g., multiplication). The width and height of the two-dimensional Gaussian filter (635) may be equal to the width and height of the map information (630) (or the width and height of the source image (410)). The pixels of the two-dimensional Gaussian filter (635) may have values ​​based on a two-dimensional Gaussian function having peak values ​​at a reference point of the source image (410). The reference point may be any point of the source image (410) spaced apart from the edges of the source image (410) (e.g., a center point or any point between the center point and the bottom edge of the source image (410)). The values ​​of the pixels of the 2D Gaussian filter (635) may be floating-point numbers between 0 and 1.

[0077] According to one embodiment, an electronic device may obtain a result of applying a two-dimensional Gaussian filter (645) to map information (640) (e.g., multiplication). The electronic device may obtain map information (650) by combining the map information (640) with the result (e.g., addition and / or weighted sum). The two-dimensional Gaussian filter (645) may have values ​​based on a two-dimensional Gaussian function having a peak value at a reference point of the source image (410). The reference point of the source image (410) corresponding to the peak value of the two-dimensional Gaussian filter (645) may be the same as or different from the reference point of the source image (410) corresponding to the peak value of the two-dimensional Gaussian filter (635).

[0078] The operation of obtaining map information (650), described above with reference to FIG. 6, can be represented as Equation 1.

[0079]

[0080] In Equation 1, Sc represents relative size-based map information (610), Gf represents a Gaussian filter (615), and Sc * Gf represents a convolution operation of relative size-based map information (610) and a Gaussian filter (615) (e.g., map information (630)). Gm in Equation 1 may represent a 2D Gaussian filter (635). S2 in Equation 1 may be the result of applying weight w1 to map information (640) (e.g., multiplication). Gw in Equation 1 may represent a 2D Gaussian filter (645). Referring to Equation 1, S3, which is a weighted sum based on weights w1 and w2, may represent map information (650).

[0081] According to one embodiment, an electronic device may obtain or generate map information (660) by combining absolute size-based map information (620) with map information (650) (e.g., addition and / or weighted sum). The map information (660) may be referred to as a final confidence map. The operation of obtaining map information (660) can be represented as shown in Equation 2.

[0082]

[0083] Sf in Equation 2 may represent map information (660). Sa in Equation 2 may represent absolute size-based map information (620). w3 in Equation 2 may represent a weight corresponding to the absolute size-based map information (620). Referring to FIG. 6, map information (660) can be obtained by applying a 2D Gaussian filter (e.g., 2D Gaussian filters (635, 645)) having a peak value at a reference point of the source image (410) to the map information (e.g., relative size-based map information (610), map information (630, 640, 650)). Using the map information (660), the electronic device can classify objects within the source image (410) into either a first group or a second group for classifying the main subject.

[0084] According to one embodiment, an electronic device can identify main subject(s) within a source image (410) using map information (660). For example, the electronic device can identify pixels of map information (660) corresponding to a specific object using map information (420), which is an individual object map. By calculating representative values ​​(e.g., average value, maximum value, minimum value, mode, and / or median value) of the values ​​of the pixels, the electronic device can calculate a score (e.g., average confidence value) corresponding to the specific object. In the above example, the electronic device can calculate scores corresponding to each of the plurality of objects identified by the map information (420). Among the scores, the electronic device can determine the object corresponding to the score(s) that exceed a threshold score as the main subject.

[0085] According to one embodiment, the electronic device may identify an edge-based mask filter (670) in order not to determine an object adjacent to the edge in the source image (410) as the main subject. Within the mask filter (670), the values ​​of pixels in the area adjacent to the edge may differ from the values ​​of pixels in the area spaced apart from the edge. The mask filter (670) may have the same size as the map information (420), the map information (660), and / or the source image (410). The electronic device may identify an object adjacent to the edge by comparing the map information (420) and the mask filter (670). Even if the score of the object exceeds a threshold score, the electronic device may determine that the object is not the main subject.

[0086] As described above, an electronic device according to one embodiment can use map information (660) to generate or identify map information (e.g., a clipper map) representing the main subject object(s) among the objects of the source image (410). For example, the map information can represent the area(s) of the source image (410) corresponding to the object(s) having a score exceeding a threshold score. Below, with reference to FIG. 7, an exemplary operation is described in which the electronic device, having identified the main subject object(s), identifies the remaining objects within the source image (410).

[0087] FIG. 7 illustrates an exemplary operation of an electronic device for determining information for classifying objects included in a source image (e.g., source image (410) of FIG. 4 to 6). The electronic device (101) of FIG. 1 to 2 and / or the processor (210) of FIG. 2 may perform the operation of the electronic device described with reference to FIG. 7. The operation of the electronic device described with reference to FIG. 7 may be related to the clipper map generator (234) and / or object recommender (235) of FIG. 2. The operation of the electronic device described with reference to FIG. 7 may be related to the operation (320) of FIG. 3.

[0088] According to one embodiment, an electronic device may generate or obtain a clipper map (710) by performing the operation described with reference to FIG. 6 on the source image (410) of FIG. 4. The electronic device may generate an initial recommended area map (720) based on map information (420), which is an individual object map, and the clipper map (710). For example, the electronic device may obtain or generate the initial recommended area map (720) by excluding objects that appear as main subjects by the clipper map (710) from the map information (420). In the above example, the result of expanding the area that appears as main subjects by the clipper map (710) may be used. In the above example, the result of shrinking (e.g., erosion) each area of ​​the objects appearing by the map information (420) may be used. The expansion of the clipper map (710) and the reduction of the map information (420) can be performed to align the clipper map (710) and the map information (420). The expansion of the clipper map (710) and the reduction of the map information (420) can be performed according to the size of the object. Connected component labeling can be performed for the operation of the map information (420) and the clipper map (710).

[0089] In one embodiment, the electronic device can identify the number (e.g., a first number) of pixels corresponding to a specific object (e.g., pixels within a source image (410)) identified from map information (420), which is an individual object map. The electronic device can identify the number (e.g., a second number) of pixels that overlap with a clipper map (720) among the pixels corresponding to the specific object. The electronic device can calculate the ratio between the first number and the second number. If the calculated ratio is below a threshold ratio, the electronic device can add the area corresponding to the object to the initial recommended area map (720).

[0090] According to one embodiment, an electronic device may use map information (530), which is a person object map, to identify object(s) of a source image (410) that are different from the main subject. For example, the electronic device may obtain or generate a residual map (730) by combining the map information (530) and the individual object map information (420) (e.g., a combination based on subtraction). For example, the pixels of the residual map (730) may have values ​​representing the difference between the pixels of the map information (530) and the pixels of the map information (420). As described above with reference to FIG. 5, since the person object map information (530) includes results of identifying people having a smaller size in the source image (410) more than the individual object map information (420), the residual map (730) may include more information about people not included in the map information (420).

[0091] According to one embodiment, the electronic device may perform operations to reduce or remove noise in the residual map (730). The operations may include erosion operations. When a person is standing, the horizontal length (e.g., width) of the area corresponding to the person within the source image (410) may be shorter than the vertical length (e.g., height). For example, the electronic device may perform erosion operations along the vertical length to remove or reduce noise caused by objects different from the person from the residual map (730). The electronic device may obtain or generate a final recommended area map (740) by combining the initial recommended area map (720) and the residual map (730) (e.g., the noise-reduced residual map (730)). The final recommended area map (740) may represent the remaining objects that are different from the main subject among the objects of the source image (410).

[0092] According to one embodiment, an electronic device may combine a final recommended area map (740) with map information (420), which is an individual object map, to obtain or generate map information (750), which is an updated individual object map. The map information (750) may be used to (quickly) identify an object corresponding to a user input in response to a user input to add an object to be removed from a source image (410). An exemplary operation of an electronic device for identifying an object corresponding to a user input using the map information (750) is described with reference to FIG. 9.

[0093] Hereinafter, with reference to FIG. 8, an exemplary operation of an electronic device for identifying a first object corresponding to a person and a second object related to said person as objects to be removed from a source image is described.

[0094] FIG. 8 illustrates an exemplary operation of an electronic device for determining at least one object to be removed from a source image (810) using map information representing the result of recognizing objects included in a source image (810). The electronic device (101) of FIG. 1 and 2 and / or the processor (210) of FIG. 2 may perform the operation of the electronic device described with reference to FIG. 8. The operation of the electronic device described with reference to FIG. 8 may be related to the object recommender (235) of FIG. 2. The operation of the electronic device described with reference to FIG. 8 may be related to the operation (320) of FIG. 3.

[0095] Referring to FIG. 8, an exemplary source image (810) is illustrated. The source image (810) may include objects that are people (811, 812, 813, 814, 816) and objects (815, 817) that are objects related to the person corresponding to the object (814). When identifying objects that can be removed from the source image (810) based on the operation described with reference to FIG. 4 through 7, objects corresponding to the person (811, 812, 813, 814, 816) may be identified, and objects related to the person (e.g., objects held by the person and / or objects worn by the person, such as objects (815, 817)) may not be identified. According to one embodiment, an electronic device may perform the operation described with reference to FIG. 8 to at least partially modify the final recommendation map (740) of FIG. 7. The operation described with reference to FIG. 8 may be a post-processing operation for the clipper map generator (234) and / or object recommender (235) of FIG. 2.

[0096] Referring to FIG. 8, an individual object map (820) obtained from a source image (810) is shown. An electronic device may obtain or generate an individual object map (820) for a source image (810) by executing the individual object segmentation model (232) of FIG. 2. The individual object map (820) may include regions (821, 822, 823, 824, 825, 826, 827) corresponding to each of the objects (811, 812, 813, 814, 815, 816, 817) of the source image (810). Using the individual object map (820), the electronic device may obtain or generate a first recommended region map (830) by performing the operation described with reference to FIG. 6 and / or FIG. 7. The first recommended area map (830) of FIG. 8 may correspond to the initial recommended area map (720) and / or the final recommended area map (740) of FIG. 7. The first recommended area map (830) may include areas (822, 823, 824, 826) corresponding to the remaining objects (812, 813, 814, 816) that are different from the main subject object (811) among the objects (811, 812, 813, 814, 816) that correspond to people.

[0097] According to one embodiment, an electronic device may compare an individual object map (820) and a first recommended area map (830) to determine at least one of the areas (821, 825, 827) of the individual object map (820) that is not included in the first recommended area map (830) as an area to be added to the first recommended area map (830). For example, the electronic device may identify from the individual object map (820) an area (e.g., areas (825, 827)) connected to areas (822, 823, 824, 825) of the first recommended area map (830).

[0098] In one embodiment, the electronic device may determine whether to add the object to the first recommended area map (830) by using the type (or class or category) of the object included in the source image (810). For example, an object that can be held by a person, worn by a person, or removed along with a person, such as a hat, umbrella, bag, book, pen, mirror, and / or shadow, may be determined as an object to be added to the first recommended area map (830). Referring to FIG. 8, since the object (815) which is a hat is connected to an area (824) included in the first recommended area map (830) and is an object that can be worn by a person, the electronic device may add an area (825) corresponding to the object (815) to the first recommended area map (830).

[0099] In one embodiment, the electronic device may identify or determine an area to be added to the first recommended area map (830) by using the ratio of the sizes of the areas. For example, if an area (825) corresponding to an object (815) that is a hat is connected to an area (824) of the first recommended area map (830), the electronic device may obtain the ratio of the sizes of the areas (824, 825). For example, in a photograph of a person wearing a hat, if the ratio of the size of the area corresponding to the person and the area corresponding to the hat is 5:1, the electronic device may determine whether to add the area (825) to the first recommended area map (830) based on whether the ratio of the sizes of the areas (824, 825) corresponds to (or is similar to) the 5:1 ratio.

[0100] Referring to FIG. 8, a second recommended area map (840) is shown as a result of adding areas (825, 827) to a first recommended area map (830). The second recommended area map (840) may include areas (822, 823, 824, 826) corresponding to people different from the main subject and areas (825, 827) corresponding to objects related to said people, among the remaining areas from which the area (821) corresponding to the main subject object (811) has been excluded. The second recommended area map (840) may be used to recommend objects that can be removed from a source image (810) (e.g., operation (340) of FIG. 3).

[0101] Hereinafter, with reference to FIG. 9, an exemplary operation of an electronic device recommending removable object(s) from a source image (e.g., source image (130) of FIG. 1, source image (410) of FIG. 4, and / or source image (810) of FIG. 8) using the final recommendation map (740) of FIG. 7 and / or the second recommendation area map (840) of FIG. 8 is described.

[0102] FIG. 9 illustrates exemplary states (193, 914, 915) of an electronic device (101) receiving user input for editing a source image (130). The electronic device (101) of FIG. 1 and 2 and / or the processor (210) of FIG. 2 may perform the operation of the electronic device (101) described with reference to FIG. 9. The operation of the electronic device (101) described with reference to FIG. 9 may be performed based on the execution of the image editing application (231) of FIG. 2. The operation of the electronic device (101) described with reference to FIG. 9 may be related to the operations of FIG. 3.

[0103] Referring to FIG. 9, an exemplary state (193) of an electronic device (101) displaying a source image (130) is illustrated. The state (193) of FIG. 9 may correspond to the state (193) of FIG. 1. Referring to the exemplary state (193) of FIG. 9, the electronic device (101) can visually highlight remaining objects (e.g., object (132)) that are different from the main subject object (131) by using a final recommendation map (e.g., final recommendation map (740) of FIG. 7) obtained based on the operation described with reference to FIG. 3 through FIG. 8. For example, the electronic device (101) can visually highlight a first group of the remaining objects, including object (132), with respect to a second group, including the main subject object (131). While recommending the remaining objects as objects to be removed from the source image (130), the electronic device (101) may receive user input to add, delete, and / or change the objects to be removed from the source image (130). The user input may include touch gestures (e.g., tap gestures) performed on the source image (130).

[0104] For example, the electronic device (101) may remove the object (132) from the object to be removed from the source image (130) based on identifying a touch gesture on the point (p1) where the object (132) is displayed while the object (132) is visually highlighted. The touch gesture may be included in a user input for selecting the object (132). The touch gesture may be included in a user input indicating the deselection of the object (132) selected as the object to be removed from the source image (130). The electronic device (101) that identifies the touch gesture on the point (p1) within state (193) may stop visually highlighting the object (132), as in state (914). Referring to state (914) of FIG. 9, the remaining objects, which are different from the main subject object (131) and the object (132) corresponding to the touch gesture, may be visually highlighted.

[0105] In the state (914) of FIG. 9, an electronic device (101) that receives user input for creating an edited image (e.g., user input related to at least one of UI objects (152, 142-1)) can acquire or create an edited image (940). Referring to the edited image (940) of FIG. 9, objects (131, 132) can be retained, and the remaining objects (e.g., objects visually highlighted in the state (914)) can be removed.

[0106] In state (193) of FIG. 9, the electronic device (101) may receive user input to add an object to be removed from the source image (130) while visually highlighting specific objects. For example, the electronic device (101) may identify a touch gesture on a point (p2) on the source image (130) (e.g., a point corresponding to an object different from the visually highlighted objects). The touch gesture may be included in the user input to add an object corresponding to the touch gesture as an object to be removed from the source image (130). Upon identifying the touch gesture on the point (p2) in state (193), the electronic device (101) may visually highlight an object (920) located at the point (p2), as in state (915). For example, the electronic device (101) may display a visual object having the shape of the outline of the object (920) superimposed on the source image (130). An electronic device (101) that identifies a touch gesture on point (p2) can identify an object (920) located on point (p2) by using map information corresponding to the source image (910) (e.g., map information (420) of FIG. 4, map information (750) of FIG. 7, and / or the second recommended area map (840) of FIG. 8). Based on identifying the object (920), the electronic device (101) can visually highlight and display the object (920) within the source image (130).

[0107] In the state (915) of FIG. 9, an electronic device (101) that receives user input for creating an edited image (e.g., user input related to at least one of the objects (152, 142-1)) can obtain or create an edited image (950). For example, the electronic device (101) can obtain or create an edited image (950) in which the main subject object (131) is retained and the visually highlighted objects (e.g., objects (132, 920)) are removed. The edited image (e.g., edited images (940, 950)) can be obtained by executing the inpainting model (236) of FIG. 2.

[0108] According to one embodiment, an electronic device (101) may obtain or generate map information (e.g., a final removal map) representing area(s) within a source image (130) corresponding to visually highlighted object(s). By using the map information, the electronic device (101) may generate an edited image by executing an inpainting model (e.g., the inpainting model (236) of FIG. 2). For example, by inputting the map information and the source image (130) into the inpainting model, the electronic device (101) may generate or obtain an edited image that represents the appearance of the object(s) expressed in the area(s) indicated by the map information being removed. An electronic device (101) that has obtained any one of the exemplary edited images (940, 950) of FIG. 9 may display the obtained edited image through a display (110). The edited image obtained by the electronic device (101) can be displayed at the location where the source image (130) was displayed.

[0109] FIG. 10 illustrates an exemplary operation of an electronic device (101) that receives user input for editing a source image (1010). The electronic device (101) of FIG. 1 and 2 and / or the processor (210) of FIG. 2 may perform the operation of the electronic device (101) described with reference to FIG. 10. The operation of the electronic device (101) described with reference to FIG. 10 may be performed based on the execution of the image editing application (231) of FIG. 2. The operation of the electronic device (101) described with reference to FIG. 10 may be related to the operations of FIG. 3.

[0110] Referring to FIG. 10, an exemplary state (1001) of an electronic device (101) displaying a source image (1010) is illustrated. The state (1001) of FIG. 10 may be an exemplary state that recommends objects that can be removed within the source image (1010) based on the operation (340) of FIG. 3. For example, the electronic device (101) may visually highlight objects (e.g., object (1012)) that are different from the main subject object (1011) based on the operation described with reference to FIG. 3 through FIG. 8. For example, the electronic device (101) may visually highlight a first group of objects including object (1012) to a second group of objects including the main subject object (1011). The electronic device (101) may display a pop-up menu (150) along with the visually highlighted objects.

[0111] Referring to FIG. 10, the popup menu (150) may include a UI object (152) corresponding to an option for removing visually highlighted objects from a source image (1010). Based on user input related to the UI object (152), the electronic device (101) may perform the operation (360) of FIG. 3 to generate or display an edited image from which the visually highlighted objects have been removed. The operation of the electronic device (101) based on the UI object (152) may be performed similarly to the descriptions in FIG. 1 through 9.

[0112] Referring to FIG. 10, the popup menu (150) may include a UI object (151) corresponding to an option for moving visually highlighted objects. In the state (1001) of FIG. 10, an electronic device (101) that has identified user input related to the UI object (151) (e.g., a tap gesture on a portion of the display (110) on which the UI object (151) is displayed) may receive (further) user input to change and / or determine the position of visually highlighted objects (e.g., objects (1012)) within the state (1001). The state (1002) of FIG. 10 may be a state after receiving user input to change the position of the visually highlighted objects. Based on receiving the user input, the electronic device (101) may obtain or generate an edited image containing the objects located at the changed position within the source image (1010). For example, within a state (1002), the electronic device (101) may generate or display an edited image including an object (1012) having a position corresponding to the user input.

[0113] Referring to FIG. 10, the popup menu (150) may include a UI object (1030) corresponding to an option for changing visually highlighted objects. In the state (1001) of FIG. 10, the electronic device (101) that receives user input related to the UI object (1030) may display a UI for receiving user input for changing visually highlighted objects, as in the state (1003) of FIG. 10. Referring to the state (1003) of FIG. 10, the electronic device (101) may display a text box (1031) in part (142) for receiving a prompt (e.g., text based on natural language to be input into the inpainting model (236) of FIG. 2) to be used to change visually highlighted objects. Based on user input related to the text box (1031), the electronic device (101) may display a software keyboard (or virtual keyboard). Text received through the software keyboard can be displayed in the text box (1031).

[0114] Referring to the state (1003) of FIG. 10, the electronic device (101), along with the text box (1031), may display UI objects (1032, 1033). The UI object (1033), which takes the form of a button containing the text “Cancel”, may be displayed to receive user input to cancel the change of the visually highlighted objects based on the state (1003). For example, the electronic device (101), upon receiving user input related to the UI object (1033), may switch to the state (1001) of FIG. 10. In the above example, the electronic device (101) may stop displaying the text box (1031) and the UI objects (1032, 1033) through the portion (142).

[0115] For example, a UI object (1032) having the form of a button containing the text “OK” may be displayed to receive user input for changing visually highlighted objects using a prompt received through a text box (1031). For example, an electronic device (101) that receives user input related to the UI object (1032) may execute the inpainting model (236) of FIG. 2 using the prompt displayed by the text box (1031) to generate or obtain an edited image representing the objects (e.g., object (1012)) changed based on the prompt. In the above example, an electronic device (101) that obtains a prompt such as “cover with mosaic” may obtain the edited image by applying a mosaic visual effect to the visually highlighted objects including object (1012). In the above example, the electronic device (101) may display the obtained edited image through a display (110). The embodiments are not limited thereto, and the electronic device (101) may receive a prompt for changing the hue, brightness, and / or style through a text box (1031).

[0116] Hereinafter, with reference to FIG. 11, the operation of the electronic device (101) described with reference to FIG. 1 to FIG. 10 is described.

[0117] FIG. 11 illustrates a flowchart for explaining operations performed by an electronic device according to one embodiment. The operations of FIG. 11 may be performed by the electronic device (101) of FIG. 1 and 2, and / or the processor (210) of FIG. 2. The order in which the operations of FIG. 11 are performed is not limited to the order shown in FIG. 11. For example, the electronic device (101) and / or the processor (210) of FIG. 2 may perform the operations of FIG. 11 in an order different from the order shown in FIG. 11. For example, the electronic device (101) and / or the processor (210) of FIG. 2 may perform at least one or two of the operations of FIG. 11 substantially simultaneously (e.g., multitasking and / or multithreading). The operations of FIG. 11 may be performed by the processor (210) that runs the image editing application (231) of FIG. 2. The operations of FIG. 11 may be at least partially related to the operations of FIG. 3.

[0118] Referring to FIG. 11, within an operation (1110), an electronic device according to one embodiment may display a source image. The state (191), state (192), and / or state (1001) of FIG. 10 may be the state of the electronic device that performed the operation (1110). The source image of the operation (1110) may include the source image (130) of FIG. 1, the source image (410) of FIG. 4, the source image (810) of FIG. 8, and / or the source image (1010) of FIG. 10.

[0119] Referring to FIG. 11, in operation (1120), an electronic device according to one embodiment can identify a plurality of objects within a source image. The electronic device can identify at least one object within the source image being displayed based on operation (1110) by executing a trained model such as the individual object segmentation model (232) of FIG. 2. Based on identifying the plurality of objects, the electronic device can perform an operation (e.g., operation (1130)) to identify a removable object among the plurality of objects. The operation (1120) of FIG. 11 can be performed similarly to the operation (320) of FIG. 3.

[0120] Referring to FIG. 11, in operation (1130), an electronic device according to one embodiment may display a UI object for recommending at least one of a plurality of objects as an object to be removed from a source image. The UI object of operation (1130) may include the UI object (141-1) of FIG. 1. The electronic device may perform operation (1130) while having identified a plurality of objects based on operation (1120).

[0121] Referring to FIG. 11, within an operation (1140), an electronic device according to one embodiment may, based on receiving a first user input related to a UI object of operation (1130), display a first group of objects among a plurality of objects with visual emphasis over a second group of objects. The second group of objects may be the main subject(s) within the source image. The first group of objects may be objects that are removable from the source image and may be objects represented by the final recommended area map (740) of FIG. 7 and / or the second recommended area map (840) of FIG. 8. To visually emphasize the first group of objects, the electronic device may display a visual object representing the outlines of the first group of objects superimposed on the source image. The state (193) of FIG. 1, the states (193, 914, 915) of FIG. 9, and / or the states (1001, 1002, 1003) of FIG. 10 may be states of an electronic device that has performed the operation (1140). The operation (1140) of FIG. 11 may be performed similarly to the operation (340) of FIG. 3.

[0122] Referring to FIG. 11, within operation (1150), an electronic device according to one embodiment may receive a second user input for selecting an object to be removed from a source image. The second user input of operation (1150) may include the user input described with reference to FIG. 9 (e.g., a touch gesture on point (p1) and / or a touch gesture on point (p2) of FIG. 9). Based on the second user input, the electronic device may identify or determine one or more objects to be removed from the source image. The operation (1150) of FIG. 11 may be performed similarly to the operation (350) of FIG. 3.

[0123] Referring to FIG. 11, within operation (1160), an electronic device according to one embodiment may obtain an edited image from which at least one object has been removed based on receiving a third user input for generating an edited image. The third user input of operation (1160) may include user input for selecting UI objects (152, 142-1) of FIG. 1. Based on the third user input, the electronic device may execute the inpainting model (236) of FIG. 2 to obtain or generate an edited image representing a source image from which at least one object has been removed, determined based on operation (1140). The edited image of operation (1160) may include the edited image (160) of FIG. 1 and / or the edited images (940, 950) of FIG. 9. The operation (1160) of FIG. 11 may be performed similarly to the operation (360) of FIG. 3.

[0124] Referring to FIG. 11, within operation (1170), an electronic device according to one embodiment may display an edited image. The state (194) of FIG. 1 may be the state of the electronic device that performed operation (1170). The operation (1170) of FIG. 11 may be performed similarly to the operation (370) of FIG. 3.

[0125] As described above, according to one embodiment, an electronic device may recommend at least one object to be removed from a source image before removing an object from a source image. For example, the electronic device may recognize or recommend remaining objects within the source image that are different from a specific object that is the main subject as at least one object to be removed from the source image. Based on the recommendation, the electronic device may automatically select and / or remove the remaining objects. Based on the recommendation, user input for selecting the remaining objects may be reduced, and an edited image with the remaining objects removed may be generated more quickly.

[0126] Hereinafter, with reference to FIG. 12, an exemplary hardware structure of the electronic device described with reference to FIG. 1 to FIG. 11 is described.

[0127] FIG. 12 is a block diagram of an electronic device (1201) in a network environment (1200) according to various embodiments. Referring to FIG. 12, in the network environment (1200), the electronic device (1201) may communicate with an electronic device (1202) through a first network (1298) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (1204) or a server (1208) through a second network (1299) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1201) may communicate with the electronic device (1204) through a server (1208). According to one embodiment, the electronic device (1201) may include a processor (1220), memory (1230), input module (1250), sound output module (1255), display module (1260), audio module (1270), sensor module (1276), interface (1277), connection terminal (1278), haptic module (1279), camera module (1280), power management module (1288), battery (1289), communication module (1290), subscriber identification module (1296), or antenna module (1297). In some embodiments, at least one of these components (e.g., connection terminal (1278)) may be omitted from the electronic device (1201), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (1276), camera module (1280), or antenna module (1297)) may be integrated into a single component (e.g., display module (1260)).

[0128] The processor (1220) can, for example, execute software (e.g., program (1240)) to control at least one other component (e.g., hardware or software component) of the electronic device (1201) connected to the processor (1220) and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (1220) can store commands or data received from other components (e.g., sensor module (1276) or communication module (1290)) in volatile memory (1232), process the commands or data stored in volatile memory (1232), and store the resulting data in non-volatile memory (1234). According to one embodiment, the processor (1220) may include a main processor (1221) (e.g., a central processing unit or an application processor) or an auxiliary processor (1223) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (1201) includes a main processor (1221) and an auxiliary processor (1223), the auxiliary processor (1223) may be configured to use less power than the main processor (1221) or to be specialized for a specified function. The auxiliary processor (1223) may be implemented separately from the main processor (1221) or as part thereof.

[0129] The auxiliary processor (1223) may control at least some of the functions or states associated with at least one component of the electronic device (1201) (e.g., display module (1260), sensor module (1276), or communication module (1290)) on behalf of the main processor (1221) while the main processor (1221) is in an inactive (e.g., sleep) state, or together with the main processor (1221) while the main processor (1221) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (1223) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (1280) or communication module (1290)). According to one embodiment, the auxiliary processor (1223) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (1201) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (1208)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0130] The memory (1230) can store various data used by at least one component of the electronic device (1201) (e.g., processor (1220) or sensor module (1276)). The data may include, for example, input data or output data for software (e.g., program (1240)) and related commands. The memory (1230) may include volatile memory (1232) or non-volatile memory (1234).

[0131] The program (1240) may be stored as software in memory (1230) and may include, for example, an operating system (1242), middleware (1244), or an application (1246).

[0132] The input module (1250) can receive commands or data to be used for a component of the electronic device (1201) (e.g., processor (1220)) from outside the electronic device (1201) (e.g., user). The input module (1250) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0133] The sound output module (1255) can output a sound signal to the outside of the electronic device (1201). The sound output module (1255) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0134] The display module (1260) can visually provide information to an external (e.g., user) of the electronic device (1201). The display module (1260) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (1260) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0135] The audio module (1270) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (1270) can acquire sound through the input module (1250) or output sound through the sound output module (1255) or an external electronic device (e.g., electronic device (1202)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (1201).

[0136] The sensor module (1276) can detect the operating state of the electronic device (1201) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (1276) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0137] The interface (1277) may support one or more specified protocols that can be used for the electronic device (1201) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (1202)). According to one embodiment, the interface (1277) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0138] The connection terminal (1278) may include a connector through which the electronic device (1201) can be physically connected to an external electronic device (e.g., electronic device (1202)). According to one embodiment, the connection terminal (1278) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0139] The haptic module (1279) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (1279) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0140] The camera module (1280) can capture still images and video. According to one embodiment, the camera module (1280) may include one or more lenses, image sensors, image signal processors, or flashes.

[0141] The power management module (1288) can manage the power supplied to the electronic device (1201). According to one embodiment, the power management module (1288) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0142] The battery (1289) can supply power to at least one component of the electronic device (1201). According to one embodiment, the battery (1289) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0143] The communication module (1290) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (1201) and an external electronic device (e.g., electronic device (1202), electronic device (1204), or server (1208)), and the performance of communication through the established communication channel. The communication module (1290) may include one or more communication processors that operate independently of the processor (1220) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1290) may include a wireless communication module (1292) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (1294) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (1204) through a first network (1298) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (1299) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1292) can identify or authenticate the electronic device (1201) within a communication network such as the first network (1298) or the second network (1299) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (1296).

[0144] The wireless communication module (1292) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (1292) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (1292) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (1292) can support various requirements specified in the electronic device (1201), external electronic device (e.g., electronic device (1204)), or network system (e.g., second network (1299)). According to one embodiment, the wireless communication module (1292) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0145] An antenna module (1297) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (1297) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (1297) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (1298) or a second network (1299), may be selected from the plurality of antennas, for example, by a communication module (1290). A signal or power may be transmitted or received between the communication module (1290) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (1297).

[0146] According to various embodiments, the antenna module (1297) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0147] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0148] According to one embodiment, commands or data may be transmitted or received between an electronic device (1201) and an external electronic device (1204) through a server (1208) connected to a second network (1299). Each of the external electronic devices (1202, or 1204) may be the same or a different type of device as the electronic device (1201). According to one embodiment, all or part of the operations performed on the electronic device (1201) may be performed on one or more of the external electronic devices (1202, 1204, or 1208). For example, if the electronic device (1201) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (1201) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (1201). The electronic device (1201) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (1201) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (1204) may include an Internet of Things (IoT) device. The server (1208) may be an intelligent server using machine learning and / or neural networks.According to one embodiment, an external electronic device (1204) or server (1208) may be included within the second network (1299). The electronic device (1201) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0149] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0150] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0151] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware and may be used interchangeably with terms such as logic, block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0152] Various embodiments of the present document may be implemented as software (e.g., program (1240)) comprising one or more instructions stored in a storage medium (e.g., internal memory (1236) or external memory (1238)) readable by a machine (e.g., electronic device (1201)). For example, a processor (e.g., processor (1220)) of the machine (e.g., electronic device (1201)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0153] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0154] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0155] In one embodiment, a method for determining at least one object to be removed based on inpainting among objects in a source image may be required. In one embodiment, a method for generating and / or compositing an edited image from a source image in which at least one object distinct from the main subject of the source image is removed may be required. An electronic device (e.g., electronic device (101) of FIG. 1 and / or electronic device (1201) of FIG. 12) according to one embodiment as described above may include a display (e.g., display (110) of FIG. 2), a memory (e.g., memory (220) of FIG. 2) comprising one or more storage media for storing instructions, and at least one processor (e.g., processor (210) of FIG. 2) comprising a processing circuit. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause a source image (e.g., source image of FIG. 1 (130), source image of FIG. 4 (410), source image of FIG. 8 (810), and / or source image of FIG. 10 (1010)) to be displayed through the display. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause a plurality of objects within the source image (e.g., objects of FIG. 1 (131, 132), objects of FIG. 8 (811, 812, 813, 814, 815, 816, 817), objects of FIG. 9 (920), and / or objects of FIG. 10 (1011, 1012)).When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause a UI (user interface) object (UI object (141-1) of FIG. 1) to be displayed for removing at least one of the plurality of objects based on identifying the plurality of objects within the source image. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause a first group of objects (e.g., object (132) of FIG. 1) among the plurality of objects to be visually highlighted against a second group of objects (e.g., object (131) of FIG. 1) based on receiving a first user input related to the UI object. The distance between the reference point of the source image and the objects in the second group may be smaller than the distance between the reference point of the source image and the objects in the first group. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the first group of objects to be removed according to inpainting and the second group of objects to be retained, based on receiving a second user input for generating an edited image (e.g., the edited image (160) of FIG. 1 and / or the edited images (940, 950)) while the first group of objects are visually highlighted. According to one embodiment, the electronic device may determine at least one object to be removed based on inpainting among the objects in the source image. According to one embodiment, the electronic device may generate or composite an edited image from the source image in which at least one object distinct from the main subject of the source image is removed.

[0156] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause a second UI object (e.g., UI objects (152, 142-1) of FIG. 1) for receiving the second user input to be superimposed on the source image based on receiving the first user input.

[0157] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause a third UI object (e.g., UI object (151) of FIG. 1) for moving the first group of objects within the source image, together with the second UI object, to be superimposed on the source image.

[0158] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may receive a fourth user input to change the position of the first group of objects based on receiving a third user input related to the third UI object. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may receive the fourth user input to obtain an edited image including the first group of objects located at the changed position within the source image based on receiving the fourth user input.

[0159] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause to obtain the edited image in which the remaining objects among the first group of objects, excluding the object selected by the third user input, based on receiving a third user input for selecting one of the first group of objects while visually highlighting the first group of objects with respect to the second group of objects.

[0160] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the first group of objects and the other object selected by the third user input to be removed, based on receiving a third user input for selecting a different object within the source image that is different from the first group of objects while visually highlighting the first group of objects against the second group of objects.

[0161] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause to acquire a first map comprising pixels corresponding to regions of the source image. Each of the pixels of the first map may have a value representing the size of an object located in a corresponding region among the plurality of objects. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause to acquire a second map by applying a two-dimensional Gaussian filter having a peak value at the reference point of the source image to the first map. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause to classify the objects into either the first group or the second group using the second map.

[0162] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the edited image to be displayed through the display as a substitute for the source image.

[0163] In one embodiment, a method of an electronic device including a display may be provided. The method may include an operation of displaying a source image through the display. The method may include an operation of identifying a plurality of objects within the source image. Based on identifying the plurality of objects within the source image, the method may include an operation of displaying a UI object for removing at least one of the plurality of objects. Based on receiving a first user input related to the UI object, the method may include an operation of visually highlighting a first group of objects among the plurality of objects with respect to a second group of objects. The distance between a reference point of the source image and the objects within the second group may be smaller than the distance between the reference point of the source image and the objects within the first group. Based on receiving a second user input for creating an edited image while the objects of the first group are visually highlighted, the method may include an operation of obtaining an edited image in which the objects of the first group are removed by inpainting and the objects of the second group are retained.

[0164] For example, the above method may include an operation of displaying a second UI object for receiving the second user input on the source image in a superimposed manner, based on receiving the first user input.

[0165] For example, the operation of displaying the second UI object may include the operation of displaying a third UI object for moving the object of the first group within the source image in a superimposed manner on the source image.

[0166] For example, the above method may include an operation of receiving a fourth user input to change the position of the first group of objects based on receiving a third user input related to the third UI object. The above method may include an operation of obtaining an edited image including the first group of objects located at the changed position within the source image based on receiving the fourth user input.

[0167] For example, the operation to acquire the above may include, based on receiving a third user input for selecting one of the objects of the first group while visually highlighting the objects of the first group with respect to the objects of the second group, the operation to acquire the edited image in which the remaining objects among the objects of the first group, excluding the object selected by the third user input, are removed.

[0168] For example, the operation to acquire the above may include, based on receiving a third user input for selecting a different object within the source image that is different from the object of the first group while visually highlighting the object of the first group with respect to the object of the second group, the operation to acquire the edited image from which the object of the first group and the other object selected by the third user input have been removed.

[0169] For example, the operation of visually highlighting and displaying the objects of the first group may include the operation of obtaining a first map containing pixels corresponding to regions of the source image. Each of the pixels of the first map may have a value representing the size of an object located in a corresponding region among the plurality of objects. The operation of visually highlighting and displaying the objects of the first group may include the operation of obtaining a second map by applying a two-dimensional Gaussian filter having a peak value at the reference point of the source image to the first map. The operation of visually highlighting and displaying the objects of the first group may include the operation of classifying the objects into either the first group or the second group using the second map.

[0170] For example, the above method may include the operation of displaying the edited image as a substitute for the source image through the display.

[0171] In one embodiment, a non-transient computer-readable storage medium comprising instructions may be provided. When executed by an electronic device including a display, the instructions may cause the electronic device to display a source image through the display. When executed by the electronic device, the instructions may cause the electronic device to receive a first user input for removing at least one object from the source image. When executed by the electronic device, the instructions may cause the electronic device to display a visual object associated with a first object among a plurality of objects included in the source image in a superposition on the source image based on the first user input. When executed by the electronic device, the instructions may cause the electronic device to obtain an edited image in which the first object among the plurality of objects is removed and the second object among the plurality of objects is retained, based on receiving a second user input for removing the first object associated with the visual object. When the above instructions are executed by the electronic device, the electronic device may cause the edited image to be displayed through the display.

[0172] For example, when the above instructions are executed by the electronic device, the electronic device may cause the visual object representing the outline of the first object to be superimposed on the source image.

[0173] For example, when the above instructions are executed by the electronic device, the electronic device may cause a UI object for receiving the first user input to be displayed through the display based on identifying the plurality of objects from the source image.

[0174] For example, the distance from a reference point spaced apart from the edges of the source image of the first object may be longer than the distance between the second object and the reference point.

[0175] According to one embodiment, an electronic device comprises a display, a memory including one or more storage media for storing instructions, and at least one processor including a processing circuit, wherein the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to display a source image through the display. The instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to receive a first user input for removing at least one object from the source image. The instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to superimpose a visual object associated with a first object among a plurality of objects included in the source image onto the source image based on the first user input. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the first object among the plurality of objects to be removed and the second object among the plurality of objects to be retained, based on receiving a second user input for removing the first object associated with the visual object. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the edited image to be displayed through the display.

[0176] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the visual object representing the outline of the first object to be superimposed on the source image.

[0177] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause a UI object for receiving the first user input to be displayed through the display based on identifying the plurality of objects from the source image.

[0178] For example, the distance from a reference point spaced apart from the edges of the source image of the first object may be longer than the distance between the second object and the reference point.

[0179] As used herein, the term “if” will be understood, depending on the context, to mean “when, upon,” “in response to a decision,” or “in response to a detection.” Similarly, “when decided to,” or “when [the mentioned condition or event] is detected,” will be understood, optionally, to mean “when decided,” or “in response to a decision,” “when [the mentioned condition or event] is detected,” or “in response to detecting [the mentioned condition or event].”

[0180] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0181] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0182] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or several hardware combined, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0183] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0184] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. In an electronic device, display; Memory comprising one or more storage media for storing instructions; and It includes at least one processor including a processing circuit, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Displaying a source image through the above display; Identifying multiple objects within the above source image; Based on identifying the plurality of objects within the source image, a UI (user interface) object for removing at least one of the plurality of objects is displayed; Based on receiving a first user input related to the above UI object, among the plurality of objects, the objects of the first group are visually highlighted with respect to the objects of the second group, and the distance between the reference point of the source image and the objects within the second group is smaller than the distance between the reference point of the source image and the objects within the first group; and Based on receiving a second user input to generate an edited image while the objects of the first group are visually highlighted, causing the objects of the first group to be removed according to inpainting and the objects of the second group to be retained in an edited image. Electronic device.

2. In claim 1, when the instructions are executed individually or collectively by the at least one processor, the electronic device: Based on receiving the first user input, causing a second UI object for receiving the second user input to be superimposed on the source image. Electronic device.

3. In claim 2, when the instructions are executed individually or collectively by the at least one processor, the electronic device: Causing a third UI object for moving the object of the first group within the source image, together with the second UI object, to be displayed superimposed on the source image. Electronic device.

4. In claim 3, when the instructions are executed individually or collectively by the at least one processor, the electronic device: Based on receiving a third user input related to the third UI object, receiving a fourth user input to change the position of the object of the first group; and Causing to obtain an edited image including the first group of objects located at the changed location within the source image, based on receiving the fourth user input. Electronic device.

5. In claims 1 to 4, when the instructions are executed individually or collectively by the at least one processor, the electronic device: While visually highlighting the objects of the first group with respect to the objects of the second group, based on receiving a third user input for selecting one of the objects of the first group, causing to obtain the edited image in which the remaining objects among the objects of the first group, excluding the object selected by the third user input, are removed. Electronic device.

6. In claims 1 to 5, when the instructions are executed individually or collectively by the at least one processor, the electronic device: While visually highlighting the objects of the first group with respect to the objects of the second group, based on receiving a third user input for selecting another object different from the objects of the first group within the source image, causing to obtain the edited image from which the objects of the first group and the other object selected by the third user input are removed. Electronic device.

7. In claims 1 to 6, when the instructions are executed individually or collectively by the at least one processor, the electronic device: A first map is obtained that includes pixels corresponding to each region of the source image, and each of the pixels of the first map has a value representing the size of an object located in a corresponding region among the plurality of objects; A second map is obtained by applying a two-dimensional Gaussian filter having a peak value at the reference point of the source image to the first map; and Using the second map above, causing the objects to be classified into either the first group or the second group, Electronic device.

8. In claims 1 to 7, when the instructions are executed individually or collectively by the at least one processor, the electronic device: Causing to display the edited image as a substitute for the source image through the above display, Electronic device.

9. A method of an electronic device including a display, The operation of displaying a source image through the above display; An operation to identify multiple objects within the above source image; An operation to display a UI object for removing at least one of the plurality of objects based on identifying the plurality of objects within the source image; Based on receiving a first user input related to the above UI object, an action of visually highlighting an object of a first group among the plurality of objects with respect to an object of a second group, wherein the distance between the reference point of the source image and the object within the second group is smaller than the distance between the reference point of the source image and the object within the first group; and Based on receiving a second user input for generating an edited image while the objects of the first group are visually highlighted, the operation of obtaining an edited image in which the objects of the first group are removed according to inpainting and the objects of the second group are retained. method.

10. In Claim 9, Based on receiving the first user input, the method further includes the operation of displaying a second UI object for receiving the second user input superimposed on the source image. method.

11. In claim 10, the operation of displaying the second UI object is, A third UI object for moving the object of the first group within the source image is superimposed on the source image, comprising the operation of displaying the third UI object on the source image. method.

12. In Claim 11, An operation to receive a fourth user input for changing the position of an object of the first group based on receiving a third user input related to the third UI object; and Based on receiving the fourth user input, the operation of acquiring an edited image including the first group of objects located at the changed location within the source image further comprises: method.

13. In claims 9 to 12, the operation to be obtained is, The method comprises the operation of obtaining the edited image in which the remaining objects among the objects of the first group are removed, excluding the object selected by the third user input, based on receiving a third user input for selecting one of the objects of the first group while visually highlighting the objects of the first group with respect to the objects of the second group, method.

14. In claims 9 to 13, the operation to be obtained is, The method comprises the operation of obtaining an edited image in which the object of the first group and the other object selected by the third user input are removed, based on receiving a third user input for selecting another object different from the object of the first group within the source image while visually highlighting the object of the first group with respect to the object of the second group, method.

15. In claims 9 to 14, the operation of visually highlighting and displaying the objects of the first group is, The operation of obtaining a first map including pixels corresponding to regions of the source image, wherein each of the pixels of the first map has a value representing the size of an object located in a corresponding region among the plurality of objects; The operation of obtaining a second map by applying a two-dimensional Gaussian filter having a peak value at the reference point of the source image to the first map; and A method comprising classifying the objects into either the first group or the second group using the second map. method.

Images