Electronic device and method for editing image

Abstract

According to one embodiment, an electronic device may comprise a display, at least one processor, and a memory storing instructions. The instructions, when executed individually or collectively by the at least one processor, may instruct the electronic device to: display a screen through the display; on the basis of an input for capturing the screen, obtain a first image corresponding to at least a portion of the screen; on the basis of an input for editing the first image, display, through the display, an image editing screen including a user interface (UI) field including a first UI area and a second UI area, which are overlaid on a representation of the first image and are used to crop a portion of the representation of the first image; and on the basis of an input for selecting the second UI area, obtain a second image in which a portion of the first image is removed from the first image.

Description

Electronic device and method for editing images

[0001] The present disclosure relates to an electronic device and method for editing an image.

[0002] Screen capture is a function that saves the content displayed on the screen of an electronic device, such as a smartphone, tablet, or computer, as an image file. To enhance user convenience, various functions need to be provided to edit the captured image into the image desired by the user.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art in relation to the present disclosure.

[0004] An electronic device is disclosed. The electronic device may include a display. The electronic device may include at least one processor including a processing circuit. The electronic device may include a memory including at least one storage medium for storing instructions. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the display to show a screen through the display. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the acquisition of a first image corresponding to at least a portion of the screen based on an input for capturing the screen. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the display to show an image editing screen through the display, the image editing screen comprising a user interface field that is superimposed on the representation to authorize viewing at least a portion of the representation of the first image based on an input for editing the first image, and is used to crop a portion of the representation of the first image. The UI field may include a first UI area superimposed on a first portion of the representation having first content identified based on an analysis of the first image. The UI field may include a second UI area superimposed on a second portion of the representation having second content identified based on the analysis of the first image, and displayed so as to be visually distinct from the first UI area.When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause to obtain a second image in which a first part of the first image having the second content is removed from the first image, based on an input for selecting the second UI area among the first UI area and the second UI area.

[0005] A method performed by an electronic device is disclosed. The method may include an operation of displaying a screen through a display. The method may include an operation of acquiring a first image corresponding to at least a portion of the screen based on an input for capturing the screen. The method may include an operation of displaying an image editing screen through the display, which includes a UI field superimposed on the representation to authorize viewing at least a portion of the representation of the first image based on an input for editing the first image, and used to crop the portion of the representation of the first image. The UI field may include a first UI area superimposed on a first portion of the representation having first content identified based on an analysis of the first image. The UI field may include a second UI area superimposed on a second portion of the representation having second content identified based on the analysis of the first image, and displayed so as to be visually distinct from the first UI area. The above method may include an operation of obtaining a second image in which a first part of the first image having the second content is removed from the first image, based on an input for selecting the second UI area among the first UI area and the second UI area.

[0006] FIG. 1 is a diagram illustrating a screen capture function of an electronic device according to one embodiment.

[0007] FIG. 2 is a flowchart illustrating an image editing function of an electronic device according to one embodiment.

[0008] FIG. 3 is a drawing for explaining an example of an image editing screen according to one embodiment.

[0009] FIG. 4 is a drawing for explaining a UI field according to one embodiment.

[0010] FIG. 5 is a diagram illustrating a screen displayed based on an input selecting a part of a UI area according to one embodiment.

[0011] FIG. 6 is a drawing illustrating a popup menu for selecting an image to be used for image editing, according to one embodiment.

[0012] FIGS. 7 and 8 are drawings for illustrating a UI field provided to remove a plurality of regions from a captured image according to one embodiment.

[0013] FIG. 9 is a drawing illustrating a popup menu for selecting conditions to be applied for image editing according to one embodiment.

[0014] FIGS. 10 and FIGS. 11 are drawings for illustrating an example of filling an empty space using a part of a captured image representation according to one embodiment.

[0015] FIGS. 12 and FIGS. 13 are drawings for illustrating an example of filling an empty space using a part of a captured image representation according to one embodiment.

[0016] FIG. 14 is a diagram illustrating a method for editing a captured image using an image stored in an electronic device according to one embodiment.

[0017] FIGS. 15 and 16 are drawings for explaining a method of editing a captured image using a webpage according to one embodiment.

[0018] FIG. 17 is a diagram illustrating a file generated from editing a captured image according to one embodiment.

[0019] FIG. 18 is a drawing for explaining text information about a captured image according to one embodiment.

[0020] FIG. 19 is a diagram illustrating the correlation between regions of a captured image according to one embodiment.

[0021] FIG. 20 is a drawing for illustrating modules for image editing according to one embodiment.

[0022] FIG. 21 is a drawing for explaining a UI displayed on the screen of a foldable device according to one embodiment.

[0023] FIG. 22 is a block diagram of an electronic device in a network environment according to one embodiment.

[0024] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0025] FIG. 1 is a diagram illustrating a screen capture function of an electronic device according to one embodiment.

[0026] Referring to FIG. 1, according to one embodiment, an electronic device (101) (e.g., a smartphone, tablet, computer, or foldable device) can display at least one content on a screen (110). For example, the electronic device (101) can display a video, text, and / or an image on the screen (110) individually or simultaneously. For example, the electronic device (101) can display a video, text describing the video, and an advertisement image on the screen (110) simultaneously.

[0027] The electronic device (101) can capture at least a portion of the content (112) displayed on the screen (110) and obtain a captured image (114) based on identifying user input for screen capture. For example, the electronic device (101) can capture the remaining content among the content (112) displayed on the screen (110), excluding the status bar (112-1) at the top of the screen and the navigation bar (112-2) at the bottom of the screen.

[0028] A captured image (114) may include a plurality of regions (114-1, 114-2, 114-3). The plurality of regions (114-1, 114-2, 114-3) may be distinguished from one another based on a description of the content (or context of the content). For example, the captured image (114) may include a first region (114-1) having first content (e.g., a frame of video or an image), a second region (114-2) having second content (e.g., text describing the video), and a third region (114-3) having third content (e.g., an advertisement image). It should be noted that FIG. 1 is merely an example for explaining the technical concept of the present disclosure and the scope of the present disclosure is not limited thereto. For example, the captured image (114) may include two regions or four or more regions.

[0029] The electronic device (101) can analyze (or recognize) the captured image (114) to divide the captured image (114) into multiple regions (114-1, 114-2, 114-3). An AI model may be used for the analysis of the captured image (114). The analysis of the captured image (114) will be described in detail with reference to FIGS. 18 and FIGS. 19.

[0030] The captured image (114) may contain content that the user does not want. For example, if the user captures content (112) displayed on the screen (110) to record a video, the captured image (114) may include an area (114-3) containing an advertisement image that is unrelated to (or has a relatively low degree of relevance to) the video.

[0031] According to one embodiment, the electronic device (101) can provide a function that allows the user to easily remove unwanted areas (or parts) from a captured image (114).

[0032] FIG. 2 is a flowchart illustrating an image editing function of an electronic device according to one embodiment.

[0033] Referring to FIG. 2, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may perform operations 210 to 250. Operations 210 to 250 may be performed sequentially, but are not limited thereto. For example, operations 210 and 220 may be performed in parallel.

[0034] In operation 210, the electronic device can acquire a captured image (e.g., the captured image (114) of FIG. 1). For example, the electronic device can store (or temporarily store) the captured image in the electronic device.

[0035] In operation 220, the electronic device may display an image editing screen (e.g., the image editing screen (310) of FIG. 3) on the display based on user input for editing the captured image. For example, user input for editing the captured image may be selecting (or tapping) a thumbnail of the captured image displayed in one area of ​​the display.

[0036] The image editing screen may include a representation of the captured image (e.g., the representation of the captured image (322) in FIG. 3) and / or a user interface field (e.g., the UI field (324) in FIG. 3) used to crop a portion of the representation of the captured image (322). In the present disclosure, the representation of the captured image may be an image that is identical or similar to the captured image displayed on the display for editing the captured image. For example, the representation of the captured image may have the same aspect ratio as the captured image, but a different size from the captured image, and may contain content identical to the content of the captured image. The UI field may be displayed so as to overlap at least a portion of the representation of the captured image. The image editing screen will be described in detail with reference to FIG. 3. In the following, for the convenience of explanation, the term ‘representation of a captured image’ may be referred to as ‘captured image representation’ as needed.

[0037] In operation 230, the electronic device may remove a portion of the representation of a captured image (e.g., an advertisement image) corresponding to the user input based on user input received through an image editing screen. For example, the user input may be selecting a UI area of ​​a UI field of the image editing screen.

[0038] In operation 240, the electronic device may fill an empty space (or whitespace) (e.g., the empty space (610) in FIGS. 6 and FIGS. 8 through 15) created by removing a portion of the representation of the captured image from said representation. To fill the empty space, the electronic device may use another area of ​​the captured image or an image different from the captured image (e.g., another image stored in the electronic device or an image from a website). The electronic device may provide a UI for selecting an image to be used to fill the empty space. In the present disclosure, the empty space (610) may refer to a graphic object that is displayed by removing a portion of the representation of the captured image. For example, the empty space (610) may be a solid color image that does not contain specific content or an image of a specific pattern (e.g., a diagonal pattern).

[0039] In operation 250, the electronic device may display an edited image (e.g., the edited image of FIG. 12 (1200), the edited image of FIG. 13 (1300), the edited image of FIG. 14 (1400), the edited image of FIG. 16 (1600), or the edited image of FIG. 17 (1704)) through a display.

[0040] FIG. 3 is a drawing for explaining an example of an image editing screen according to one embodiment.

[0041] Referring to FIG. 3, according to one embodiment, an image editing screen (310) may include a representation (322) of a captured image (e.g., the captured image (114) of FIG. 1) and a UI field (324). The UI field (324) may be used to edit the captured image. In one example, the image editing screen (310) may further include one or more UI elements necessary for image editing. For example, the UI field (324) may further include a button for saving the edited image to a photo management app (e.g., a gallery) or a button for undoing the last editing action.

[0042] According to one embodiment, the representation (322) of a captured image may include a plurality of parts (or regions). For example, the representation (322) of a captured image may include a first part (332) having first content (e.g., a frame of video or an image), a second part (342) having second content (e.g., text describing the video), and a third part (352) having third content (e.g., an advertisement image). Although FIG. 3 illustrates the representation (322) of a captured image as including three parts (332, 342, 352), it should be noted that this is an example for illustrative purposes only and the scope of the disclosure is not limited thereto. For example, as described in FIG. 8, the representation of a captured image may include four or more parts and two or more parts having a relatively low degree of relevance.

[0043] The UI field (324) may be superimposed on at least a portion of the representation (322) of the captured image to allow the user to view at least a portion of the representation (322) of the captured image.

[0044] The UI field (324) may include UI areas (334, 344, 354) corresponding to parts (332, 342, 352) of the representation (322) of the captured image. The parts (332, 342, 352) of the representation (322) of the captured image may be visually distinguished from one another through the UI areas (334, 344, 354). For example, a UI area (354) corresponding to one or more parts (352) having a relatively low degree of relevance and one or more UI areas (334, 344) corresponding to one or more parts (332, 342) having a relatively high degree of relevance may be displayed so as to be visually distinguished from one another. The UI field (324) will be described in detail with reference to FIG. 4.

[0045] According to one embodiment, when the ratio (e.g., area ratio) of an area of ​​a captured image having a relatively low degree of relevance within a captured image (e.g., the third area (114-3) of FIG. 1) is relatively high (e.g., when the ratio is greater than a threshold ratio), the electronic device may display a part (352) of a captured image representation (322) corresponding to the area of ​​the captured image having a relatively low degree of relevance visually identical or similar to other parts (332, 342) of the captured image representation (322). For example, when a captured image (e.g., the captured image (114) of FIG. 1) and an area of ​​the captured image having a relatively low degree of relevance (e.g., the third area (114-3) of FIG. 1) are respectively represented by 10,000 pixels and 19,000 pixels, and when the threshold ratio is 0.5, the electronic device can represent parts (332, 342, 352) of the captured image representation (332) as being visually identical or similar to each other. This is because the higher the proportion of the area of ​​the captured image having a relatively low degree of relevance (e.g., an area containing an advertisement image) within the captured image, the higher the probability that the user intentionally captured the content (e.g., an advertisement image) included in the area of ​​the captured image.

[0046] In the present disclosure, the relevance of each part of the representation of a captured image (or parts of the captured image) may be a numerical value indicating the relationship between each part and other parts. For example, a relatively low relevance of a part of a specific captured image (or representation of a captured image) may mean that the relevance (e.g., score) of said part is lower than a threshold value. The relevance of each part of the representation of the captured image will be described in detail with reference to FIG. 19.

[0047] FIG. 4 is a drawing for explaining a UI field according to one embodiment. In FIG. 4, for convenience of explanation, the UI field (324) is described under the assumption that among the parts (332, 342, 352) of the representation (322) of the captured image, the third part (352) has a relatively low degree of relevance compared to the other parts (332, 342).

[0048] Referring to FIG. 4, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can display a UI field (324) through a display such that the UI field (324) is superimposed on a representation (322) of a captured image.

[0049] The UI field (324) may include a first UI area (334), a second UI area (344), and a third UI area (354) that respectively correspond to parts (332, 342, 352) of the captured image representation (322). The location of each of the UI areas (334, 344, 354) can be determined based on the location of each part (334, 344, or 354) of the capture image representation (322) corresponding to each UI area (334, 344, or 354). For example, the first UI area (334) may be displayed such that at least a portion of the first UI area (334) overlaps on the first part (332). For example, the second UI area (344) may be displayed such that at least a portion of the second UI area (344) overlaps on the second part (342). The third UI area (354) may be displayed such that at least a portion of the third UI area (354) overlaps on the third part (352).

[0050] According to one embodiment, each of the UI areas (334, 344, 354) may include a part (or area) for authorizing a user to view each part (332, 342, or 352) of a representation (322) of a captured image corresponding to each UI area (334, 344, or 354). For example, the first UI area (334) may include an outline corresponding to the shape (e.g., a rectangle) of the first part (332) and a transparent or translucent area inside the outline. For example, the second UI area (344) may include an outline corresponding to the shape of the second part (342) and a transparent or translucent area inside the outline. For example, the third UI area (354) may include an outline corresponding to the shape of the third part (352) and a transparent or translucent area inside the outline.

[0051] According to one embodiment, a third UI area (354) superimposed on a third part (352) of a capture image representation (322) having a relatively low degree of relevance may be displayed so as to be visually distinguished from a first UI area (334) or a second UI area (344) having a relatively high degree of relevance.

[0052] For example, the color of the outline of the third UI area (354) (e.g., red) may be different from the color of the outline of the first UI area (334) (e.g., blue) or the color of the outline of the second UI area (344) (e.g., blue).

[0053] For example, when the first UI area (334) and the second UI area (344) each include a transparent area, the third UI area (354) may include a translucent area.

[0054] For example, when each of the first UI area (334) and the second UI area (344) includes a translucent area, the third UI area (354) includes a translucent area, and the color of the translucent area of ​​the third UI area (354) (e.g., red) may be different from the color of the translucent area of ​​the first UI area (334) (e.g., blue) or the color of the translucent area of ​​the second UI area (344) (e.g., blue).

[0055] According to one embodiment, the electronic device may display a first UI element (412, 422, or 432) (e.g., a button) based on user input selecting a part (e.g., an area inside an outline) of each of the UI areas (334, 344, or 354). The first UI element (412, 422, or 432) may be a UI element provided to display text describing the content of each part (332, 342, or 352) of the captured image representation (322) corresponding to each UI area (334, 344, or 354). When the user selects the first UI element (412, 422, or 432), the text displayed on the display will be described in detail with reference to FIG. 5.

[0056] According to one embodiment, the electronic device may display a second UI element (442) (e.g., a button) based on user input selecting a portion of a third UI area (354) (e.g., an inner area of ​​an outline) that overlaps a third part (352) having a relatively low degree of relevance. For example, the electronic device may display the second UI element (442) together with the first UI element (432). The second UI element (442) may be a UI element provided to crop (or remove) the third part (352) having a relatively low degree of relevance from the representation (322) of the captured image. When the user selects the button (442), the screen displayed on the display will be described in detail with reference to FIG. 6.

[0057] According to one embodiment, the electronic device may display a second UI element (e.g., a second UI element (442)) for cropping (or removing) a portion (332 or 342) on a UI area (334 or 344) corresponding to a portion (332 or 342) having a relatively high degree of relevance.

[0058] For example, the electronic device may display a second UI element on the UI area (334 or 344) based on user input selecting a portion of the UI area (334 or 344) identified while the first UI element (412 or 422) is displayed.

[0059] For example, based on user input in which a portion of the identified UI area (334 or 344) is pressed for longer than a threshold time duration while the first UI element (412 or 422) is not displayed on the UI area (334, 344), the electronic device may display a second UI element on the UI area (334 or 344). The electronic device may display the second UI element together with the first UI element (412 or 422) or alone.

[0060] FIG. 5 is a diagram illustrating a screen displayed based on an input selecting a part of a UI area according to one embodiment.

[0061] Referring to FIG. 5, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may display text describing each part of a captured image representation (e.g., part (332, 342, or 352) of FIG. 4) corresponding to each UI area (e.g., UI area (334, 344, or 354) of FIG. 4) based on detecting an input (or event) selecting a first UI element (e.g., the first UI element (432) of FIG. 4) corresponding to a third part (e.g., the third part (352) of FIG. 4). For example, the electronic device may display text (510) describing the content of a third part based on an input selecting a first UI element (e.g., the first UI element (432) of FIG. 4) corresponding to a third part. The text of each part of the captured image representation may be displayed so as to be superimposed on each part and / or the UI area corresponding to each part.

[0062] According to one embodiment, the user can easily grasp the content of each part of the image representation in a short time through text describing each part of the image representation.

[0063] FIG. 6 is a drawing illustrating a popup menu for selecting an image to be used for image editing, according to one embodiment.

[0064] Referring to FIG. 6, an electronic device (e.g., the electronic device (101) of FIG. 1) may remove a portion of a captured image representation having a relatively low degree of relevance (e.g., the third portion (352) of FIG. 3 and 4) from a captured image representation based on an input selecting a second UI element (e.g., the second UI element (442) of FIG. 4) within a UI area (e.g., the third UI area (354) of FIG. 3 and 4) corresponding to said portion. For example, the electronic device may display said portion as an empty space (610).

[0065] According to one embodiment, the electronic device may display a first pop-up menu (620) in parallel with displaying the space (610) or after displaying the empty space (610). For example, the electronic device may display the first pop-up menu (620) such that the first pop-up menu (620) overlaps at least a portion of the empty space (610).

[0066] The first popup menu (620) may be for selecting an image to be used to fill the empty space (610). For example, the first popup menu (620) may include a plurality of options regarding the source of the image to be used to fill the empty space. For example, the first popup menu (620) may include a first option (622), a second option (624), and a third option (626). It should be noted that the options illustrated in FIG. 6 are merely examples to explain the technical concept of the present disclosure and the scope of the present disclosure is not limited thereto.

[0067] For example, the first option (622) may be to fill the empty space (610) based on a portion of the captured image representation adjacent to the portion removed from the captured image representation. The first option (622) will be described in detail with reference to FIGS. 8 through 13.

[0068] For example, the second option (624) may be to fill the empty space (610) using an image stored in an electronic device (e.g., an image stored in a photo management app). The second option (624) will be described in detail in FIG. 14.

[0069] For example, a third option (626) may be to fill the empty space (610) using at least a part of the website (e.g., an image included in the website). The third option (626) will be described in detail with reference to FIGS. 15 and FIGS. 16.

[0070] FIGS. 7 and 8 are drawings for illustrating a UI field provided to remove a plurality of regions from a captured image according to one embodiment.

[0071] Referring to FIGS. 7 and 8, according to one embodiment, a representation (722) of a captured image corresponding to a captured image may include a plurality of parts (732, 742) having a relatively low degree of relevance. The electronic device may provide a UI field (824) for removing said parts (732, 742) from the captured image representation (722). Since the attributes of the UI field (824) may be substantially the same as the attributes of the UI field (324) of FIGS. 3 and 4 described above, a redundant description is omitted.

[0072] The UI field (824) may include a first UI area (834) and a second UI area (844) that are respectively superimposed on parts (743, 742) of a capture image table (722) having a relatively low degree of relevance.

[0073] The electronic device may display UI elements (812, 822) (e.g., check boxes) for selecting a target to be deleted among a plurality of parts (714-1, 714-2) having relatively low relevance, based on an input selecting a part of the first UI area (834) or a part of the second UI area (844). For example, the electronic device may display UI elements (812, 822) based on detecting an event in which a user presses a part of the first UI area (834) (e.g., a translucent area inside an outline) or a part of the second UI area (844) for longer than a threshold time duration.

[0074] The electronic device may remove parts (732, 742) of a representation (722) of a captured image selected through UI elements (812, 822) based on input for removing parts (732, 742) of a representation (722) of a captured image selected through UI elements (812, 822). For example, the electronic device may remove parts (732, 742) from a representation (722) of a captured image based on detecting an event in which a user presses a button (e.g., a delete button) displayed on a first UI area (834) or a second UI area (844). The electronic device may display an empty space (610) created as parts (732, 742) are removed from a representation (722) of a captured image.

[0075] According to one embodiment, a user can easily remove multiple advertisement images included in a captured image from the captured image.

[0076] FIG. 9 is a drawing illustrating a popup menu for selecting conditions to be applied for image editing according to one embodiment.

[0077] Referring to FIG. 9, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may display a second pop-up menu (910) based on a first user input obtained through a first pop-up menu (e.g., the first pop-up menu (620) of FIG. 6). The first user input may be selecting an option to fill the empty space (610) using a portion (342) of a representation of a captured image adjacent to the empty space (610) among the options included in the first pop-up menu (e.g., options (622, 624, 626) of FIG. 6).

[0078] Based on identifying the first user input, the electronic device may display the portion (342) of the captured image representation adjacent to the empty space (610) and the portion (332) of the captured image representation not adjacent to the empty space (610) so that they are visually distinguishable from each other. For example, the electronic device may dim the portion (332) of the captured image representation not adjacent to the empty space (610). For example, the electronic device may blur the portion (332) of the captured image representation not adjacent to the empty space (610).

[0079] The electronic device can identify a second user input that selects a portion (342) of a captured image representation adjacent to an empty space (610). The electronic device can determine the portion (342) of the captured image representation selected by the second user input as an image to be used to fill the empty space (610). After the second user input is identified, the electronic device can display a second pop-up menu (910).

[0080] The second popup menu (910) may be for selecting a constraint related to the aspect ratio of the image to be applied during the process of filling the empty space (610). For example, the constraint related to the aspect ratio may be applied independently of filling the empty space.

[0081] The second popup menu (910) may include one or more constraints. For example, the second popup menu (910) may include a first constraint (912) and a second constraint (914).

[0082] For example, the first constraint (912) may be to fill the empty space based on maintaining the size ratio of the representation of the captured image. The first constraint (912) will be described in detail with reference to FIG. 12.

[0083] For example, the second constraint (914) may be to fill the empty space based on maintaining the size ratio of the portion of the captured image representation selected to fill the empty space. The second constraint (914) will be described in detail with reference to FIG. 13.

[0084] FIGS. 10 and FIGS. 11 are drawings for illustrating an example of filling an empty space using a part of a captured image representation according to one embodiment.

[0085] Referring to FIG. 10, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may remove a portion from a representation of a captured image and display an empty space (610) in place of the removed portion. After displaying the empty space (610) or in parallel with displaying the empty space, the electronic device may display a handle (1011, 1021, or 1031) (e.g., a resize handle) for each of one or more portions (1012, 1022, 1032) of the captured image representation adjacent to the empty space (610). Based on user input identified through the handle (1011, 1021, or 1032), the electronic device may fill the empty space (610) using a first portion (1022), a second portion (1032), or a third portion (1032).

[0086] As described above, the electronic device can display one or more parts (1012, 1022, 1032) of a captured image representation adjacent to the empty space (610) and one or more parts (1042, 1052, 1062, 1072, 1082) of a captured image representation not adjacent to the empty space (610) so as to be visually distinguishable from each other. Redundant descriptions are omitted.

[0087] Referring to FIG. 11, according to one embodiment, when a blank space (610) is filled using a part (1012, 1022, or 1032) of a captured image representation adjacent to the blank space (610), one or more parts of the captured image representation adjacent to the blank space (610) may be removed. For example, when a blank space (610) is filled using a part (1032) of a captured image representation diagonally adjacent to the blank space (610), at least a portion of one or more parts (1012, 1022) of the captured image representation adjacent to both the blank space (610) and the selected part (1032) of the captured image representation may be removed.

[0088] FIGS. 12 and FIGS. 13 are drawings for illustrating an example of filling an empty space using a part of a captured image representation according to one embodiment.

[0089] Referring to FIG. 12, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may maintain the size ratio of the portion (342) of the captured image representation selected to fill the empty space (610), independently of filling the empty space (610) using a portion (342) of the captured image representation (representation (322) of FIG. 3 and FIG. 4). For example, when the size ratio of the captured image representation is 1 (horizontal) : 3 (vertical), the size ratio of the edited image (1200) may be 1:3.

[0090] According to one embodiment, upscaling can be applied to fill the empty space (610) using a portion (342) of the representation of the selected captured image.

[0091] FIG. 13 may be illustrated on the premise that among parts (1332, 1342, 1352) of a captured image representation, a first part (1342) is selected to fill the empty space (610). Referring to FIG. 13, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may maintain the size ratio of the first part (1342) independently of using the first part (1342) to fill the empty space (610). For example, when the size ratio of the first part (1342) is 2 (horizontal) : 1 (vertical), the size ratio of the part (1310) of the edited image (1300) corresponding to the first part (1342) may be 2 : 1.

[0092] FIG. 14 is a diagram illustrating a method for editing a captured image using an image stored in an electronic device according to one embodiment.

[0093] Referring to FIG. 14, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can identify user input selecting an option (624) to fill a blank space (610) using an image stored in the electronic device among the options (622, 624, 626) of a first pop-up menu (620). For example, a captured image representation (1422) may include a portion (1432) providing information about a restaurant's meal menu and a portion of an advertisement image that is not related to (or has low relevance to) said meal menu. Based on the user input, the electronic device may select, among the images stored in memory (e.g., images stored in a photo management app), an image having the same or similar attributes to the attributes of the portion (1432) of the captured image representation (1422) that is not to be removed, as the image to be used to fill the blank space (610). For example, the electronic device can search for an image to be used to fill the empty space (610) based on the location where the image was captured (e.g., GPS coordinates), the time the image was captured (image capture time), and / or text related to the content of the image (or the context of the image).

[0094] The electronic device may display a popup menu (1440) when an image having attributes similar to those of a part (1432) of a captured image representation (1422) that is not to be removed (e.g., an image providing information about a restaurant's beverage menu) is found. The popup menu (1440) may include a thumbnail (1442) of the found image and text confirming whether to combine the found image with the part (1432) that is not to be removed.

[0095] The electronic device can obtain an edited image (1400) based on user input that accepts the merging between the retrieved image and a portion (1432) of the captured image representation (1422) that is not to be removed.

[0096] According to one embodiment, a user can easily edit the content into a single image after capturing content that is difficult to capture simultaneously into multiple separate images. For example, if a user wants to create an image containing a restaurant's menu and address, the user can capture a webpage providing the restaurant's menu information and a webpage providing the restaurant's address information, and then easily create an edited image containing the restaurant's address and menu through the method described above.

[0097] FIGS. 15 and 16 are drawings for explaining a method of editing a captured image using a webpage according to one embodiment.

[0098] Referring to FIGS. 15 and 16, according to one embodiment, a captured image representation (1522) may correspond to a captured image that is captured while at least a portion of a webpage is displayed on a screen (e.g., screen (110) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1). For example, the captured image representation (1522) may include a first portion (1532) containing text information about a specific topic (e.g., a neural network), a second portion (1542) containing a portion of an image associated with said topic, and a third portion (1552) containing an advertisement image superimposed on at least a portion of the image associated with said topic.

[0099] The user can provide user input to the electronic device to remove parts (1552) having a relatively low degree of relevance among parts of the captured image representation (1522).

[0100] The electronic device may use at least a portion of a webpage corresponding to the captured image representation (1522) to fill the empty space (610) created by removing a portion (1552) having a relatively low degree of relevance from the captured image representation (1522). The electronic device may search for the original image (e.g., original neural network image) of the webpage corresponding to the image of the second portion (1542) (e.g., neural network image) that is obscured by the image of the third portion (1552) (e.g., advertisement image) based on an input selecting an option (626) to fill the empty space (610) using at least a portion of the webpage corresponding to the captured image representation (1522) among the options (622, 624, 626) of the first popup menu (620).

[0101] When an original image (e.g., original neural network image) corresponding to an image (e.g., neural network image) that is partially obscured by a part (1552) of a captured image representation (1522) having a relatively low degree of relevance is found, the electronic device may display a popup menu (1640). The popup menu (1640) may include a thumbnail (1642) of the found original image and text to check whether to fill the empty space (610) using the found original image.

[0102] The electronic device can obtain an edited image (1600) based on filling the empty space (610) using at least a portion of the webpage corresponding to the captured image representation (2022).

[0103] According to one embodiment, when a user uses the screen capture function of an electronic device to record information included in a webpage, the user can easily remove unwanted information from the captured image through the method described above.

[0104] FIG. 17 is a diagram illustrating a file generated from editing a captured image according to one embodiment.

[0105] Referring to FIG. 17, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can generate at least one file based on removing one or more parts (1722, 1752) having a relatively low degree of relevance from a captured image representation (1722).

[0106] For example, the electronic device can generate a text file (1700) or an image file (1700) using the texts included in the captured image representation (1722).

[0107] For example, the electronic device can download the video included in the captured image (1722) and save it as a video file (1702).

[0108] For example, the electronic device may select an image file (1704) based on parts (1742, 1762, 1772) of a captured image representation (1722) that are not subject to removal.

[0109] FIG. 18 is a drawing for explaining text information about a captured image according to one embodiment.

[0110] Referring to FIG. 18, an electronic device (e.g., the electronic device (101) of FIG. 1) may generate text information (1820, 1812, 1814, or 1816) describing a captured image (114). For example, the electronic device may generate text information (1812, 1814, or 1816) for each of the contents included in the captured image (114) (e.g., one frame of the video, text describing the video, and an advertisement image) individually based on analyzing (or recognizing) the captured image (114). For example, the electronic device may generate text information (1812, 1814, or 1816) based on recognizing the object, color, scene, and / or action of the captured image (114).

[0111] For example, if the captured image (114) includes a first area (114-1) containing one frame (or image) of the video, a second area (114-2) containing text describing the video, and a third area (114-3) containing an advertisement image, the electronic device may obtain first text information (1812) corresponding to the first area (114-1), second text information (1814) corresponding to the second area (114-2), and third text information (1816) corresponding to the third area (114-3). In one example, the electronic device may also generate fourth text information (1820) corresponding to the entire captured image (114).

[0112] For example, the electronic device may store text information so that text information about the captured image (114) is included in the captured image (114) (e.g., image file). For example, the text information may be stored as a tag (e.g., metadata) of the captured image (114).

[0113] For example, the electronic device may store text information about the captured image (114) separately from the captured image (114). For example, the electronic device may store the text information as a separate file from the captured image (114) and on the electronic device and / or server. For example, the text information may be stored in the electronic device's database for managing the user's personal information (e.g., the user's schedule or photos) in an integrated manner.

[0114] According to one embodiment, the electronic device may use an AI module (e.g., the AI ​​module (2040) of FIG. 20) to generate text information for a captured image (114). For example, the AI ​​module may include cloud-based artificial intelligence or on-device AI.

[0115] FIG. 19 is a diagram illustrating the correlation between regions of a captured image according to one embodiment.

[0116] Referring to FIG. 19, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can determine the degree of relevance of each of the regions of a captured image (e.g., the regions (114-1, 114-2, 114-3) of FIG. 18) based on text information (1812, 1814, or 1816) for the captured image (e.g., the captured image (114) of FIG. 18).

[0117] For example, the electronic device can use an AI module to analyze text information (1812, 1814, or 1816) to determine the relevance of each of the regions of the captured image.

[0118] For example, the electronic device can determine the degree of correlation between regions of a captured image based on the similarity of words contained in text information (1812, 1814, 1816), with or without using an AI module. For example, the electronic device can determine that the degree of correlation between two text information is relatively high as the number of words common to both texts increases relative to the length of each of the two texts (e.g., the total number of words).

[0119] FIG. 20 is a drawing for illustrating modules for image editing according to one embodiment.

[0120] Referring to FIG. 20, according to one embodiment, an image editor (2020) may be an application or software module for image editing. The image editor (2020) may be stored in the memory of an electronic device (e.g., the electronic device (101) of FIG. 1).

[0121] The image editor (2020) may include an image analyzer (2022) and an image generator (2024). It should be noted that FIG. 20 is merely an example of a software module for carrying out embodiments of the present disclosure, and the scope of the present disclosure is not limited thereto.

[0122] The image analyzer (2022) may request the AI ​​module (artificial intelligence module) (2040) to analyze the captured image. For example, the image analyzer (2022) may request the AI ​​module (2040) to generate text information of the captured image (e.g., the first text information (1812), the second text information (1814), the third text information (1816), or the fourth text information (1820) of FIG. 13).

[0123] The image generator (2024) can generate UI fields using the AI ​​module (2040). For example, the image generator (2024) can generate UI fields based on text information of a captured image generated by the AI ​​module (2040). For example, text information corresponding to each of the regions of the captured image (e.g., the first region (114-1), the second region (114-2), and the third region (114-3) of FIG. 18) (e.g., the first text information (1812), the second text information (1814), or the third text information (1816) of FIG. 18 and FIG. 19)) may include location information regarding the relative position of each region to the captured image (e.g., the coordinates (or range) of each region expressed in pixels), and the image generator (2024) can generate UI fields based on the location information of each region.

[0124] For example, the image generator (2024) may request the AI ​​module (2040) to generate UI fields for the captured image. For example, the AI ​​module (2040) may generate UI fields based on text information of the captured image. The AI ​​module (2040) may be an on-device model or a cloud-based model.

[0125] The operations performed by the image generator (2024) may be substantially the same as the operations of the electronic device described above. Accordingly, redundant descriptions will be omitted.

[0126] FIG. 21 is a drawing for explaining a UI displayed on the screen of a foldable device according to one embodiment.

[0127] Referring to FIG. 21, according to one embodiment, when an electronic device (e.g., the electronic device (101) of FIG. 1) is a foldable device (2101) (e.g., a multi-foldable device), the electronic device may display respective thumbnails (2112, 2114, 2116) of one or more candidate images (e.g., other parts of the captured image representation, images stored in the gallery of the electronic device, or images of a webpage) to be used to fill the empty space (610) of the captured image representation. For example, the electronic device may display the thumbnails (2112, 2114, 2116) on a part of the display (e.g., a display area (2110)) separated by a fold line (e.g., a first fold line (2122) or a second fold line (2124)).

[0128] According to one embodiment, the electronic device may display a user element (2130) that provides guide information for image editing to the user. For example, the user element (2130) may be displayed superimposed on at least a portion of the empty space (610).

[0129] FIG. 22 is a block diagram of an electronic device in a network environment according to one embodiment.

[0130] Referring to FIG. 22, in a network environment (100), an electronic device (101) (e.g., the foldable device (2021) of FIG. 21) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0131] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0132] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0133] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0134] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0135] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0136] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0137] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0138] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0139] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0140] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0141] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0142] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0143] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0144] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0145] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0146] The communication module (190) can support the establishment of a direct (e.g., wired) communication link or a wireless communication link between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication link. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0147] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0148] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0149] According to one embodiment, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0150] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0151] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0152] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0153] An electronic device (101, 2101) is disclosed. The electronic device may include a display. The electronic device (101, 2101) may include at least one processor (120) including a processing circuit. The electronic device (101, 2101) may include a memory (130) including at least one storage medium for storing instructions. When the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may cause a screen (110) to be displayed through the display. When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may be caused to acquire a first image (114) corresponding to at least a part of the screen (110) based on an input for capturing the screen (110).When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may cause the display to show an image editing screen (310) that includes a UI field (user interface field) (324, 824) which is superimposed on the representation (322, 722, 1522, 1722) of the first image (114) to allow viewing at least a portion of the representation (322, 722, 1422, 1522, 1722) of the first image (114) based on an input for editing the first image (114). The above UI field (324, 824) may include a first UI area (344) superimposed on a first part (342) of the representation (322, 722, 1522, 1722) having first content identified based on the analysis of the first image (114). The above UI field (324, 824) may include a second UI area (354) superimposed on a second part (352) of the representation (322, 722, 1522, 1722) having second content identified based on the analysis of the first image (114), and displayed so as to be visually distinguishable from the first UI area (344).When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may cause the first part (114-3) of the first image (114) having the second content to be removed from the first image (114) based on an input for selecting the second UI area (354) among the first UI area (344) and the second UI area (354).

[0154] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may cause a popup menu (620) to display an image to be used to fill a blank space (610) created by removing a part (352) of the representation (322, 722, 1522, 1722) corresponding to the first part (114-3) of the first image (114), based on an input for selecting the second UI area (354). When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may cause the blank space (610) to be filled based on the selected image in order to acquire the second image.

[0155] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may cause the empty space (610) created by removing a part of the representation (322, 722, 1522, 1722) corresponding to the first part of the first image (114) in order to acquire the second image, to be filled using a part (342) of the representation (322, 722, 1522, 1722) having the first content.

[0156] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may cause to display a popup menu (910) for selecting an object in which the aspect ratio of the first image (114) and the part (342) of the expression (322, 722, 1522, 1722) having the first content must be maintained, based on an input to fill the empty space (610) using the part (342) of the expression (322, 722, 1522, 1722) having the first content, independently of filling the empty space (610).

[0157] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may cause the empty space (610) to be filled using the portion (342) of the representation (322, 722, 1522, 1722) having the first content, based on maintaining the size ratio of the first image (114) to acquire the second image.

[0158] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may cause the empty space (610) to be filled using the portion (342) of the representation (322, 722, 1522, 1722) having the first content, based on maintaining the size ratio of the second portion of the first image (114) to acquire the second image.

[0159] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may cause the empty space (610) created by removing a portion of the representation (322, 722, 1522, 1722) corresponding to the first portion of the first image (114) in order to acquire the second image, to be filled using an image stored in the memory (130) or at least a portion of a webpage corresponding to the first image (114).

[0160] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may be caused to acquire the second image based on filling the empty space (610) using the at least part of the webpage, in the case where the screen (110) includes an image superimposed on the second part of the webpage so that the first part of the webpage and the second part of the webpage are not visible.

[0161] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may be caused to obtain first text information (1814) describing the first content and second text information (1816) describing the second content based on the analysis of the first image (114).

[0162] In order to obtain the first text information (1814) and the second text information (1816), an analysis of the first image (114) can be performed using an AI model.

[0163] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may cause the second UI area (354) to be displayed so that the first UI area (344) and the second UI area (354) are visually distinguished from each other based on the first degree of relevance between the first text information (1814) and the second text information (1816).

[0164] The above UI fields (324, 824) may further include a third UI area (334) superimposed on a third part (332) of the representation (322, 722, 1522, 1722) having third content identified based on the analysis of the first image (114). The first degree of relevance may be lower than the second degree of relevance between the third text information (1812) describing the third content and the first text information (1814).

[0165] The first UI area (344) may include a first outline corresponding to the shape of the first part (342) of the representation (322, 722, 1522, 1722) and a transparent area inside the first outline. The second UI area (354) may include a second outline corresponding to the shape of the second part (352) of the representation (322, 722, 1522, 1722) and a translucent area inside the second outline.

[0166] When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101, 2101) may be caused to display text describing the second content based on identifying an input selecting a first part of the second UI area (354). Based on identifying an input selecting a second part of the second UI area (354), the electronic device may be caused to remove a part of the representation (322, 722, 1522, 1722) corresponding to the first part of the first image (114) from the representation (322, 722, 1522, 1722).

[0167] When the electronic device (101, 2101) is a foldable device (2101) including a foldable display, and the first image (114) corresponds to at least a part of a screen (110) displayed on the foldable display, the instructions, when executed individually or collectively by the at least one processor (120), may cause the electronic device (101, 2101) to display a thumbnail of at least one candidate image to be used to fill a blank space (610) created by removing a part of the representation (322, 722, 1522, 1722) corresponding to the first part of the first image (114), through a first display area (2110) or a second display area (2110) separated from each other by a fold line (2124 or 2122) of the foldable display.

[0168] A method performed by an electronic device (101, 2101) is disclosed. The method may include an operation of displaying a screen (110) through a display. The method may include an operation of acquiring a first image (114) corresponding to at least a portion of the screen (110) based on an input for capturing the screen (110). The above method may include an operation of displaying an image editing screen (310) through the display, the UI field (324, 824) which is superimposed on the representation (322, 722, 1522, 1722) of the first image (114) to allow viewing of at least a portion of the representation (322, 722, 1522, 1722) of the first image (114) based on an input for editing the first image (114). The above UI field (324, 824) may include a first UI area (344) superimposed on a first part (342) of the representation (322, 722, 1522, 1722) having first content identified based on the analysis of the first image (114). The above UI field (324, 824) may include a second UI area (354) superimposed on a second part of the representation (322, 722, 1522, 1722) having second content identified based on the analysis of the first image (114), and displayed so as to be visually distinct from the first UI area (344). The above method may include an operation to obtain a second image in which a first part of the first image (114) having the second content is removed from the first image (114), based on an input for selecting the second UI area (354) among the first UI area (344) and the second UI area (354).

[0169] The operation of acquiring the second image may include an operation of displaying a popup menu (620) for selecting an image to be used to fill the empty space (610) created by removing a part (352) of the representation (322, 722, 1522, 1722) corresponding to the first part of the first image (114) based on an input for selecting the second UI area (354). The operation of acquiring the second image may include an operation of filling the empty space (610) based on the selected image in order to acquire the second image.

[0170] The operation of acquiring the second image may include, in order to acquire the second image, filling the empty space (610) created by removing a part of the expression (322, 722, 1522, 1722) corresponding to the first part of the first image (114) using a part (342) of the expression (322, 722, 1522, 1722) having the first content.

[0171] The above method may further include the operation of displaying a popup menu (910) for selecting an object whose aspect ratio must be maintained among the first image (114) and the part of the expression (322, 722, 1522, 1722) having the first content, based on an input for filling the empty space (610) using the part (342) of the expression (322, 722, 1522, 1722) having the first content, independently of filling the empty space (610).

[0172] The operation of filling the empty space (610) using a part (342) of the expression (322, 722, 1522, 1722) having the first content may include filling the empty space (610) using the part (342) of the expression (322, 722, 1522, 1722) having the first content, based on maintaining the size ratio of the first image (114) to obtain the second image.

[0173] For example, the above method may include an operation of filling the empty space using the portion of the expression having the first content, based on maintaining the size ratio of the portion of the expression having the first content in order to obtain the second image.

[0174] For example, the above method may include an operation of filling the empty space created by removing a part of the representation corresponding to the first part of the first image in order to obtain the second image, using an image stored in the memory or at least a part of a webpage corresponding to the first image.

[0175] For example, the above method may include an operation of acquiring the second image based on filling the empty space using at least a portion of the webpage, in the case where the screen includes an image superimposed on the second portion of the webpage so that the first portion of the webpage and the second portion of the webpage are not visible.

[0176] For example, the above method may include the operation of obtaining first text information describing the first content and second text information describing the second content based on the analysis of the first image.

[0177] For example, the above method may include an operation of filling the empty space created by removing a part of the representation corresponding to the first part of the first image in order to obtain the second image, using an image stored in the memory or at least a part of a webpage corresponding to the first image.

[0178] For example, the above method may include an operation of acquiring the second image based on filling the empty space using at least a portion of the webpage, in the case where the screen includes an image superimposed on the second portion of the webpage so that the first portion of the webpage and the second portion of the webpage are not visible.

[0179] For example, the above method may include the operation of obtaining first text information describing the first content and second text information describing the second content based on the analysis of the first image.

[0180] For example, in order to obtain the first text information and the second text information, an analysis of the first image may be performed using an AI model.

[0181] For example, the above method may include an operation of displaying the second UI area so that the first UI area and the second UI area are visually distinguished from each other based on a first degree of relevance between the first text information and the second text information.

[0182] For example, the UI field may further include a third UI area superimposed on a third part of the representation having third content identified based on the analysis of the first image.

[0183] For example, the first degree of relevance may be lower than the second degree of relevance between the third text information describing the third content and the first text information.

[0184] For example, in order to obtain the first text information and the second text information, an analysis of the first image may be performed using an AI model.

[0185] For example, the above method may include an operation of displaying the second UI area so that the first UI area and the second UI area are visually distinguished from each other based on a first degree of relevance between the first text information and the second text information.

[0186] For example, the UI field may further include a third UI area superimposed on a third part of the representation having third content identified based on the analysis of the first image.

[0187] For example, the first degree of relevance may be lower than the second degree of relevance between the third text information describing the third content and the first text information.

[0188] For example, the first UI area may include a first outline corresponding to the shape of the first part of the expression and a transparent area inside the first outline.

[0189] For example, the second UI area may include a second outline corresponding to the shape of the second part of the expression and a translucent area inside the second outline.

[0190] For example, the above method may include an operation of displaying text describing the second content based on identifying an input selecting a first part of the second UI area.

[0191] For example, the above method may include an operation to remove a part of the expression corresponding to a first part of the first image from the expression based on detecting an input selecting a second part of the second UI area.

[0192] The first UI area may include a first outline corresponding to the shape of the first part of the expression and a transparent area inside the first outline.

[0193] For example, the second UI area may include a second outline corresponding to the shape of the second part of the expression and a translucent area inside the second outline.

[0194] For example, the above method may include an operation of displaying text describing the second content based on identifying an input selecting a first part of the second UI area.

[0195] For example, the above method may include an operation to remove a part of the expression corresponding to a first part of the first image from the expression based on detecting an input selecting a second part of the second UI area.

[0196] For example, if the electronic device is a foldable device including a foldable display and the first image corresponds to at least a portion of a screen displayed on the foldable display, the method may include the operation of displaying a thumbnail of at least one candidate image to be used to fill an empty space created by removing a portion of the representation corresponding to the first portion of the first image, through a first display area or a second display area separated from each other by a fold line of the foldable display.

[0197] According to one embodiment, a non-transient computer-readable storage medium is provided for storing one or more programs. When the one or more programs are executed by at least one processor of an electronic device having a display, the instructions may include instructions that cause the electronic device to display a screen through the display when executed individually or collectively by the at least one processor.

[0198] For example, the above one or more programs may include instructions that cause the electronic device to acquire a first image corresponding to at least a portion of the screen based on an input for capturing the screen when executed by the at least one processor.

[0199] For example, based on an input for editing the first image, it may be superimposed on the representation to allow viewing at least a portion of the representation of the first image.

[0200] For example, the above one or more programs may include instructions that cause the electronic device to display an image editing screen through the display, the image editing screen including a UI field (user interface field) used to crop a portion of the representation of the first image when executed by the at least one processor.

[0201] For example, the UI field may include a first UI area superimposed on a first part of the representation having first content identified based on the analysis of the first image, and a second UI area superimposed on a second part of the representation having second content identified based on the analysis of the first image, and displayed so as to be visually distinguishable from the first UI area.

[0202] For example, the above one or more programs may include instructions that cause the electronic device to obtain a second image in which a first part of the first image having the second content is removed from the first image, based on an input for selecting the second UI area among the first UI area and the second UI area when executed by the at least one processor.

[0203] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0204] The electronic devices according to the various embodiments disclosed in this document may be of various forms. The electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or consumer electronics. The electronic devices according to the embodiments of this document are not limited to the devices described above.

[0205] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0206] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0207] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (120) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0208] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store's server, or a relay server's memory (130).

[0209] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, display; At least one processor including a processing circuit; and The memory includes at least one storage medium for storing instructions, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Displaying a screen through the above display; Based on the input for capturing the above screen, a first image corresponding to at least a part of the above screen is obtained; An image editing screen is displayed through the display, comprising a UI field (user interface field) that is superimposed on the representation to authorize viewing at least a portion of the representation of the first image based on an input for editing the first image, and is used to crop a portion of the representation of the first image, wherein the UI field is A first UI area superimposed on a first part of the representation having first content identified based on the analysis of the first image, and Includes a second UI area superimposed on a second portion of the representation having second content identified based on the analysis of the first image, and displayed so as to be visually distinguishable from the first UI area; and Based on an input for selecting the second UI area among the first UI area and the second UI area, causing to obtain a second image in which a first part of the first image having the second content is removed from the first image. Electronic device.

2. In paragraph 1, when the instructions are executed individually or collectively by the at least one processor, the electronic device, Based on the input for selecting the second UI area, a popup menu is displayed for selecting an image to be used to fill the empty space created by removing a part of the expression corresponding to the first part of the first image. To obtain the second image above, causing the empty space to be filled based on the selected image, Electronic device.

3. In paragraph 1, when the instructions are executed individually or collectively by the at least one processor, the electronic device, To obtain the second image, causing the empty space created by removing a part of the expression corresponding to the first part of the first image to be filled using a part of the expression having the first content. Electronic device.

4. In paragraph 3, when the instructions are executed individually or collectively by the at least one processor, the electronic device, Causing to display a popup menu for selecting an object whose aspect ratio between the first image and the part of the expression having the first content must be maintained, based on an input for filling the empty space using the part of the expression having the first content, independently of filling the empty space. Electronic device.

5. In paragraph 3, when the instructions are executed individually or collectively by the at least one processor, the electronic device, In order to obtain the second image, based on maintaining the size ratio of the first image, causing the empty space to be filled using the part of the representation having the first content, Electronic device.

6. In paragraph 3, when the instructions are executed individually or collectively by the at least one processor, the electronic device, In order to obtain the second image, based on maintaining the size ratio of the portion of the expression having the first content, causing the empty space to be filled using the portion of the expression having the first content, Electronic device.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, To obtain the second image, causing the empty space created by removing a part of the representation corresponding to the first part of the first image to be filled using an image stored in the memory or at least a part of the webpage corresponding to the first image. Electronic device.

8. In Paragraph 7, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, If the above screen includes an image superimposed on the second part of the webpage so that the first part of the webpage and the second part of the webpage are not visible, the second image is obtained based on filling the empty space using the at least part of the webpage. Electronic device.

9. In paragraph 1, when the instructions are executed individually or collectively by the at least one processor, the electronic device, Causing to obtain first text information describing the first content and second text information describing the second content based on the analysis of the first image above. Electronic device.

10. In claim 9, in order to obtain the first text information and the second text information, an analysis of the first image is performed using an AI model, Electronic device.

11. In paragraph 9, when the instructions are executed individually or collectively by the at least one processor, the electronic device, Causing the second UI area to be displayed so that the first UI area and the second UI area are visually distinguished from each other based on the first degree of relevance between the first text information and the second text information. Electronic device.

12. In paragraph 11, the UI field further comprises a third UI area superimposed on a third part of the representation having third content identified based on the analysis of the first image, and The first correlation mentioned above is lower than the second correlation between the third text information describing the third content and the first text information. Electronic device.

13. In Paragraph 11, The above-mentioned first UI area is, It includes a first contour corresponding to the shape of the first part of the above expression and a transparent area inside the first contour, The above second UI area is, A second contour line corresponding to the shape of the second part of the above expression and a translucent area inside the second contour line, Electronic device.

14. In Paragraph 13, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Based on identifying an input selecting a first part of the second UI area, text describing the second content is displayed, and Based on detecting an input selecting a second part of the second UI area, causing to remove a part of the expression corresponding to the first part of the first image from the expression. Electronic device.

15. In claim 1, wherein the electronic device is a foldable device comprising a foldable display, and the first image corresponds to at least a portion of a screen displayed on the foldable display, when the instructions are executed individually or collectively by the at least one processor, the electronic device, Causing to display, through a first display area or a second display area separated from each other by a fold line of the foldable display, a thumbnail of at least one candidate image to be used to fill the empty space created by removing a part of the representation corresponding to the first part of the first image. Electronic device.

Images