Electronic device, method, and non-transitory computer-readable storage medium for displaying virtual object of size determined by size of area identified on screen

Abstract

This electronic device may comprise a memory for storing instructions, a display, a camera, a communication circuit, and at least one processor. The instructions may instruct the electronic device to: execute application software to display, on the display, a user interface for a video call with an external electronic device; display, in the user interface, a second screen area corresponding to second video images; recognize a gesture from the second video images for a video call; on the basis of having identified a visual object representing a defined gesture via gesture recognition, identify a graphic emoji object representing the defined gesture; display, on the display, the graphic emoji object superimposed on the second screen area; with the graphic emoji object superimposed on the second video images in the second screen area, identify the change in the size of the visual object included in the second video images; and change the size of the graphic emoji object superimposed on the second video images in the second screen area.

Description

Electronic device, method, and non-transient computer-readable storage medium for displaying a virtual object of a size determined based on the size of an area identified through a screen

[0001] The present disclosure relates to an electronic device, a method, and a non-transient computer-readable storage medium for displaying a virtual object of a size determined based on the size of an area identified through a screen.

[0002] An electronic device can perform object detection and / or object recognition on an image. For example, the electronic device can determine whether a visual object is included within the image by performing object detection on the image. For example, the electronic device can identify the type of visual object included within the image by performing object detection on the image. For example, the object detection or object recognition can be performed through a trained artificial intelligence model.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure.

[0004] No claim or determination is made as to whether any of the foregoing can be applied as prior art related to the present disclosure.

[0005] An electronic device is described. The electronic device may include a memory comprising one or more storage media for storing instructions. The electronic device may include a communication circuit. The electronic device may include a display. The electronic device may include a camera. The electronic device may include at least one processor comprising a processing circuit. The instructions may cause the electronic device to execute application software to display, through the display, a user interface of application software for a video call with an external electronic device performed using the communication circuit, when executed individually or collectively by the at least one processor. The user interface of the application software may include a first screen area for displaying first video images provided from the external electronic device. The instructions may cause the electronic device to display, through the display, a second screen area corresponding to the second video images within the user interface, based on the second video images for the video call being acquired through the camera, when executed individually or collectively by the at least one processor. The above instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to perform gesture recognition on the second video images for the video call acquired through the camera while displaying the second video images within the second screen area of ​​the user interface.The above instructions may cause the electronic device to identify an emoji graphic object representing a definition gesture based on identifying a visual object representing a definition gesture according to the gesture recognition performed while displaying second video images within the second screen area within the user interface, when executed individually or collectively by the at least one processor. The visual object may correspond to a part of the body associated with the definition gesture. The above instructions may cause the electronic device to display the emoji graphic object superimposed on the second screen area through the display, when executed individually or collectively by the at least one processor. The above instructions may cause the electronic device to identify a change in the size of the visual object included in the second video images acquired through the camera while the emoji graphic object is displayed superimposed on the second video images within the second screen area, when executed individually or collectively by the at least one processor. The above instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to change the size of the emoji graphic object superimposed on the second video images within the second screen area based on identifying the change in the size of the visual object.

[0006] A method is provided. The method may be executed within an electronic device having a communication circuit, a display, and a camera. The method may include an operation of executing application software to display, through the display, a user interface of application software for a video call with an external electronic device performed using the communication circuit. The user interface of the application software may include a first screen area for displaying first video images provided from the external electronic device. The method may include an operation of displaying, through the display, a second screen area corresponding to the second video images within the user interface based on the second video images for the video call being acquired through the camera. The method may include an operation of performing gesture recognition on the second video images for the video call acquired through the camera while displaying the second video images within the second screen area within the user interface. The above method may include an operation of identifying an emoji graphic object representing a definition gesture based on identifying a visual object representing a definition gesture according to the gesture recognition performed while displaying second video images within the second screen area within the user interface. The visual object may correspond to a part of the body associated with the definition gesture. The above method may include an operation of displaying the emoji graphic object superimposed on the second screen area through the display. The above method may include an operation of identifying a change in the size of the visual object included in the second video images acquired through the camera while the emoji graphic object is displayed superimposed on the second video images within the second screen area.The above method may include an operation to change the size of the emoji graphic object superimposed on the second video images within the second screen area based on identifying the change in the size of the visual object.

[0007] A non-transient computer-readable storage medium is provided. The non-transient computer-readable storage medium may store one or more programs. The one or more programs may include instructions that cause the electronic device to execute the application software to display, through the display, a user interface of the application software for a video call with an external electronic device performed using the communication circuit, when executed by the electronic device having a communication circuit, a display, and a camera. The user interface of the application software may include a first screen area for displaying first video images provided from the external electronic device. The one or more programs may include instructions that cause the electronic device to display, through the display, a second screen area corresponding to the second video images within the user interface, based on the second video images for the video call being acquired through the camera, when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to perform gesture recognition on the second video images for the video call acquired through the camera while displaying the second video images within the second screen area within the user interface when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to identify an emoji graphic object representing the definition gesture based on identifying a visual object representing the definition gesture according to the gesture recognition performed while displaying the second video images within the second screen area within the user interface when executed by the electronic device. The visual object may correspond to a part of the body associated with the definition gesture.The above one or more programs may include instructions that cause the electronic device to display the emoji graphic object superimposed on the second screen area through the display when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to identify a change in the size of the visual object included in the second video images acquired through the camera while the emoji graphic object is displayed superimposed on the second video images within the second screen area when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to change the size of the emoji graphic object superimposed on the second video images within the second screen area when executed by the electronic device, based on identifying the change in the size of the visual object.

[0008] Figure 1 illustrates an example of an electronic device that displays video.

[0009] Figure 2 is a simplified block diagram of an exemplary electronic device.

[0010] Figure 3 is a flowchart illustrating the operation of an electronic device that displays a video combined with virtual objects.

[0011] FIGS. 4a and 4b illustrate an exemplary operation of an electronic device displaying a virtual object having a size determined based on the size of an area associated with a part of a user expressing a gesture.

[0012] FIGS. 5A and 5B illustrate an exemplary operation of an electronic device that displays a virtual object at a location determined based on the location of an area associated with a part of a user expressing a gesture.

[0013] FIG. 6 illustrates an exemplary operation of an electronic device for acquiring a video combined with virtual objects.

[0014] FIG. 7 illustrates an exemplary operation of an electronic device that identifies a gesture using a Region of Interest (ROI).

[0015] FIG. 8 illustrates an exemplary operation of an electronic device that combines virtual objects using a background area and a foreground area.

[0016] FIG. 9 illustrates an exemplary operation of an electronic device that determines the size of a virtual object based on distance.

[0017] FIGS. 10a and FIGS. 10b illustrate exemplary operation of an electronic device that determines the position of a virtual object based on different gestures.

[0018] FIG. 11 illustrates an exemplary operation of an electronic device that changes a virtual object based on an identified gesture.

[0019] FIGS. 12a and FIGS. 12b illustrate exemplary operation of an electronic device displaying a User Interface (UI) for displaying virtual objects.

[0020] FIGS. 13a and FIGS. 13b illustrate exemplary operation of an electronic device that combines virtual objects based on identifying facial expressions and gestures.

[0021] FIGS. 14a and FIGS. 14b illustrate exemplary operation of an electronic device that combines a single virtual object corresponding to gestures.

[0022] FIG. 15 illustrates an exemplary operation of an electronic device that displays virtual objects based on the number of visual objects identified within an image.

[0023] FIGS. 16a and FIGS. 16b illustrate exemplary operation of an electronic device displaying a visual indication representing a time interval for displaying a virtual object.

[0024] FIGS. 17a and FIGS. 17b illustrate exemplary operation of an electronic device that displays a visual indication of where a virtual object is to be displayed.

[0025] FIG. 18 illustrates an exemplary operation of an electronic device displaying a video in which a virtual object is synthesized.

[0026] FIG. 19 is a block diagram of an electronic device in a network environment according to various embodiments.

[0027] Figure 1 illustrates an example of an electronic device that displays video.

[0028] Referring to FIG. 1, an electronic device (100) may be used to display a video (150). For example, the video (150) may be acquired through a camera included in the electronic device (100) (e.g., camera (203) in FIG. 2). For example, the electronic device (100) may acquire a video (150) containing a visual object (160) through the camera. For example, the visual object may represent a part of the user of the electronic device (100) (e.g., a hand or a face). For example, the visual object may represent a gesture (or facial expression). For example, the electronic device (100) may display a virtual object (170) through a display (e.g., display (208) in FIG. 2) based on identifying the visual object (160) representing the gesture. For example, the electronic device (100) can display a video in which a virtual object (170) is combined (or synthesized) through the display, based on identifying a visual object (160) that expresses a specific gesture within the video (150).

[0029] The electronic device (100) may execute a software application for video calls. For example, when the electronic device (100) executes the software application for video calls, it may control a camera (e.g., the camera (203) of FIG. 2) to provide video calls. For example, when the electronic device (100) provides a function for video calls, it may acquire a video (150) combined with a virtual object (170) corresponding to a specific gesture based on identifying a visual object (160) that indicates (or expresses) a specific gesture within the video (150) acquired through the camera. For example, the electronic device (100) may transmit the video (150) combined with the virtual object (170) to another electronic device connected to the electronic device. For example, the user of the electronic device (100) may express rich emotions using a gesture expressed by a part of the user (e.g., face or hand). For example, a user of the electronic device (100) may enjoy a video call using the gesture. For example, the electronic device (100) may be required to display a virtual object (170) corresponding to a visual object (160) based on identifying a visual object (160) corresponding to a part of the user within the video (150). Although an embodiment for a video call has been described, the embodiment is not limited thereto. For example, when the electronic device (100) streams the video (150), it may stream the video (150) combined with the virtual object (170). For example, when the electronic device (100) streams the video (150), it may stream the video (150) combined with the virtual object (170) corresponding to the specific gesture based on identifying that a visual object (160) representing a part of the user of the electronic device (100) is a specific gesture.

[0030] State (140) may be described as a state in which a video (150) is displayed according to a software application for streaming. State (180) may be described as a state in which an image (or video) related to a video call is displayed according to a software application for a video call. For example, the electronic device (100) may display a user interface (185) for a video call through a display (e.g., the display (208) of FIG. 2) based on running a software application. For example, the user interface (185) may include a first preview screen (190) and a second preview screen (195). For example, the first preview screen (190) may be described as a screen for displaying an image (or video) received through an external electronic device where the video call is performed. For example, the first preview screen (190) may be referred to as a first screen area. For example, the second preview screen (195) may be described as a screen for displaying an image (or video) obtained through a camera of the electronic device (100). For example, the second preview screen (195) may be referred to as a second screen area. For example, the electronic device (100) may display the first preview screen (190) and the second preview screen (195) when displaying a user interface (185) for video calls. For example, the electronic device (100) may display image(s) obtained through a camera (e.g., camera (203) of FIG. 2) on at least one of the first preview screen (190) and the second preview screen (195).

[0031] The electronic device (100) may be required to display the size of the virtual object (170) differently. For example, the electronic device (100) may determine the size of the virtual object (170) based on the size of the area identified within the video (150) where a visual object (160) representing part of the user is identified. For example, the electronic device (100) may be required to combine the virtual object (170) at different locations within the video (150). For example, the electronic device (100) may determine the location within the video (150) to which the virtual object (170) is combined based on the location where the visual object (160) is identified. For example, the electronic device (100) may use the video (150) to obtain a video (150) in which the virtual object (170) corresponding to the gesture is combined based on identifying a gesture expressed by part of the user. For example, the electronic device (100) can determine the location and size of a virtual object (170) based on the location and size of the area of ​​the video (150) where the gesture is identified. For example, the electronic device (100) can combine the virtual object (170) having the determined size on the determined location of the video (150). For example, the electronic device (100) can display the video (150) with the virtual object (170) combined through a display (e.g., the display (208) of FIG. 2).

[0032] For example, the electronic device (100) may include hardware components used to perform or execute the above operations. The hardware components are described and illustrated with reference to FIG. 2.

[0033] Figure 2 is a simplified block diagram of an exemplary electronic device.

[0034] Referring to FIG. 2, the electronic device (100) may include at least one processor (207), memory (206), communication circuit (205), camera (203), and display (208).

[0035] At least one processor (207) may include a hardware component for processing data using instructions stored in memory (206). The hardware component for processing data may include a CPU (central processing unit) (e.g., including processing circuits). The hardware component for processing data may include a GPU (graphic processing unit) (e.g., including processing circuits). The hardware component for processing data may include a DPU (display processing unit) (e.g., including processing circuits). The hardware component for processing data may include a NPU (neural processing unit) (e.g., including processing circuits).

[0036] At least one processor (207) may include one or more cores. For example, at least one processor (207) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0037] Memory (206) may include a hardware component for storing data and / or instructions that are input to and / or output from at least one processor (207). Memory (206) may include, for example, volatile memory such as RAM (random-access memory) and / or non-volatile memory such as ROM (read-only memory). Volatile memory may include, for example, at least one of DRAM (dynamic RAM), SRAM (static RAM), cache RAM, and PSRAM (pseudo SRAM). Non-volatile memory may include, for example, at least one of PROM (programmable ROM), EPROM (erasable PROM), EEPROM (electrically erasable PROM), flash memory, hard disk, compact disk, and EMMC (embedded multimedia card).

[0038] The communication circuit (205) may include hardware components to support the transmission and / or reception of signals between the electronic device (100) and an external electronic device. The communication circuit (205) may include, for example, at least one of a modem, an antenna, and an O / E (optic / electronic) converter. The communication circuit (205) may support the transmission and / or reception of signals based on various types of protocols such as Ethernet, LAN (local area network), WAN (wide area network), WiFi (wireless fidelity), Bluetooth, BLE (Bluetooth low energy), Zigbee, LTE (long term evolution), and 5G NR (new radio).

[0039] The camera (203) may include one or more light sensors (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that generate an electrical signal indicating the color and / or brightness of light. For example, the camera (203) may be described as an image sensor. For example, the camera (203) may be available to acquire an image of the environment surrounding the electronic device (100).

[0040] The display (208) can output visualized information. For example, the display (208) can output visualized information to a user under the control of at least one processor (207). The display (208) may include hardware components of an electronic device (100) used to display a screen. For example, the display (208) may include light-emitting elements and circuits (e.g., transistors) that control the light-emitting elements to emit light. For example, each of the light-emitting elements may include an organic light-emitting diode (OLED) or a micro LED. However, it is not limited thereto. For example, the display (208) may include a liquid crystal display (LCD).

[0041] At least one processor (207) can acquire a video (e.g., video (605) of FIG. 6) through a camera (203). For example, at least one processor (207) can use the video to identify a part of a user (e.g., face or hand) that is making a gesture. For example, based on identifying the part of the user making the gesture, at least one processor (207) can determine the location and size of a virtual object (e.g., virtual object (417) of FIG. 4a) corresponding to the gesture by using the location and size of an area of ​​the video (e.g., area (415) of FIG. 4a) associated with the part of the user making the gesture. For example, at least one processor (207) can combine the virtual object of the determined size at the determined location within the video. For example, at least one processor (207) can display the video combined with the virtual object (e.g., the video (690) of FIG. 6) through the display (208). The communication circuit (205) can be used to stream images acquired through the camera (203) or to perform a video call with an external electronic device.

[0042] FIG. 3 is a flowchart illustrating the operation of an electronic device that displays a video combined with a virtual object. This method may be executed by the electronic device (100) illustrated in FIG. 2 or by at least one processor (207) of the electronic device (100). In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0043] Referring to FIG. 3, in operation 310, at least one processor (207) can acquire video (e.g., video (605) of FIG. 6) through a camera (203). For example, at least one processor (207) can acquire the video to perform a function for a video call. For example, at least one processor (207) can acquire the video to stream the video. For example, at least one processor (207) can acquire the video based on receiving user input to acquire the video through the camera (203).

[0044] In operation 320, at least one processor (207) can identify a part of a user (e.g., face or hand) expressing a gesture using a video (e.g., video (605) of FIG. 6). For example, the video may include a visual object corresponding to said part of the user (e.g., visual object (413) of FIG. 4a). For example, said visual object may occupy an area of ​​said video (e.g., area (415) of FIG. 4a). For example, at least one processor (207) can identify said visual object occupying said area within said video. For example, said visual object may express a gesture. For example, said user may express a gesture (or facial expression) using a part of the user (e.g., face or hand). For example, at least one processor (207) can identify said visual object expressing the gesture (e.g., visual object (413) of FIG. 4a) within said video.

[0045] In operation 330, at least one processor (207) can determine the location and size of a virtual object corresponding to the gesture (e.g., virtual object (417) of FIG. 4a) by using the location and size of an area (e.g., area (415) of FIG. 4a) of a video (e.g., video (605) of FIG. 6) associated with said part of the user expressing the gesture, based on identifying said part of the user. For example, at least one processor (207) can determine a virtual object corresponding to the gesture (e.g., virtual object (417) of FIG. 4a) to be combined with a video (e.g., video (605) of FIG. 6) among a plurality of virtual objects. For example, at least one processor (207) can determine the size of a virtual object corresponding to the visual object (e.g., virtual object (417) of FIG. 4a) by identifying the size of a visual object corresponding to said part of the user (e.g., visual object (413) of FIG. 4a). For example, at least one processor (207) may determine the location on the video where the virtual object is to be displayed, based on the location on the video (e.g., video (605) of FIG. 6) where the visual object is displayed. For example, the location on the video where the virtual object is to be combined may be mapped to the location on the video where a gesture corresponding to the virtual object is identified. For example, the location on the video where the virtual object is to be combined may be determined to a preset location (e.g., 100 pixels to the left, 100 pixels up) from the location on the video where a gesture corresponding to the virtual object is identified. For example, the virtual object may include at least one of an image, a video, and an animation. For example, the size of the virtual object based on the size of the visual object and the location of the virtual object based on the location of the visual object will be described later with reference to the description of FIG. 4a and FIG. 5a.

[0046] In operation 340, at least one processor (207) can combine a virtual object (417) of a determined size at a determined location within a video (e.g., video (605) of FIG. 6). For example, based on identifying a part of a user expressing a gesture through a visual object (413), at least one processor (207) can combine a virtual object (417) of a second size corresponding to a first size of an area (415) containing a visual object (413) at a second location corresponding to a first location of the area (415) within the video.

[0047] According to one embodiment, the operation of combining a virtual object (417) of a determined size at a determined location within a video (e.g., video (605) of FIG. 6) may include the operation of compositing or overlapping the virtual object (417). For example, at least one processor (207) may composite the virtual object (417) to at least a portion of a frame (e.g., the determined location) of the video (e.g., video (605) of FIG. 6). For example, at least one processor (207) may combine a file corresponding to the video (e.g., video (605) of FIG. 6) and the virtual object (417) (e.g., a file for the virtual object or an animation file). For example, at least one processor (207) can obtain a video (e.g., video (690) of FIG. 6) in which a virtual object (417) is superimposed by superimposing the virtual object (417) through a layer at a determined position of a frame of the video (e.g., video (605) of FIG. 6). For example, the operation of superimposing the virtual object (417) through a layer may be an example of the synthesis operation.

[0048] In operation 350, at least one processor (207) can display a video (e.g., the video (690) of FIG. 6) combined with a virtual object (417) through a display (208). For example, at least one processor (207) can obtain a video (e.g., the video (690) of FIG. 6) by combining or synthesizing a virtual object (417) with a video (e.g., the video (605) of FIG. 6) obtained through a camera (203). For example, at least one processor (207) can display the video through a display (208). For example, the electronic device (100) can display the video (690) combined with a virtual object (417) on an area of ​​the display (208) where a preview image (or preview video) is displayed. However, it is not limited thereto. For example, at least one processor (207) can transmit the video to another electronic device through a communication circuit (205). For example, at least one processor (207) can stream or transmit the video.

[0049] The electronic device (100) can stream or transmit a video combined with a virtual object (e.g., the video (690) of FIG. 6). For example, a user of the electronic device (100) can interact with other users using the video. For example, a user of the electronic device (100) can express emotions richly using the video. For example, a user of the electronic device (100) can enjoy a fun video created using the virtual object with other users.

[0050] The electronic device (100) can acquire or display a video (e.g., the video (690) of FIG. 6) in which a virtual object (417) is combined (or synthesized) using a video (e.g., the video (605) of FIG. 6) acquired through a camera (203). The operation of acquiring a video (e.g., the video (690) of FIG. 6) from the video (e.g., the video (605) of FIG. 6) will be described in more detail later with reference to FIG. 6.

[0051] FIGS. 4a and 4b illustrate an exemplary operation of an electronic device displaying a virtual object having a size determined based on the size of an area associated with a part of a user expressing a gesture.

[0052] Referring to FIG. 4a, according to one embodiment, the state (410) may be described as a state in which a virtual object (417) of a second size is displayed based on an area (415) of a first size. For example, at least one processor (207) may determine the size of the virtual object (417) based on the size of the area (415) containing a visual object (413) that expresses a gesture. For example, at least one processor (207) may determine the size of the virtual object (417) corresponding to the size of the area (415) by identifying the size of the area (415). At least one processor (207) may display a video that corresponds to a part of the user and includes a visual object (413) that expresses a gesture through a display (208). For example, at least one processor (207) may display a video (411) combined with a virtual object (417) through a display (208) based on identifying a visual object (413) expressing a gesture within the video. At least one processor (207) may identify the size of an area (415) where the visual object (413) is displayed. For example, at least one processor (207) may display a virtual object (417) having a second size corresponding to (or mapped to) the first size through a display (208) based on a determination that the size of the area (415) is a first size. For example, the size of the virtual object (417) may be proportional to the size of the area (415).

[0053] The state (420) can be described as a state in which a virtual object (427) of a fourth size is displayed based on an area (425) of a third size. For example, at least one processor (207) can identify the size of a visual object (423) that represents a gesture and part of the user. For example, at least one processor (207) can identify the size of an area (425) containing the visual object (423). For example, at least one processor (207) can display a virtual object (427) having a fourth size corresponding to the third size through the display (208) based on the determination that the size of the area (425) is a third size. For example, at least one processor (207) can display a video (421) combined with the virtual object (427) through the display (208) based on identifying the area (425).

[0054] According to one embodiment, at least one processor (207) can display a video (421) combined with a virtual object (427) having a fourth size larger than the second size of the virtual object (417) through a display (208), based on identifying a region (425) having a third size larger than the first size of the region (415). For example, since the distance between the electronic device (100) and part of the user in state (410) is greater than the distance between the electronic device (100) and part of the user in state (420), the size of the region (415) may be smaller than the size of the region (425). However, it is not limited thereto. For example, if the size of part of the user (e.g., hand) is different from the size of another part of another user (e.g., hand), the size of the region corresponding to said user may be different from the size of another region corresponding to said other user.

[0055] According to one embodiment, the electronic device (100) may further include at least one of an actuator (not shown) and a speaker (not shown). At least one processor (207) may provide a notification (e.g., at least one of audio and / or vibration notification) through the at least one when displaying a video (e.g., video (690) of FIG. 6) combined with a virtual object (417). For example, when the electronic device (100) displays the video through a display (208), it may output audio corresponding to the virtual object through the speaker. For example, when the electronic device (100) displays the video through a display (208), it may provide a vibration notification corresponding to the virtual object through the actuator.

[0056] According to one embodiment, at least one processor (207) may combine a virtual object corresponding to a facial expression into the video (411) based on identifying a part of a user (e.g., face) that displays a facial expression within the video (411). For example, an operation to display a video in which a virtual object is combined based on a gesture has been described above, but the embodiment is not limited thereto. For example, at least one processor (207) may identify a part of a user that displays a facial expression within a video (e.g., video (605) of FIG. 6) acquired through a camera (203). For example, at least one processor (207) may determine the location and size of a virtual object corresponding to the facial expression based on the location and size of the area in which the facial expression is identified. For example, at least one processor (207) may combine the virtual object corresponding to the facial expression into the video based on identifying the facial expression in the area. For example, at least one processor (207) may display a video (e.g., video (690) of FIG. 6) combined with the virtual object corresponding to the facial expression through a display (208). However, it is not limited thereto. For example, at least one processor (207) may combine a virtual object corresponding to at least one of the facial expressions and gestures into the video based on identifying a user who expresses at least one of the facial expressions and gestures within the video (605) acquired through the camera (203). For example, at least one processor (207) may display the video combined with the virtual object through a display (208).

[0057] States (410, 420) for streaming images (or videos) acquired through the camera (203) are illustrated in FIG. 4a, but embodiments of the present disclosure are not limited thereto. For example, states (430, 440) related to a user interface (185) for video calls are illustrated in FIG. 4b. For example, a description of the states (410, 420) may be referenced as a description of the states (430, 440).

[0058] At least one processor (207) can determine the location within the video where the virtual object (417) is to be combined (or composited) based on the location within the video where the region (415) is identified. For example, the determination of the location where the virtual object (417) is to be combined will be described further later with reference to FIG. 5a.

[0059] Referring to FIG. 4b, the state (430) can be described as a state in which a first preview screen (190) and a second preview screen (195) are displayed when a video call is performed. For example, at least one processor (207) may display an image (or video) obtained through a camera (203) on the second preview screen (195) when displaying a user interface (185) for a video call. For example, at least one processor (207) may perform gesture recognition on the image while displaying the image on the second preview screen (195). For example, at least one processor (207) may identify a visual object (413) within an area (435) on the second preview screen (195). For example, at least one processor (207) may identify a first visual object (e.g., visual object (413)) that represents a first defined gesture based on the gesture recognition. For example, at least one processor (207) can determine the size of a virtual object (417) represented by the first visual object based on identifying the size of the first visual object. For example, at least one processor (207) can display the virtual object (417) having the determined size through the display (208) superimposed on the second preview screen (195) based on identifying the first visual object representing the first definition gesture.

[0060] The state (440) can be described as a state in which the size of a virtual object (427) is changed based on the size of a first visual object representing a first definition gesture. For example, at least one processor (207) can identify a change in the size of a first visual object (e.g., visual object (423)) representing a first definition gesture. For example, at least one processor (207) can identify the visual object (423) within an area (445) of the second preview screen (195). For example, at least one processor (207) can change the size of the virtual object based on identifying the change. For example, at least one processor (207) can display a virtual object (427) larger than the virtual object (417) superimposed on the second preview screen (195) based on the size of the visual object (423) being larger than the size of the visual object (413).

[0061] At least one processor (207) may execute application software to display, through a display (208), a user interface (185) of application software for video calling with an external electronic device (not shown) performed using a communication circuit (205). For example, the application software may be referred to as a software application, software, or application. The user interface (185) of the application software may include an area capable of displaying a first preview screen (190) provided by the external electronic device. For example, the user interface (185) of the application software may include a first screen area (e.g., a first preview screen (190)) for displaying first video images provided by the external electronic device. For example, the first preview screen (190) may be referred to as a first screen area. At least one processor (207) may display a second preview screen (195) within a user interface (185) via a display (208) based on images for a video call being acquired through a camera (203). For example, the second preview screen (195) may be referred to as a second screen area. For example, at least one processor (207) may display a second screen area (e.g., a second preview screen (195)) corresponding to the second video images within the user interface (185) via a display (208) based on second video images for a video call being acquired through a camera (203). For example, the first video images may be described as images acquired from an external electronic device connected to perform a video call with the electronic device (100). For example, the second video images may be described as images acquired through the camera (203) when performing a video call with the external electronic device.

[0062] At least one processor (207) can perform gesture recognition on images for a video call acquired through a camera (203) while displaying a second preview screen (195) within a user interface (185). For example, at least one processor (207) can perform gesture recognition on second video images for a video call acquired through a camera (203) while displaying second video images within a second screen area (e.g., second preview screen (195)) within the user interface (185). At least one processor (207) can identify an emoji graphic object (e.g., virtual object (417)) representing a defined gesture based on identifying a visual object (e.g., visual object (413)) representing the defined gesture according to the gesture recognition performed while displaying the second preview screen (195) within the user interface (185). For example, at least one processor (207) may identify an emoji graphic object (e.g., virtual object (417)) representing a definition gesture (e.g., heart gesture) based on identifying a visual object (e.g., visual object (413)) representing said definition gesture according to gesture recognition performed while displaying second video images within a second screen area (e.g., second preview screen (195)) within the user interface (185). said visual object may correspond to a part of the body associated with said definition gesture. For example, said definition gesture may be described as a gesture that can be identified through gesture recognition. For example, the virtual object may be referred to as an emoji graphic object.

[0063] At least one processor (207) can display the emoji graphic object overlaid on the second preview screen (195) via the display (208). For example, at least one processor (207) can display the emoji graphic object overlaid on the second screen area (e.g., the second preview screen (195)) via the display (208).

[0064] At least one processor (207) can identify a change in the size of a visual object (413) included in images acquired through the camera (203) while an emoji graphic object (e.g., a virtual object (417)) is superimposed on a second preview screen (195) (e.g., a difference between the size of the visual object (413) and the size of the visual object (423)). For example, at least one processor (207) can identify a change in the size of a visual object (e.g., a visual object (413)) included in second video images acquired through the camera (203) while an emoji graphic object (e.g., a virtual object (417)) is superimposed on second video images within a second screen area (e.g., a second preview screen (195)). At least one processor (207) can change the size of an emoji graphic object (e.g., virtual object (427)) superimposed on a second preview screen (195) based on identifying a change in the size of a visual object (413). For example, at least one processor (207) can change the size of an emoji graphic object (e.g., virtual object (417)) superimposed on second video images within a second screen area (e.g., second preview screen (195)) based on identifying a change in the size of a visual object (e.g., visual object (413)).

[0065] According to one embodiment, at least one processor (207) may transmit a signal through a communication circuit (205) to an external electronic device performing a video call with the electronic device (100), which causes the emoji graphic object (e.g., virtual object (417)) to be superimposed on second video images acquired through a camera (203), based on displaying the emoji graphic object. For example, the external electronic device may display the emoji graphic object superimposed on the second video images based on receiving the signal. For example, the signal may cause a change in the size of the emoji graphic object according to the size of a visual object (e.g., visual object (413)) identified within the second video images.

[0066] FIGS. 5A and 5B illustrate an exemplary operation of an electronic device that displays a virtual object at a location determined based on the location of an area associated with a part of a user expressing a gesture.

[0067] Referring to FIG. 5a, the state (510) can be described as a state that displays a virtual object (417) combined on a second location corresponding to a first location of the region (415). For example, the first location can be described as a location of the region (415) within the video (511). For example, the second location can be described as a location where the virtual object (417) is combined within the video (511). For example, at least one processor (207) can represent a gesture and identify a region (415) containing a visual object (413) corresponding to a part of the user. For example, at least one processor (207) can determine or identify a location where the virtual object (417) corresponding to the gesture is combined based on identifying the location of the region (415). For example, at least one processor (207) may display a virtual object (417) on a second location corresponding to the first location based on identifying a region (415) on a first location of the video (511). For example, the second location may be mapped to the first location. For example, the second location may correspond to the first location. However, it is not limited thereto. For example, the second location may be the same as the first location.

[0068] According to one embodiment, the state (520) may be described as a state of displaying a virtual object (527) coupled on a fourth position corresponding to a third position of a region (525). For example, the third position may be described as a position of the region (525) within the video (521). For example, the fourth position may be described as a position where the virtual object (527) is coupled within the video (521). For example, at least one processor (207) may identify a part of a user expressing a gesture through the region (525) at the third position of the video. For example, at least one processor (207) may display the virtual object (527) coupled to the fourth position corresponding to the third position through the display (208) based on identifying the region (525) at the third position. For example, at least one processor (207) may display the video (521) where the virtual object (527) is coupled to the fourth position through the display (208). For example, the location where the virtual object (527) is combined may be determined according to the location of the area (525) where the gesture is identified. For example, the fourth location may be mapped to the third location. For example, the fourth location may correspond to the third location. However, it is not limited thereto. For example, the fourth location may be the same as the third location.

[0069] According to one embodiment, a virtual object (527) may be attached at a location (e.g., 100 pixels to the left and 50 pixels upward) away from an area (525) where the gesture is identified. For example, the location where the virtual object (527) is attached may be mapped to a location in an area (415) where the gesture is identified within the video (511). For example, at least one processor (207) may attach a virtual object (527) corresponding to the gesture at a location (e.g., 100 pixels to the left and 50 pixels upward) away from the area (415) based on identifying the gesture within the area (415).

[0070] States (510, 520) for streaming images (or videos) acquired through the camera (203) are illustrated in FIG. 5a, but embodiments of the present disclosure are not limited thereto. For example, states (530, 540) related to a user interface (185) for video calls are illustrated in FIG. 5b. For example, a description of the states (510, 520) may be referenced to a description of the states (530, 540).

[0071] Referring to FIG. 5b, states (530) and (540) may be described as states in which a virtual object (417) is displayed on a second preview screen (195) included in a user interface (185) for video calls. For example, at least one processor (207) may determine the location (or area) where the virtual object (417) is to be displayed based on the location (or area) of a visual object expressing a gesture on the second preview screen (195).

[0072] Because the position (or area) of a visual object (e.g., visual object (413)) representing a definition gesture in state (530) on the second preview screen (195) is different from the position of a visual object representing a definition gesture in state (540) on the second preview screen (195), the position of a virtual object (417) superimposed on the second preview screen (195) in state (530) may be different from the position of a virtual object (527) superimposed on the second preview screen (195) in state (540). For example, at least one processor (207) can identify the visual object (413) on the area (435) of the second preview screen (195). For example, at least one processor (207) may display a virtual object (527) superimposed on the second preview screen (195) based on the identification of the visual object within the area (525) of the second preview screen (195). At least one processor (207) may identify a change in the position of the visual object (413) while the emoji graphic object (e.g., virtual object (417)) is displayed superimposed on the second preview screen (195). For example, at least one processor (207) may change the position of the emoji graphic object (e.g., virtual object (527)) superimposed on the second preview screen (195) based on the identification of a change in the position of the visual object (413).

[0073] According to one embodiment, the second preview screen (195) may be superimposed on the first preview screen (190). The size of the second preview screen (195) may be smaller than the size of the first preview screen (190). However, the present disclosure is not limited thereto.

[0074] FIG. 6 illustrates an exemplary operation of an electronic device for acquiring a video combined with virtual objects.

[0075] Referring to FIG. 6, the electronic device (100) can acquire video (605) through a camera (203). For example, the electronic device (100) may include at least one of a Region Of Interest (ROI) recognition unit (610), a gesture recognition unit (620), an expression recognition unit (630), a foreground recognition unit (640), a user input recognition unit (650), a virtual object determination unit (660), a virtual object combination unit (670), and / or a virtual object storage unit (680). For example, the electronic device (100) can acquire a video (690) from a video (605) based on at least one of a Region Of Interest (ROI) recognition unit (610), a gesture recognition unit (620), a facial expression recognition unit (630), a foreground recognition unit (640), a user input recognition unit (650), a virtual object determination unit (660), a virtual object combination unit (670), and a virtual object storage unit (680).

[0076] The ROI recognition unit (610) can identify an ROI within the video (605). For example, the ROI may be described as an area that is presumed to contain a visual object (413) corresponding to a part of the user and expressing a gesture. For example, the ROI recognition unit (610) can identify an ROI within the video (605) by performing object detection (or object recognition) on the video (605). For example, the ROI recognition unit (610) can identify an ROI through a first model (615). For example, the first model (615) may be described as a model trained to acquire or identify an area presumed to contain a visual object (413) corresponding to a part of the user using an image or video. For example, the first model (615) may include a model trained through machine learning or deep learning techniques. For example, the ROI recognition unit (610) can provide at least one of the identified ROI and video (605) to the gesture recognition unit (620) and the facial expression recognition unit (630).

[0077] The gesture recognition unit (620) can identify a visual object (413) expressing a gesture within an ROI and an area (415) corresponding to the visual object (413). For example, the gesture recognition unit (620) can identify the location and size of the area (415). For example, the gesture recognition unit (620) can identify the location and size of the area (415) associated with a part of the user expressing a gesture within the ROI of a video (e.g., information of a frame of the video) (605) and a remaining area different from the ROI. For example, the gesture recognition unit (620) may include a second model (625). For example, the second model (625) may be described as a model trained to identify the location and size of the area (415) by performing object detection or object recognition within an image or video. For example, the gesture recognition unit (620) may provide information indicating the result of recognizing the gesture to the virtual object determination unit (660). The gesture recognition unit (620) may identify the location and size of an area (415) related to a part of the user expressing the gesture through the second model (625) by using at least one of the ROI information and video (605) obtained through the ROI recognition unit (610). The gesture recognition unit (620) may provide information indicating the result of the identification to the virtual object determination unit (660).

[0078] The facial expression recognition unit (630) can identify other visual objects expressing facial expressions within the ROI and other regions corresponding to said other visual objects. For example, the facial expression recognition unit (630) can identify the location and size of other regions. For example, the facial expression recognition unit (630) can identify the location and size of other regions related to a part of the user expressing facial expressions (e.g., face) within the ROI of the video (605) and other remaining regions different from the ROI. For example, the facial expression recognition unit (630) may include a third model (635). For example, the third model (635) may be described as a model trained to identify the location and size of other regions by performing object detection or object recognition within an image or video. For example, the facial expression recognition unit (630) may provide information indicating the result of recognition of a gesture to the virtual object determination unit (660). The facial expression recognition unit (630) can identify the location and size of an area (415) related to a part of a user expressing a facial expression, using a third model (635), by using at least one of the ROI information and video (605) obtained through the ROI recognition unit (610). The facial expression recognition unit (630) can provide information indicating the result of the identification to the virtual object determination unit (660).

[0079] According to one embodiment, the second model (625) and the third model (635) may each be described as an artificial intelligence model based on a Convolutional Neural Network (CNN). For example, the number of parameters in the input layer of each of the second model (625) and the third model (635) (e.g., when each pixel has R (Red), G (Green), and B (Blue) channels, the number of parameters in the input layer is three times the number of pixels in the input image) may be based on the number of channels of the pixels. When using high-dimensional images, the second model (625) and the third model (635) may each undergo dimensionality reduction (e.g., Principal Component Analysis (PCA)) to prevent overfitting. The second model (625) and the third model (635) may be different from each other, but are not limited thereto. For example, the second model (625) may be identical to the third model (635).

[0080] According to one embodiment, the second model (625) and the third model (635) may be included within the electronic device (100). However, they are not limited thereto. For example, at least one of the second model (625) and the third model (635) may be included within an external electronic device (e.g., a server). For example, the electronic device (100) may identify a gesture (or facial expression) through the external electronic device containing at least one of the second model (625) and the third model (635).

[0081] According to one embodiment, the second model (625) or the third model (635) may be used to identify a gesture (or facial expression) through a part of the user (e.g., hand or face). For example, the second model (625) or the third model (635) may generate information for identifying the gesture (or facial expression). For example, the information may indicate at least one of the type of the gesture (or facial expression), a confidence value (e.g., a confidence value for the recognition of the gesture or facial expression), and an area where the gesture (or facial expression) is identified.

[0082] The user input recognition unit (650) can identify user input for at least one of the executable objects included in the UI (User Interface) (e.g., executable objects of FIG. 12a (1201, 1202, 1203, 1204, 1205, 1206)). For example, based on identifying the user input, the user input recognition unit (650) can provide information indicated by the received user input to the virtual object determination unit (660).

[0083] The virtual object determination unit (660) can determine the type and location of the virtual object to be combined by the virtual object combining unit (670) based on at least one of the gestures and facial expressions identified within the video (605). For example, the virtual object determination unit (660) can determine at least one of the type, size, and location of the virtual object based on the gestures and / or facial expressions expressed within the region (415) identified through the ROI of the video (605). For example, the virtual object determination unit (660) can determine to combine a virtual object corresponding to at least one of the gestures and facial expressions expressed by the visual object among a plurality of virtual objects based on the result of object recognition performed on the visual object included within the region (415). For example, the virtual object determination unit (660) can determine the size of the virtual object based on the size of the region (415). For example, the virtual object determination unit (660) may determine the location of the virtual object based on the location of the region (415). However, it is not limited thereto. For example, the virtual object determination unit (660) may determine the type of virtual object combined with the video (605) based on user input for at least one of the executable objects (e.g., executable objects of FIG. 12a (1201, 1202, 1203, 1204, 1205, 1206)). For example, each of the virtual objects may correspond to each of the executable objects. For example, an executable object (e.g., executable object of FIG. 12a (1201)) may correspond to a first virtual object (e.g., a heart-shaped virtual object). For example, an executable object (e.g., the executable object (1202) of FIG. 12a) can correspond to a second virtual object (e.g., a V-shaped virtual object).

[0084] The foreground recognition unit (640) can identify a foreground area and a background area from the video (605). For example, the background area can be described as an area among the areas of the video (605) where a virtual object can be combined. For example, the background area can be referred to as a combined area. For example, the foreground area can be described as an area corresponding to the user within the video (605). For example, the foreground recognition unit (640) can identify a foreground area and a background area different from said foreground area from the video (605). For example, the foreground recognition unit (640) can identify a foreground area and an area remaining relative to said foreground area from the video (605). For example, said area remaining relative to said foreground area can be referred to as a background area. For example, the foreground recognition unit (640) can identify the foreground area and the background area using the fourth model (645). For example, the fourth model (645) may be described as an artificial intelligence model trained to identify or determine a region corresponding to a user using an image or video. For example, the fourth model (645) may include a model trained through machine learning or deep learning techniques. For example, the foreground region and / or background region may be obtained through human segmentation techniques. For example, the foreground recognition unit (640) may provide information representing at least one of the foreground region and background region to the virtual object combining unit (670). For example, the information may include a binary image or a 1-channel image for at least one of the foreground region and background region. For example, the binary image may be described as an image for converting a color image into a black and white image. For example, the 1-channel image may be described as an image that appears in grayscale.For example, each pixel of a 1-channel image can represent the confidence value of each pixel.

[0085] The virtual object combining unit (670) can combine or composite a virtual object (417) on a video (605). For example, the virtual object combining unit (670) can obtain a video (690) by combining a virtual object (417) of a determined size at a determined location of the video (605). For example, the virtual object combining unit (670) can obtain a video (690) by using information regarding at least one of the type, location, and size of the virtual object (417). For example, the virtual object combining unit (670) can obtain a video (690) in which the virtual object (417) is combined by using data regarding the virtual object (417) obtained from the virtual object storage (680). For example, the virtual object storage (680) can be described as a storage in which data (e.g., an asset) representing the virtual object (417) is stored. For example, the virtual object storage (680) may be included in the electronic device (100). However, the embodiments are not limited thereto. For example, the virtual object storage (680) may be located outside the electronic device (100). For example, an external electronic device (e.g., an external storage device or a cloud server) may include the virtual object storage (680). For example, the electronic device (100) may obtain a video (690) combined with a virtual object (417) by receiving data representing a virtual object (417) from the virtual object storage (680).

[0086] According to one embodiment, the virtual object combining unit (670) may acquire or generate a video (690) in which a virtual object (417) is combined on a foreground area. However, it is not limited thereto. For example, the virtual object combining unit (670) may acquire or generate a video (690) in which a virtual object is combined on a background area among a foreground area and a background area. For example, the background area may be referred to as a combining area. For example, some of the plurality of virtual objects may be combined on the foreground area and the background area. For example, some of the remaining portions of the plurality of virtual objects may be combined on the background area among the foreground area and the background area. For example, at least one processor (207) may refrain from, bypass, or block the combination of the remaining portions on the foreground area.

[0087] At least one processor (207) can perform object recognition on the ROI to reduce the time required to identify gestures and / or facial expressions. For example, object recognition on the ROI is described and illustrated in more detail with reference to FIG. 7.

[0088] FIG. 7 illustrates an exemplary operation of an electronic device that identifies a gesture using a Region of Interest (ROI).

[0089] Referring to FIG. 7, the state (710) can be described as a state in which the video (605) is displayed before the virtual object (417) is combined. For example, at least one processor (207) may display a video (711) containing a visual object (413) expressing a gesture through a display (208). For example, the video (711) may be an example of the video (605) of FIG. 6. For example, at least one processor (207) may use the video (711) to determine or identify an ROI to perform object recognition. For example, the ROI may be described as an area among the regions of the video (e.g., the video (605) of FIG. 6) to perform object recognition to identify a gesture and / or facial expression. For example, at least one processor (207) may determine or identify the ROI and the remaining area for said ROI before identifying the area (415) associated with the part of the user expressing the gesture. For example, the ROI may include a visual object (413) and an area (415) containing the visual object (413).

[0090] The state (720) can be described as a state in which an ROI is identified. For example, at least one processor (207) can determine the region (725) as an ROI. For example, at least one processor (207) can determine the region (725) as an ROI within the video (711) using the first model (615). For example, at least one processor (207) can reduce the time required to perform object recognition by performing object recognition on the region (725) among the region (725) and the remaining region different from the region (725). For example, at least one processor (207) can recognize or identify a gesture within the region (725) faster than performing object recognition on the region (725) and the remaining region by performing object recognition on the region (725) among the region (725) and the remaining region. For example, at least one processor (207) may refrain from, bypass, or block object recognition with respect to the remaining area. For example, at least one processor (207) may reduce the resources (or power) consumed to identify gestures by performing object recognition with respect to the area (725) among the area (725) and the remaining area. For example, at least one processor (207) may refrain from, bypass, or block object recognition to identify gestures within the remaining area.

[0091] At least one processor (207) can combine or composite a virtual object (417) in the background area among the foreground area and the background area. For example, the acquisition of a video in which a virtual object (417) is combined in the background area is described and illustrated in more detail with reference to FIG. 8.

[0092] FIG. 8 illustrates an exemplary operation of an electronic device that combines virtual objects using a background area and a foreground area.

[0093] Referring to FIG. 8, the state (810) can be described as a state in which an area (415) representing a gesture is identified. For example, at least one processor (207) can identify a visual object (413) representing a gesture within a video (811). For example, at least one processor (207) can identify an area (415) corresponding to a part of the user representing a gesture within the video (811). For example, the video (811) may be an example of a video (605). For example, based on identifying a first location of the area (415) and a first size of the area (415), at least one processor (207) can display a virtual object (417) of a second size associated with the first size on a second location associated with the first location through a display (208). For example, at least one processor (207) can identify or determine the size of the region (415) by performing at least one of object recognition and object segmentation on the video (811). For example, a virtual object (e.g., virtual object (417) of FIG. 4a) may be located on the foreground region (825) corresponding to the user. However, it is not limited thereto.

[0094] The state (820) can be described as a state in which a video (821) with a virtual object (827) combined on a background area is displayed. For example, at least one processor (207) can identify at least one of a foreground area (825) and a background area within the video (811) through the fourth model (645). For example, the foreground area (825) can be described as an area corresponding to the user within the video (605). For example, the foreground area (825) can be used to obtain a video (690) with a virtual object combined. For example, the background area can be referred to as a combination area where a virtual object can be combined. For example, at least one processor (207) can determine to combine the virtual object (827) with the background area among the foreground area (825) and a background area different from the foreground area (825). For example, at least one processor (207) may determine to combine the virtual object (827) corresponding to a gesture identified within the area (415) with the background area among the foreground area (825) and the background area, based on the determination that the virtual object (827) is a preset virtual object among a plurality of virtual objects. For example, some of the preset virtual objects may be set to be combined only with the background area. For example, some of the remaining virtual objects may be set to be combined with both the background area and the foreground area. For example, at least one processor (207) may display the virtual object (827) combined with the background area through the display (208) without combining the virtual object (827) with the foreground area (825). The electronic device (100) may naturally composite or combine the virtual object with the video (605) based on combining the virtual object (827) with the background area.

[0095] According to one embodiment, at least one processor (207) may determine whether to combine a virtual object corresponding to a gesture in a background area based on the type of gesture identified within an area of ​​video (605). For example, some of the plurality of virtual objects may be configured to be combined only in the background area. For example, some of the remaining of the plurality of virtual objects may be configured to be combined in both the background area and the foreground area. For example, at least one processor (207) may combine a first virtual object corresponding to the first gesture in the background area among the background area and the foreground area based on the determination that the gesture identified within an area of ​​video (605) is a first gesture. For example, the first virtual object may be described as a virtual object configured to be combined in the background area among the background area and the foreground area. For example, at least one processor (207) may display a video (690) in which the first virtual object is combined in the background area through a display (208). For example, at least one processor (207) may combine a second virtual object corresponding to the second gesture in a background area and a foreground area based on a determination that a gesture identified within an area of ​​the video (605) is the second gesture. For example, the second virtual object may be described as a virtual object configured to be combined in at least one of the background area and the foreground area. For example, at least one processor (207) may display the video (690) combined with the second virtual object through a display (208). For example, the second virtual object to be combined on the foreground area (825) and the first virtual object to be combined on the background area, respectively, may each be determined in advance. However, they are not limited thereto. For example, the first virtual object and the second virtual object, respectively, may each be determined based on user input.For example, at least one processor (207) can combine the second virtual object, which can be combined to a foreground area based on the user input, to the background area among the foreground area and the background area.

[0096] According to one embodiment, a virtual object can be combined according to various methods. A first method may be described as combining the virtual object to both a background area and a foreground area. A second method may be described as combining the virtual object to the background area among the background area and the foreground area, such that the virtual object is combined around the area where the user's face is identified. A third method may be described as combining the virtual object to the background area among the background area and the foreground area, such that the virtual object is combined around the area where the user's hand is identified.

[0097] According to one embodiment, at least one processor (207) can identify a first area in which a body is identified and a second area different from the first area through a fourth model (645) using images acquired through a camera (203). A second preview screen (195) may include the first area and the second area. At least one processor (207) can display an emoji graphic object through a display (208) superimposed on the second area among the first area and the second area.

[0098] At least one processor (207) can identify the distance between the electronic device (100) and a part of the user to determine the size of the virtual object (417). For example, at least one processor (207) can determine the size of the virtual object (417) corresponding to a gesture expressed by the part based on the distance. For example, the operation of determining the size of the virtual object (417) based on the distance is described and illustrated in more detail with reference to FIG. 9.

[0099] FIG. 9 illustrates an exemplary operation of an electronic device that determines the size of a virtual object based on distance.

[0100] Referring to FIG. 9, at least one processor (207) can identify the distance (910) between an electronic device (100) and a user (920) using a sensor (not shown). For example, the sensor can acquire or identify the distance (910) through a Time of Flight (ToF) technique. For example, the sensor may include a Light Detection And Ranging (LiDAR) sensor. However, it is not limited thereto.

[0101] At least one processor (207) can determine the size of a virtual object (417) corresponding to a gesture expressed by a part (925) based on the distance (910) between a part (925) of a user (920) obtained through the sensor. For example, at least one processor (207) can acquire or generate a video (690) combined with a virtual object (417) of a first size corresponding to the first distance based on the determination that the distance between the electronic device (100) and the part is a first distance. For example, at least one processor (207) can acquire or generate a video (690) combined with a virtual object (417) of a second size corresponding to the second distance based on the determination that the distance between the electronic device (100) and the part is a second distance. For example, the first distance may be longer than the second distance. For example, since the first distance is longer than the second distance, the first size may be smaller than the second size. For example, the distance (910) between the electronic device (100) and a part of the user (920) may be inversely proportional to the size of a virtual object corresponding to a gesture represented by said part. However, it is not limited thereto.

[0102] According to one embodiment, the electronic device (100) may include a VST (Video See-Through) device. For example, the electronic device (100) may further include a sensor (not shown) for obtaining a distance (910) between a part of the user and the electronic device (100). For example, the sensor may include a sensor for obtaining a depth value. For example, the electronic device (100) may determine the size of a virtual object corresponding to a gesture expressed through a part of the user using the depth value obtained through the sensor. For example, the size of the virtual object to be combined with the video (690) may be determined according to the depth value obtained through the sensor. For example, the electronic device (100) may obtain a video (690) combined with a virtual object of a first size corresponding to the first depth value (e.g., virtual object (427) of FIG. 4a) based on identifying a gesture having a first depth value through the sensor. For example, the electronic device (100) can acquire a video (690) combined with a virtual object of a second size smaller than the first size (e.g., the virtual object (417) of FIG. 4a) based on identifying a gesture having a second depth value greater than the first depth value through the sensor.

[0103] According to one embodiment, the electronic device (100) may further include a sensor for measuring distance. For example, when at least one processor (207) acquires images through a camera (203), it may identify the distance (910) between a part of the body expressing a gesture and the electronic device (100) through the sensor. For example, a part (925) of the user (920) may be referred to as a part of the body. For example, at least one processor (207) may determine the size of an emoji graphic object expressing an identified defined gesture based on performing gesture recognition on the images based on the identified distance. For example, at least one processor (207) may display an emoji graphic object having the determined size superimposed on a second preview screen (195) through a display (208).

[0104] According to one embodiment, at least one processor (207) can identify another gesture to determine the location where the virtual object (417) is to be displayed. For example, at least one processor (207) can acquire a video (690) in which the virtual object (417) is combined at the location indicated by the other gesture. The other gesture to determine the location of the virtual object (417) is described and illustrated in more detail with reference to FIG. 10a.

[0105] FIGS. 10a and FIGS. 10b illustrate exemplary operation of an electronic device that determines the position of a virtual object based on different gestures.

[0106] Referring to FIG. 10a, the state (1010) may be described as a state in which another gesture is identified within the area (1015) to determine the location where a virtual object (1027) is to be displayed. For example, at least one processor (207) may identify another gesture in the area (1015) of the video (605). For example, based on identifying the other gesture, at least one processor (207) may determine the location where the virtual object (1027) to be displayed is to be combined. For example, after identifying the other gesture, at least one processor (207) may identify the gesture corresponding to the virtual object (1027) within the area (415). However, it is not limited thereto. For example, at least one processor (207) can identify an area (415) where a gesture is identified, and then identify an area (1015) where another gesture (e.g., a pointing gesture) is identified to determine the location to which the gesture is combined.

[0107] The state (1020) may be described as a state of identifying a gesture within an area (415) containing a visual object (413). For example, at least one processor (207) may identify the visual object (413) representing the gesture in the area (415) after identifying the other gesture. For example, at least one processor (207) may determine the size of the virtual object (1027) to be displayed based on identifying the size of the area (415).

[0108] The state (1030) can be described as a state in which a virtual object (1027) is displayed on an area (1025) that appears by another gesture (e.g., a pointing gesture) identified in the area (1015). For example, at least one processor (207) can acquire a video (690) in which the virtual object (1027) is combined on the area (1025) that appears by the other gesture. For example, at least one processor (207) can display the video (690) in which the virtual object (1027) is combined on the area (1025) through a display (208).

[0109] According to one embodiment, at least one processor (207) may receive user input to change a virtual object corresponding to a gesture. For example, at least one processor (207) may change a gesture mapped to the virtual object based on the user input. For example, at least one processor (207) may change a gesture mapped to the virtual object from a first gesture to a second gesture based on receiving the user input. For example, after the change, at least one processor (207) may acquire a video (690) in which the virtual object is combined based on identifying the second gesture among the first gesture and the second gesture. However, it is not limited thereto. For example, at least one processor (207) may change a virtual object mapped to a gesture from a first virtual object to a second virtual object based on receiving another user input. For example, at least one processor (207) can obtain a video (690) in which a second virtual object different from the first virtual object is combined based on identifying the gesture after the change.

[0110] According to one embodiment, at least one processor (207) may receive user input for editing or creating a virtual object. For example, based on receiving user input for editing or creating a virtual object, at least one processor (207) may create or acquire a different virtual object that is different from the virtual objects stored in a virtual object repository (e.g., virtual object repository (680) of FIG. 6). For example, based on receiving the user input, at least one processor (207) may create the different virtual object through a trained model. For example, the trained model may be described as a model trained to generate an image using a prompt. For example, the trained model may include a Large Multimodal Model (LMM). For example, at least one processor (207) may provide or transmit the virtual object acquired through the trained model to another electronic device. For example, at least one processor (207) may provide or transmit the virtual object obtained through the trained model to another electronic device using a software application for sharing. For example, at least one processor (207) may obtain a video (690) combined with the virtual object obtained through the trained model based on identifying a gesture in an area within the video (605).

[0111] According to one embodiment, at least one processor (207) can acquire a video (690) combined with virtual objects based on the number of users identified within the video (605). For example, when acquiring an ROI, at least one processor (207) can determine the number of users included within the video (605). For example, at least one processor (207) can determine the number of users by performing object detection on the video (605). For example, if the number of one or more users is two or more, at least one processor (207) can acquire a video (690) combined with virtual objects corresponding to said gestures based on identifying a gesture expressed through a first part of the first user and a second part of the second user. The size of the virtual object corresponding to the gesture identified according to the first part of the first user and the second part of the second user may be larger than the size of another virtual object corresponding to a different gesture identified according to a single part of the user. However, it is not limited thereto.

[0112] States (1010, 1020, 1030) for streaming images (or videos) acquired through a camera (203) are illustrated in FIG. 10a, but embodiments of the present disclosure are not limited thereto. For example, states (1040, 1050, 1060) related to a user interface (185) for video calls are illustrated in FIG. 10b. For example, a description of states (1010, 1020, 1030) may be referenced as a description of states (1040, 1050, 1060).

[0113] According to one embodiment, at least one processor (207) may stop displaying a virtual object (417) based on the cessation of identifying a gesture. For example, when at least one processor (207) combines a first virtual object corresponding to a first gesture into a video (605) based on identifying a first gesture, it may combine a second virtual object corresponding to a second gesture into a video (605) based on identifying a second gesture. For example, the cessation of displaying the first virtual object and the display of the second virtual object will be described later with reference to FIG. 11.

[0114] Referring to FIG. 10b, states (1040), (1050), and (1060) can be described as states in which a lyrics object (1027) is superimposed and displayed on a second preview screen (195) within a user interface (185) for a video call. For example, the virtual object may be referred to as an emoji graphic object. For example, the first definition gesture may be described as a gesture for displaying the emoji graphic object. For example, the second definition gesture may be described as a gesture for indicating the location (or area) where the emoji graphic object will be displayed. For example, the first visual object may be described as a visual object representing the first definition gesture. For example, the second visual object may be described as a visual object representing the second definition gesture.

[0115] In state (1040), at least one processor (207) can identify a second visual object representing a second defined gesture to indicate an area where a virtual object is to be displayed, based on performing gesture recognition on images acquired through the camera (203). For example, at least one processor (207) can identify the second visual object based on gesture recognition performed while displaying images acquired through the camera (203) on a second preview screen (195). For example, the second visual object can be identified within an area (1045).

[0116] In state (1050), at least one processor (207) can identify a first visual object (e.g., visual object (413)) representing a first defined gesture based on performing gesture recognition on an image displayed on a second preview screen (195).

[0117] In state (1060), at least one processor (207) may superimpose a virtual object (1027) representing the first definition gesture on an area (or location) on the second preview screen (195) that appears by the second definition gesture, based on identifying the first visual object.

[0118] At least one processor (207) can identify a second visual object (e.g., visual object (1013)) representing a second definition gesture based on gesture recognition performed while displaying a second preview screen (195) within the user interface (185) before identifying a first visual object (e.g., visual object (413)) representing a first definition gesture. At least one processor (207) can display an emoji graphic object (e.g., virtual object (1027)) superimposed on an area (or area (1065)) of the second preview screen (195) that appears by the second definition gesture, via a display (208).

[0119] According to one embodiment, at least one processor (207) may store images acquired through a camera (203) in memory (206). For example, after storing the images, at least one processor (207) may identify candidate images acquired during a time interval (e.g., 5 seconds) from among the stored images, based on identifying a first visual object (e.g., visual object (413)) that represents a first defined gesture according to gesture recognition performed while displaying a second preview screen (195) in a user interface (185). For example, the candidate images may be described as images acquired during the time interval from among images acquired before the first visual object is identified. For example, the end time of the time interval may be described as the time when the first visual object is identified. For example, at least one processor (207) can identify a second definition gesture based on gesture recognition performed to identify a definition gesture indicating an area (e.g., area (1065)) where an emoji graphic object is to be displayed for candidate images. For example, a second visual object (e.g., visual object (1013)) can represent the second definition gesture. At least one processor (207) can display an emoji graphic object superimposed on an area (e.g., area (1065)) of a second preview screen (195) that is represented by the second definition gesture, through a display (208).

[0120] FIG. 11 illustrates an exemplary operation of an electronic device that changes a virtual object based on an identified gesture.

[0121] Referring to FIG. 11, the state (1110) can be described as a state in which a part of the user expressing a gesture is identified. For example, at least one processor (207) can display a video (1111) including a visual object (413) and an area (415) containing the visual object (413) through a display (208). For example, the video (1111) may be an example of a video (605). For example, at least one processor (207) can identify the visual object (413) expressing the gesture within the video (1111). For example, at least one processor (207) can identify the gesture in the area (415). For example, as at least one processor (207) identifies the location and size of the area (415), the location and size of the virtual object (417) to be combined can be determined.

[0122] The state (1120) can be described as a state in which a video (1121) is displayed in which a virtual object (417) of a determined size is combined at a determined location. For example, at least one processor (207) may combine a virtual object (417) corresponding to a gesture into the video (605) based on identifying a gesture within an area (415). For example, at least one processor (207) may display the video (1121) in which the virtual object (417) is combined at a determined location through a display (208). For example, the video (1121) may be an example of a video (690).

[0123] The state (1130) can be described as a state in which a video is displayed in which the combination of the virtual object (417) is interrupted. For example, when displaying the virtual object (417), at least one processor (207) can identify a different gesture (e.g., a hand heart gesture) and another gesture (e.g., a hand wave gesture) that is different from the gesture corresponding to the virtual object (417). For example, when displaying the virtual object (417), at least one processor (207) can identify the other gesture without identifying the gesture. For example, when displaying a video in which the virtual object (417) is combined, at least one processor (207) can identify the other gesture among the gesture and the other gesture. For example, at least one processor (207) may stop combining the virtual object (417) corresponding to the gesture with the video (605) based on a determination that the gesture is not identified within the video (1131). For example, at least one processor (207) may refrain from, block, or bypass combining the virtual object (417) based on a determination that the area (415) where the gesture is identified is not identified when displaying the virtual object (417).

[0124] For example, at least one processor (207) can identify an area (1135) within the video (1131) where another gesture (e.g., waving a hand) is identified. For example, at least one processor (207) can determine the location and size of a virtual object (1147) corresponding to the other gesture based on identifying the area (1135).

[0125] The state (1140) can be described as a state in which a virtual object (1147) is combined based on an area (1135) in which a different gesture is identified. For example, at least one processor (207) can combine a virtual object (1147) corresponding to a different gesture with a video (605) based on a different gesture identified within the area (1135). For example, at least one processor (207) can display a virtual object (1147) that is combined at a location determined based on the area (1135) and has a size determined based on the area (1135) through a display (208). For example, at least one processor (207) can acquire a video (1141) in which the virtual object (1147) is combined based on identifying a different gesture corresponding to the virtual object (1147) within the area (1135). For example, video (1141) may be an example of video (690).

[0126] According to one embodiment, at least one processor (207) may refrain from, bypass, or block the identification of other gestures while a virtual object (417) corresponding to a gesture is displayed. For example, at least one processor (207) may acquire or display a video (690) combined with a virtual object (417) for a preset time based on identifying a gesture within an area (415). For example, at least one processor (207) may acquire a video (690) combined with a virtual object (417) for a preset time based on identifying a part of the user expressing a gesture within a video (605). For example, at least one processor (207) may refrain from, bypass, or block the identification of other gestures different from the gesture within the video (605) for the preset time. However, it is not limited thereto. For example, at least one processor (207) may refrain from, bypass, or block the attachment of a virtual object (e.g., virtual object (1147)) corresponding to another gesture to the video (605) while the video (690) to which the virtual object (417) corresponding to the gesture is attached is being displayed, despite identifying another gesture within the video (605).

[0127] According to one embodiment, at least one processor (207) can perform gesture recognition on images acquired through a camera (203) while displaying a first emoji graphic object. When at least one processor (207) identifies a fourth visual object expressing a fourth definition gesture for displaying a second emoji graphic object, it may refrain from, bypass, or block displaying the second emoji graphic object on the second preview screen (195) during a time interval.

[0128] According to one embodiment, at least one processor (207) can acquire a video in which virtual objects corresponding to gestures are combined without interruption of identifying the gestures or without interruption of combining the virtual objects. For example, at least one processor (207) can sequentially identify a first gesture and a second gesture within a video (605). For example, based on the identification, at least one processor (207) can acquire a video (690) in which a first virtual object corresponding to the first gesture and a second virtual object corresponding to the second gesture are combined. For example, based on the identification of the first gesture, at least one processor (207) can identify the second gesture when the video in which the first virtual object corresponding to the first gesture is combined is displayed. For example, at least one processor (207) may display a second virtual object corresponding to the second gesture together with the first virtual object through a display (208) based on identifying the second gesture. For example, at least one processor (207) may acquire a video in which the first virtual object and the second virtual object are sequentially combined based on sequentially identifying the first gesture and the second gesture. For example, the first virtual object and the second virtual object may overlap. However, it is not limited thereto.

[0129] According to one embodiment, at least one processor (207) can identify at least one of a user's facial expression and a gesture within a video (605). For example, based on identifying a user's facial expression within the video (605), at least one processor (207) can display a video combined with a first virtual object corresponding to the facial expression through a display (208). For example, based on identifying a user's gesture within the video (605), at least one processor (207) can display a video combined with a second virtual object corresponding to the gesture through a display (208). For example, when at least one processor (207) simultaneously identifies a user's facial expression and a gesture within the video (605), it can combine either the first virtual object or the second virtual object with the video (605) based on the priority for the facial expression and the priority for the gesture, respectively. For example, when at least one processor (207) simultaneously identifies the facial expression and the gesture, it may display a video (690) combined with either the first virtual object or the second virtual object through a display (208). However, it is not limited thereto. For example, when at least one processor (207) simultaneously identifies the facial expression and the gesture, it may display a video (690) combined with the first virtual object and the second virtual object through a display (208). For example, the location and size of the first virtual object may be determined based on the location and size of the area where the facial expression is identified. For example, the location and size of the second virtual object may be determined based on the location and size of the area where the gesture is identified.

[0130] According to one embodiment, at least one processor (207) may refrain from, bypass, or block identifying a gesture corresponding to the virtual object (417) within the video (605) for a preset time interval after displaying a video (690) combined with a virtual object (417). For example, at least one processor (207) may identify a part of the user through a video acquired via a camera (203) after displaying a video (690) combined with a virtual object (417). For example, at least one processor (207) may identify a part of the user expressing a gesture corresponding to the virtual object (417) within the video. For example, at least one processor (207) may refrain from, bypass, or block the virtual object (417) corresponding to the gesture from being combined with the video (690) obtained through the camera (203) after displaying the video (690) to which the virtual object (417) is combined, even if the part of the user expressing the gesture is identified during a preset time interval.

[0131] For example, at least one processor (207) may display a UI object for displaying a virtual object (e.g., a UI object (1215) of FIG. 12a). At least one processor (207) may display executable objects for displaying a virtual object (e.g., executable objects (1201, 1202, 1203, 1204, 1205, 1206) of FIG. 12a) may display executable objects (e.g., executable objects (1201, 1202, 1203, 1204, 1205, 1206)) through a display (208). For example, the UI object and the executable objects are described and illustrated in more detail with reference to FIG. 12a and FIG. 12b.

[0132] FIGS. 12a and FIGS. 12b illustrate exemplary operation of an electronic device displaying a User Interface (UI) for displaying virtual objects.

[0133] Referring to FIG. 12a, the state (1210) can be described as a state in which a UI object (1215) for displaying a virtual object and executable objects (1201, 1202, 1203, 1204, 1205, 1206) are displayed. For example, at least one processor (207) may display a UI object (1215) for displaying a virtual object corresponding to a gesture identified within the video (605) via a display (208). For example, the UI object (1215) may be used to receive user input for controlling the virtual object. For example, at least one processor (207) may turn on or turn off the creation of the video (690) combined with the virtual object based on receiving user input for the UI object (1215).

[0134] For example, at least one processor (207) may acquire a video (690) combined with at least one virtual object corresponding to said at least one, based on receiving user input representing at least one of the executable objects (1201, 1202, 1203, 1204, 1205, 1206) while the creation of the video (690) is turned on. For example, each of the executable objects (1201, 1202, 1203, 1204, 1205, 1206) may correspond to each of the virtual objects (e.g., a heart-shaped virtual object, a V-sign-shaped virtual object, a thumbs-up-shaped virtual object, etc.). For example, at least one processor (207) may acquire a video (690) combined with a virtual object based on receiving user input for at least one of the executable objects (1201, 1202, 1203, 1204, 1205, 1206) or identifying a gesture (or facial expression) within the video (605).

[0135] The state (1220) may be described as a state in which an executable object (1222) for performing text recognition and an executable object (1224) for performing speech recognition are displayed. For example, at least one processor (207) may acquire or identify text through a display (208) based on receiving user input for an executable object (1222). For example, at least one processor (207) may display a virtual object corresponding to the text through the display (208) based on identifying the text. For example, at least one processor (207) may acquire a video (690) combined with a virtual object corresponding to the text based on identifying the text. However, it is not limited thereto.

[0136] For example, at least one processor (207) may acquire a user's voice based on receiving user input for an executable object (1224) for performing voice recognition. For example, the electronic device (100) may further include a microphone (e.g., the input module (1950) of FIG. 19). For example, at least one processor (207) may acquire a user's voice through the microphone based on receiving user input for an executable object (1224). For example, at least one processor (207) may determine a virtual object corresponding to the voice based on performing voice recognition on the acquired voice. For example, at least one processor (207) may determine at least one of the type, location, and size of the virtual object based on performing voice recognition on the voice. For example, at least one processor (207) may acquire a video (690) combined with a virtual object corresponding to the voice. For example, at least one processor (207) can display a video (690) combined with a virtual object corresponding to the voice, based on user input for an executable object (1224), through a display (208).

[0137] According to one embodiment, at least one processor (207) can create or search for virtual objects based on user input regarding an executable object (1226). For example, based on receiving user input regarding an executable object (1226), at least one processor (207) can create a virtual object to be displayed superimposed upon an image when a gesture is identified through the image, using a trained model. For example, the trained model may be described as a model for creating virtual objects based on text and / or audio. For example, the trained model may include a Large Multimodal Model (LMM), but is not limited thereto. For example, at least one processor (207) can use the trained model to search for a virtual object to be displayed among a plurality of virtual objects when a specific gesture is identified within the image. For example, at least one processor (207) can use the trained model to determine or identify a virtual object corresponding to the specific gesture among pre-generated (or pre-stored) virtual objects.

[0138] According to one embodiment, at least one processor (207) can acquire video (605) through a camera (203). At least one processor (207) can acquire metadata related to the request from a software application for an operating system. The metadata may be described as information provided by the system of the camera HAL (Hardware Abstraction Level) to a configuration requesting a preview image in the operating system. The metadata may be described as a set of values ​​having a specific key (or tag). For example, the metadata may be composed of a key-value structure. At least one processor (207) can use the metadata to identify whether user input is received or to identify a function corresponding to the received user input by identifying a value for a preset key.

[0139] Referring to FIG. 12b, the state (1230) can be described as a state in which a window (1235) for registering an icon for a newly created virtual object is displayed. For example, at least one processor (207) can create a new virtual object based on receiving user input for at least one of an executable object (1222) for text input and an executable object (1224) for voice input. For example, at least one processor (207) can display a virtual object (1237) through a display (208) based on obtaining text representing "splatter flower rain" through an executable object (1222). For example, at least one processor (207) can display a virtual object (1237) through a display (208) based on obtaining audio representing "splatter flower rain" through an executable object (1224). For example, at least one processor (207) can create a new virtual object (1237) based on at least one of the acquired text and the acquired audio.

[0140] According to one embodiment, at least one processor (207) may display, through a display (208), a window (1235) for adding an executable object (1208) (or icon) corresponding to the virtual object (1237) when displaying a new virtual object (1237). For example, at least one processor (207) may display a window (1235) for inquiring whether to add an executable object (1208) for displaying the virtual object (1237) into the UI.

[0141] State (1240) may be described as a state in which an executable object (1208) for displaying a virtual object (1237) is added to the UI. For example, at least one processor (207) may receive user input for creating or adding an executable object (or icon) for an executable object (1236) when displaying a window (1235). For example, at least one processor (207) may display a UI with an added executable object (1208) through a display (208) based on receiving user input for an executable object (1236). Although the operation of displaying a window (1235) to add or display an executable object (1208) has been described, embodiments of the present disclosure are not limited thereto. For example, at least one processor (207) can add an executable object (1208) into the UI based on displaying a virtual object (1237) without displaying a window (1235).

[0142] FIGS. 13a and FIGS. 13b illustrate exemplary operation of an electronic device that combines virtual objects based on identifying facial expressions and gestures.

[0143] Referring to FIG. 13a, the state (1310) can be described as a state of identifying facial expressions and gestures through a video (1311). For example, the video (1311) may be an example of a video (e.g., video (605) of FIG. 6). For example, at least one processor (207) can identify facial expressions (e.g., a smiling face) and gestures (e.g., waving a hand) within the video (1311).

[0144] The state (1320) can be described as a state that displays a video (1321) in which virtual objects (1322, 1324) are combined based on facial expressions and gestures identified in the state (1310). For example, the video (1321) may be an example of the video (690) of FIG. 6. For example, at least one processor (207) may combine a virtual object (1322) corresponding to an identified facial expression into the video (1311) based on identifying a facial expression within an area (1312) of the video (1311). For example, at least one processor (207) may display the video (1321) in which the virtual object (1322) is combined through a display (208). For example, at least one processor (207) can combine a virtual object (1324) corresponding to an identified gesture with the video (1311) based on identifying a gesture within an area (1314) of the video (1311). For example, at least one processor (207) can display the video (1321) with the virtual object (1324) combined through a display (208).

[0145] According to one embodiment, at least one processor (207) may combine at least one of a virtual object (1322) and a virtual object (1324) into a video (1311) based on identifying a facial expression within a region (1312) and a gesture within a region (1314). For example, at least one processor (207) may combine at least one of a virtual object (1322) and a virtual object (1324) into a video (1311) based on a priority (or weight) for each of the facial expression identified within the region (1312) and the gesture identified within the region (1314). For example, at least one processor (207) may display the video (1321) combined with the at least one through a display (208).

[0146] States (1310, 1320) for streaming images (or videos) acquired through the camera (203) are illustrated in FIG. 13a, but embodiments of the present disclosure are not limited thereto. For example, states (1330, 1340) related to a user interface (185) for video calls are illustrated in FIG. 13b. For example, a description of states (1310, 1320) may be referenced to a description of states (1330, 1340).

[0147] Referring to FIG. 13b, states (1330) and (1340) may be described as states in which virtual objects (1322, 1324) are superimposed on a second preview screen (195) within a user interface (185) for video calls. For example, at least one processor (207) may identify a visual object expressing an expression and a visual object expressing a gesture, respectively, based on performing gesture recognition and / or facial expression recognition on an image displayed on the second preview screen (195) in state (1330). For example, at least one processor (207) may identify a visual object expressing an expression within area (1332) and a visual object expressing a defined gesture within area (1334).

[0148] In state (1340), at least one processor (207) can display virtual objects (1322, 1324) superimposed on the second preview screen (195) based on identifying a visual object expressing a facial expression within the area (1332) and a visual object expressing a definition gesture within the area (1334).

[0149] According to one embodiment, at least one processor (207) may display a virtual object (1322) superimposed on a second preview screen (195) based on identifying a facial expression within an area (1332). At least one processor (207) may display a virtual object (1324) superimposed on a second preview screen (195) based on identifying a defined gesture within an area (1334). For example, at least one processor (207) may change the size of the virtual object (1322) and / or the virtual object (1324) when at least a portion of the first area to which the virtual object (1322) is to be displayed overlaps with at least a portion of the second area to which the virtual object (1324) is to be displayed. For example, at least one processor (207) may change at least one of the first area where the virtual object (1322) is to be displayed and / or the second area where the virtual object (1324) is to be displayed when at least a part of the first area where the virtual object (1322) is to be displayed overlaps with at least a part of the second area where the virtual object (1324) is to be displayed.

[0150] FIGS. 14a and FIGS. 14b illustrate exemplary operation of an electronic device that combines a single virtual object corresponding to gestures.

[0151] Referring to FIG. 14a, the state (1410) can be described as a state in which a video (1411) combined with a virtual object (417) corresponding to a gesture (e.g., a hand heart gesture) is displayed. For example, at least one processor (207) may display the video (1411) combined with the virtual object (417) through a display (208) based on identifying a visual object (413) representing the gesture within an area (415). For example, the video (1411) may be an example of the video (690) of FIG. 6.

[0152] The state (1420) can be described as a state in which a video (1421) combined with a virtual object (1427) corresponding to another gesture (e.g., a finger heart gesture) is displayed. For example, the video (1421) may be an example of the video (690) of FIG. 6. For example, at least one processor (207) may display the video (1421) combined with the virtual object (1427) through a display (208) based on identifying a visual object (1423) representing the other gesture within an area (1425). For example, the gesture represented by the visual object (413) (e.g., a hand heart gesture) may be different from the other gesture represented by the visual object (1423) (e.g., a finger heart gesture). For example, even though the gesture and the other gesture are different, the virtual object (417) combined with the video (1411) may be the same as the virtual object (1427) combined with the video (1421). For example, two or more gestures may be mapped to a single virtual object.

[0153] States (1410, 1420) for streaming images (or videos) acquired through the camera (203) are illustrated in FIG. 14a, but embodiments of the present disclosure are not limited thereto. For example, states (1430, 1440) related to a user interface (185) for video calls are illustrated in FIG. 14b. For example, a description of states (1410, 1420) may be referenced as a description of states (1430, 1440).

[0154] Referring to FIG. 14b, states (1430) and (1440) may be described as states in which a virtual object (417) (or virtual object (1427)) is continuously maintained in a superimposed manner on a second preview screen (195) within a user interface (185) for video calls. For example, at least one processor (207) may display the virtual object (417) in a superimposed manner on the second preview screen (195) based on identifying a visual object (413) representing a first definition gesture. For example, while displaying the virtual object (417) on the second preview screen (195), at least one processor (207) may maintain the display of the virtual object (417) based on identifying a visual object (1423) for displaying the virtual object (1427). For example, the virtual object (417) may be identical to the virtual object (1427). For example, at least one processor (207) may display a virtual object (1427) having a size different from that of a virtual object (417) on the second preview screen (195) by overlapping it, because the size of the visual object (413) identified in state (1430) is different from the size of the visual object (1423) identified in state (1440). For example, at least one processor (207) may maintain the virtual object (417) on the second preview screen (195) by overlapping it, based on performing gesture recognition on the image displayed on the second preview screen (195) for the visual object to display the same virtual object. For example, at least one processor (207) may change the size of the virtual object (417) based on identifying a change in the size of the visual object to display the virtual object (417).For example, at least one processor (207) can reduce the size of a virtual object (417) based on the determination that the size of the area (1445) where the visual object (1423) is identified is smaller than the area (435) where the visual object (413) is identified.

[0155] At least one processor (207) can perform gesture recognition on images acquired through the camera (203) while displaying an emoji graphic object (e.g., virtual object (417)). At least one processor (207) can maintain the display of the emoji graphic object (e.g., virtual object (1427)) based on identifying a third visual object (e.g., visual object (1423)) that represents a third defined gesture for displaying the emoji graphic object. At least one processor (207) can change the size of the emoji graphic object based on the size of the third visual object.

[0156] FIG. 15 illustrates an exemplary operation of an electronic device that displays virtual objects based on the number of visual objects identified within an image.

[0157] Referring to FIG. 15, the state (1510) can be described as a state in which a virtual object is displayed based on the number of visual objects identified within an image. For example, at least one processor (207) can identify visual objects (1515, 1516) corresponding to bodies within an image acquired through a camera (203). For example, at least one processor (207) can use the image to identify a gesture represented by the visual objects (1515, 1516). For example, at least one processor (207) can determine the size of a virtual object (1517) corresponding to a gesture based on the size of the area (1513) in which the gesture is identified. For example, at least one processor (207) can display the virtual object (1517) having the determined size through a display (208). For example, a first virtual object (e.g., virtual object (417)) corresponding to a first gesture expressed through a single visual object may differ from a second virtual object (e.g., virtual object (1517)) corresponding to a second gesture expressed through two or more visual objects (1515, 1516). For example, even if the size of the area where the first gesture is identified (e.g., area (415)) is the same as the size of the area where the second gesture is identified (e.g., area (1513)), the size of the second virtual object (e.g., virtual object (1517)) corresponding to the second gesture may be larger than the size of the first virtual object (e.g., virtual object (417)) corresponding to the first gesture. However, the present disclosure is not limited thereto.

[0158] The state (1520) can be described as a state in which a virtual object (1517) is superimposed on a second preview screen (195) within a user interface (185) for video calls. For example, at least one processor (207) may superimpose the virtual object (1517) on the second preview screen (195) based on identifying visual objects (1515, 1516) that represent a defined gesture according to gesture recognition performed on images acquired through the camera (203). For example, at least one processor (207) may superimpose the virtual object (1517) on the second preview screen (195) based on identifying visual objects (1515, 1516) within an area (1523) on the second preview screen (195).

[0159] FIGS. 16a and FIGS. 16b illustrate exemplary operation of an electronic device displaying a visual indication representing a time interval for displaying a virtual object.

[0160] Referring to FIG. 16a, states (1610), (1620), and (1630) can be described as states in which a visual indication (1622) representing a time interval based on a second definition gesture is displayed when streaming. For example, at least one processor (207) can identify a second definition gesture indicating an area (or location) where a virtual object is to be displayed based on performing gesture recognition on images acquired through the camera (203) in state (1610).

[0161] At least one processor (207) may display a visual indication (1622) indicating a time interval in a superposition based on identifying the second definition gesture in state (1620). For example, the visual indication (1622) may be described as an indication for indicating a time interval for displaying a virtual object (1027) on an area represented by the second definition gesture. The second definition gesture may be described as a gesture for indicating an area (or location) where a virtual object (1027) representing the first definition gesture will be displayed. For example, in state (1630), at least one processor (207) may display a virtual object (1027) on an area (1065) represented by the second definition gesture based on identifying a visual object (413) within the time interval. For example, at least one processor (207) may display a virtual object (1027) at a location associated with the visual object (413) based on not identifying the visual object (413) within the time interval. For example, the display of a virtual object (1027) at a location associated with the visual object (413) is described above with reference to FIG. 5a.

[0162] According to one embodiment, the visual indication (1622) may be displayed superimposed only during the time interval. For example, at least one processor (207) may display the visual indication (1622) based on identifying the second definition gesture in the state (1610). For example, at least one processor (207) may stop displaying the visual indication (1622) based on displaying the visual indication (1622) during the time interval. For example, when at least one processor (207) identifies the visual object (413) while the display of the visual indication (1622) is stopped, it may refrain from, bypass, or block displaying the virtual object (417) on the area (1065) that appears due to the second definition gesture.

[0163] States (1610, 1620, 1630) for streaming images (or videos) acquired through the camera (203) are illustrated in FIG. 16a, but embodiments of the present disclosure are not limited thereto. For example, states (1640, 1650, 1660) related to a user interface (185) for video calls are illustrated in FIG. 16b. For example, a description of states (1610, 1620, 1630) may be referenced to a description of states (1640, 1650, 1660).

[0164] Referring to FIG. 16b, states (1640), (1650), and (1660) can be described as states in which a visual indication (1622) is displayed on a second preview screen (195) within a user interface (185) for video calls. For example, at least one processor (207) can identify a second definition gesture to indicate the location where a virtual object (1027) is to be displayed in state (1640) based on performing gesture recognition on an image acquired through a camera (203). For example, at least one processor (207) may superimpose a visual indication (1622) on the second preview screen (195) indicating a time interval for superimposing a virtual object (1027) on an area (1065) that appears by the second definition gesture, based on identifying the second definition gesture on the second preview screen (195) in state (1650). For example, at least one processor (207) may superimpose a virtual object (1027) on an area (1065) of the second preview screen (195) based on identifying a visual object (413) representing the virtual object (1027) within the time interval in state (1660).

[0165] At least one processor (207) can identify whether a definition gesture for displaying an emoji graphic object is detected within a time interval based on a gesture recognition performed while displaying a second preview screen (195) within a user interface (185), based on identifying a second visual object (e.g., visual object (1013)) representing a second definition gesture based on a gesture recognition performed while displaying a second preview screen (195) within a user interface (185). At least one processor (207) can display an emoji graphic object superimposed on an area (e.g., area (1065)) of the second preview screen (195) that appears by the second definition gesture, through a display (208), based on identifying that the definition gesture is detected as a first definition gesture within the time interval.

[0166] At least one processor (207) may display a visual indication (1622) on the second preview screen (195) superimposed on the second visual screen (195) that indicates a time interval associated with an area (e.g., area (1065)) represented by the second visual object, based on identifying the second visual object (e.g., visual object (1013)). For example, at least one processor (207) may display an emoji graphic object (e.g., virtual object (1027)) superimposed on the area, based on identifying the first visual object (e.g., visual object (413)) that represents the first definition gesture within the time interval represented by the visual indication (1622).

[0167] FIGS. 17a and FIGS. 17b illustrate exemplary operation of an electronic device that displays a visual indication of where a virtual object is to be displayed.

[0168] Referring to FIG. 17a, states (1710), (1720), and (1730) may be described as states that display a visual indication (1722) indicating an area (or location) where a virtual object (1027) is to be displayed. For example, at least one processor (207) may identify a second definition gesture indicating an area (1025) where a virtual object (1027) can be displayed in state (1710) by performing gesture recognition on images acquired through a camera (203).

[0169] At least one processor (207) may display a visual indication (1722) based on identifying a second definition gesture in state (1720). For example, the visual indication (1722) may indicate the location where a virtual object (1027) is to be displayed.

[0170] For example, at least one processor (207) can display the virtual object (1027) superimposed on an area (1025) indicated by a visual indication (1722) based on identifying a visual object (413) representing the virtual object (1027) in a state (1730).

[0171] States (1710, 1720, 1730) for streaming images (or videos) acquired through the camera (203) are illustrated in FIG. 17a, but embodiments of the present disclosure are not limited thereto. For example, states (1740, 1750, 1760) related to a user interface (185) for video calls are illustrated in FIG. 17b. For example, a description of the states (1710, 1720, 1730) may be referenced to a description of the states (1740, 1750, 1760).

[0172] Referring to FIG. 17b, states (1740), (1750), and (1760) may be described as states indicating a visual indication (1722) that indicates the location where a virtual object (1027) is to be displayed on a second preview screen (195) within a user interface (185) for video calls. For example, at least one processor (207) may identify a second visual object (e.g., a visual object identified within area (1045)) that represents a second defining gesture indicating an area (1065) by performing gesture recognition on images acquired through a camera (203) in state (1740). For example, at least one processor (207) may superimpose a visual indication (1722) on the second preview screen (195) indicating an area (1065) where a virtual object (1027) is to be displayed, based on identifying the second visual object in state (1750). For example, at least one processor (207) may superimpose a virtual object (1027) on an area (1065) on the second preview screen (195), based on identifying a visual object (413) representing the virtual object (1027) in state (1760).

[0173] At least one processor (207) may display a visual indication (1722) associated with an area (e.g., area (1065)) on a second preview screen (195) based on identifying a second visual object (e.g., visual object (1013)) representing a second definition gesture based on gesture recognition performed while displaying a second screen within a user interface (185). For example, at least one processor (207) may display an emoji graphic object (e.g., virtual object (1027)) superimposed on the area based on identifying a visual object (e.g., visual object (413)) representing a first definition gesture based on gesture recognition performed while displaying the visual indication (1722).

[0174] According to one embodiment, with reference to FIGS. 16a and 16b, a visual indication (1622) indicating a time interval associated with a virtual object is described, and with reference to FIGS. 17a and 17b, a visual indication (1722) indicating an area (or location) associated with a virtual object is described, but the present disclosure is not limited thereto. For example, a visual indication for indicating a time interval (e.g., visual indication (1622)) may be identical to a visual indication for indicating an area where the virtual object is to be displayed (e.g., visual indication (1722)). For example, a visual indication (1622) for indicating a time interval may be displayed to indicate an area where the virtual object is to be displayed.

[0175] FIG. 18 illustrates an exemplary operation of an electronic device displaying a video in which a virtual object is synthesized.

[0176] Referring to FIG. 18, states (1810) and (1820) can be described as states in which a virtual object displays a composite video. For example, at least one processor (207) may store images (or videos) for a video call in memory (206) when performing a video call. For example, when storing the images, at least one processor (207) may store other images obtained from an external electronic device (not shown) through a communication circuit (205) together with the images.

[0177] The state (1810) can be described as a state in which a video (or images) acquired from an external electronic device is displayed on the first preview screen (190). For example, at least one processor (207) can display the video (1812) on the first preview screen (190). For example, at least one processor (207) can display the video (1814) on the second preview screen (195). For example, the video (1814) displayed on the second preview screen (195) can be described as a video in which a virtual object is combined (or a virtual object is superimposed). For example, the video (1814) can be described as a video in which a virtual object is combined (or a virtual object is superimposed) on a video acquired through the camera (203). For example, the video (1812) displayed on the first preview screen (190) can be described as a video received from an external electronic device. At least one processor (207) can display an executable object (1816) through the display (208) while displaying the video (1812) and video (1814) through the display (208). For example, at least one processor (207) can switch or change the screen where the video (1812) is displayed to the screen where the video (1814) is displayed based on receiving user input regarding the executable object (1816).

[0178] State (1820) can be described as a state in which the video (1812) is displayed on the second preview screen (195) and the video (1814) is displayed on the first preview screen (190). For example, at least one processor (207) may switch or change the state of displaying the recorded video from state (1810) to state (1820) based on receiving user input regarding the executable object (1816). For example, at least one processor (207) may switch or change the state of displaying the recorded video from state (1820) to state (1810) based on receiving user input regarding the executable object (1816) in state (1820).

[0179] According to one embodiment, at least one processor (207) can transmit a signal to an external electronic device through a communication circuit (205) that causes the external electronic device to display an emoji graphic object superimposed on images acquired through a camera (203) when displaying an emoji graphic object superimposed on a second preview screen (195).

[0180] FIG. 19 is a block diagram of an electronic device (1901) in a network environment (1900) according to various embodiments. Referring to FIG. 19, in the network environment (1900), the electronic device (1901) may communicate with an electronic device (1902) through a first network (1998) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (1904) or a server (1908) through a second network (1999) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1901) may communicate with the electronic device (1904) through a server (1908). According to one embodiment, the electronic device (1901) may include a processor (1920), memory (1930), input module (1950), sound output module (1955), display module (1960), audio module (1970), sensor module (1976), interface (1977), connection terminal (1978), haptic module (1979), camera module (1980), power management module (1988), battery (1989), communication module (1990), subscriber identification module (1996), or antenna module (1997). In some embodiments, at least one of these components (e.g., connection terminal (1978)) may be omitted from the electronic device (1901), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (1976), camera module (1980), or antenna module (1997)) may be integrated into a single component (e.g., display module (1960)).

[0181] The processor (1920) can, for example, execute software (e.g., program (1940)) to control at least one other component (e.g., hardware or software component) of the electronic device (1901) connected to the processor (1920) and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (1920) can store commands or data received from other components (e.g., sensor module (1976) or communication module (1990)) in volatile memory (1932), process the commands or data stored in volatile memory (1932), and store the resulting data in non-volatile memory (1934). According to one embodiment, the processor (1920) may include a main processor (1921) (e.g., a central processing unit or an application processor) or an auxiliary processor (1923) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (1901) includes a main processor (1921) and an auxiliary processor (1923), the auxiliary processor (1923) may be configured to use less power than the main processor (1921) or to be specialized for a designated function. The auxiliary processor (1923) may be implemented separately from the main processor (1921) or as part thereof.

[0182] The auxiliary processor (1923) may control at least some of the functions or states associated with at least one component of the electronic device (1901) (e.g., display module (1960), sensor module (1976), or communication module (1990)) on behalf of the main processor (1921) while the main processor (1921) is in an inactive (e.g., sleep) state, or together with the main processor (1921) while the main processor (1921) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (1923) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (1980) or communication module (1990)). According to one embodiment, the auxiliary processor (1923) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (1901) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (1908)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0183] The memory (1930) may store various data used by at least one component of the electronic device (1901) (e.g., a processor (1920) or a sensor module (1976)). The data may include, for example, software (e.g., a program (1940)) and input or output data for related commands. The memory (1930) may include volatile memory (1932) or non-volatile memory (1934).

[0184] The program (1940) may be stored as software in memory (1930) and may include, for example, an operating system (1942), middleware (1944), or an application (1946).

[0185] The input module (1950) can receive commands or data to be used for a component of the electronic device (1901) (e.g., processor (1920)) from outside the electronic device (1901) (e.g., user). The input module (1950) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0186] The sound output module (1955) can output a sound signal to the outside of the electronic device (1901). The sound output module (1955) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0187] The display module (1960) can visually provide information to an external (e.g., user) of the electronic device (1901). The display module (1960) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (1960) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0188] The audio module (1970) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (1970) can acquire sound through an input module (1950) or output sound through an audio output module (1955) or an external electronic device (e.g., electronic device (1902)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (1901).

[0189] The sensor module (1976) can detect the operating state of the electronic device (1901) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (1976) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0190] The interface (1977) may support one or more specified protocols that can be used for the electronic device (1901) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (1902)). According to one embodiment, the interface (1977) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0191] The connection terminal (1978) may include a connector through which the electronic device (1901) can be physically connected to an external electronic device (e.g., electronic device (1902)). According to one embodiment, the connection terminal (1978) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0192] The haptic module (1979) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (1979) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0193] The camera module (1980) can capture still images and video. According to one embodiment, the camera module (1980) may include one or more lenses, image sensors, image signal processors, or flashes.

[0194] The power management module (1988) can manage the power supplied to the electronic device (1901). According to one embodiment, the power management module (1988) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0195] The battery (1989) can supply power to at least one component of the electronic device (1901). According to one embodiment, the battery (1989) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0196] The communication module (1990) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (1901) and an external electronic device (e.g., electronic device (1902), electronic device (1904), or server (1908)), and the performance of communication through the established communication channel. The communication module (1990) may include one or more communication processors that operate independently of the processor (1920) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1990) may include a wireless communication module (1992) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (1994) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (1904) through a first network (1998) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (1999) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1992) can identify or authenticate the electronic device (1901) within a communication network such as the first network (1998) or the second network (1999) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (1996).

[0197] The wireless communication module (1992) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. The NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (1992) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (1992) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (1992) can support various requirements specified in the electronic device (1901), external electronic device (e.g., electronic device (1904)), or network system (e.g., second network (1999)). According to one embodiment, the wireless communication module (1992) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0198] An antenna module (1997) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (1997) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (1997) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (1998) or a second network (1999), may be selected from the plurality of antennas, for example, by a communication module (1990). A signal or power may be transmitted or received between the communication module (1990) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (1997).

[0199] According to various embodiments, the antenna module (1997) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0200] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0201] According to one embodiment, commands or data may be transmitted or received between the electronic device (1901) and an external electronic device (1904) through a server (1908) connected to a second network (1999). Each of the external electronic devices (1902, or 1904) may be the same or a different type of device as the electronic device (1901). According to one embodiment, all or part of the operations performed on the electronic device (1901) may be performed on one or more of the external electronic devices (1902, 1904, or 1908). For example, if the electronic device (1901) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (1901) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (1901). The electronic device (1901) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (1901) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1904) may include an Internet of Things (IoT) device. The server (1908) may be an intelligent server using machine learning and / or neural networks.According to one embodiment, an external electronic device (1904) or server (1908) may be included within the second network (1999). The electronic device (1901) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0202] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0203] An electronic device as described above (e.g., electronic device (100)) may include a memory (e.g., memory (206)) for storing instructions. The electronic device may include a communication circuit (e.g., communication circuit (205)). The electronic device may include a display (e.g., display (208)). The electronic device may include a camera (e.g., camera (203)). The electronic device may include at least one processor (e.g., at least one processor (207)). The instructions may cause the electronic device to execute the application software to display, through the display, a user interface of the application software for a video call with an external electronic device performed using the communication circuit, when executed individually or collectively by the at least one processor. The user interface of the application software may include a first screen area for displaying first video images provided from the external electronic device. The above instructions may cause the electronic device to display, through the display, a second screen area corresponding to the second video images within the user interface, based on the second video images for the video call being acquired through the camera, when executed individually or collectively by the at least one processor. The above instructions may cause the electronic device to perform gesture recognition on the second video images for the video call acquired through the camera while displaying the second video images within the second screen area within the user interface, when executed individually or collectively by the at least one processor.The above instructions may cause the electronic device to identify an emoji graphic object representing a definition gesture based on identifying a visual object representing a definition gesture according to the gesture recognition performed while displaying second video images within the second screen area within the user interface, when executed individually or collectively by the at least one processor. The visual object may correspond to a part of the body associated with the definition gesture. The above instructions may cause the electronic device to display the emoji graphic object superimposed on the second screen area through the display, when executed individually or collectively by the at least one processor. The above instructions may cause the electronic device to identify a change in the size of the visual object included in the second video images acquired through the camera while the emoji graphic object is displayed superimposed on the second video images within the second screen area, when executed individually or collectively by the at least one processor. The above instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to change the size of the emoji graphic object superimposed on the second video images within the second screen area based on identifying the change in the size of the visual object.

[0204] According to one embodiment, the instructions may cause the electronic device to transmit a signal to the external electronic device through the communication circuit, which causes the emoji graphic object to be superimposed on the second video images acquired through the camera, based on displaying the emoji graphic object, when executed individually or collectively by the at least one processor.

[0205] According to one embodiment, the definition gesture may be a first definition gesture. The visual object may be a first visual object. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify a second visual object representing a second definition gesture according to the gesture recognition performed while displaying the second screen area within the user interface, before identifying the first visual object representing the first definition gesture. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to display the emoji graphic object through the display in superposition on the area of ​​the second screen area that appears by the second definition gesture.

[0206] As a first example according to one embodiment, the instructions may cause the electronic device to identify whether a definition gesture for displaying an emoji graphic object is detected within a time interval based on the gesture recognition performed while displaying the second screen area within the user interface, based on identifying the second visual object expressing the second definition gesture based on the gesture recognition performed while displaying the second screen area within the user interface when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to display the emoji graphic object through the display in superposition on the area of ​​the second screen area appearing by the second definition gesture, based on identifying that the definition gesture is detected as the first definition gesture within the time interval when executed individually or collectively by the at least one processor. The above instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to display the emoji graphic object through the display in superposition on a different area distinct from the area, based on identifying that the definition gesture is not detected as the first definition gesture within the time interval.

[0207] As a second example according to one embodiment, the instructions may cause the electronic device to identify candidate images acquired during a time interval among the stored second video images, based on identifying the first visual object expressing the first definition gesture according to the gesture recognition performed while displaying the second screen area within the user interface, after storing the second video images acquired through the camera in memory when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to identify the second definition gesture according to the gesture recognition performed to identify the definition gesture indicating the area where an emoji graphic object is to be displayed for the candidate images when executed individually or collectively by the at least one processor. When the above instructions are executed individually or collectively by the at least one processor, they may cause the electronic device to display the emoji graphic object through the display in a superposition on the area of ​​the second screen region that appears by the second definition gesture.

[0208] As a third example according to one embodiment, the instructions may cause the electronic device to display a first visual indication associated with the area on the second screen area, based on identifying the second visual object expressing the second defined gesture according to the gesture recognition performed while displaying the second screen area within the user interface, when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to display the emoji graphic object superimposed on the area, based on identifying the first visual object expressing the first defined gesture according to the gesture recognition performed while displaying the first visual indication, when executed individually or collectively by the at least one processor.

[0209] As a fourth example according to the first example of an embodiment or the third example of an embodiment, the instructions may cause the electronic device to display a second visual indication representing a time interval associated with the area represented by the second visual object, based on identifying the second visual object when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to display the emoji graphic object superimposed on the area, based on identifying the first visual object representing the first defined gesture within the time interval represented by the second visual indication when executed individually or collectively by the at least one processor.

[0210] According to one embodiment, the definition gesture may be a first definition gesture. The instructions may cause the electronic device to perform the gesture recognition on second video images acquired through the camera while displaying the emoji graphic object when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to maintain the display of the emoji graphic object based on identifying a third visual object expressing a third definition gesture for displaying the emoji graphic object when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to change the size of the emoji graphic object based on the size of the third visual object when executed individually or collectively by the at least one processor.

[0211] According to one embodiment, the definition gesture may be a first definition gesture. The emoji graphic object may be a first emoji graphic object. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to perform the gesture recognition on second video images acquired through the camera while displaying the first emoji graphic object. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to refrain from displaying the second emoji graphic object on the second screen area during a time interval when identifying a fourth visual object expressing a fourth definition gesture for displaying the second emoji graphic object.

[0212] According to one embodiment, the electronic device may further include a sensor. The instructions may cause the electronic device to identify, through the sensor, the distance between the part of the body expressing the gesture and the electronic device when acquiring second video images through the camera, when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to determine the size of the emoji graphic object expressing the defined gesture identified based on performing gesture recognition on the second video images, based on the identified distance, when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to display the emoji graphic object having the determined size through the display, superimposed on the second screen area, when executed individually or collectively by the at least one processor.

[0213] According to one embodiment, the instructions may cause the electronic device to identify, through a first trained model, an ROI where a body is presumed to be identified, based on performing object detection on the second video images before performing gesture recognition on the second video images acquired through the camera, when executed individually or collectively by the at least one processor. The shape of the ROI may correspond to a rectangle. The instructions may cause the electronic device to identify the visual object expressing the defined gesture, based on performing gesture recognition on the ROI among the ROI and a remaining area different from the ROI, when executed individually or collectively by the at least one processor.

[0214] According to one embodiment, the instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to identify, through a second trained model, a first area in which a body is identified and a second area different from the first area, using second video images acquired through the camera. The second screen area may include the first area and the second area. The instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to display the emoji graphic object through the display, superimposed on the second area among the first area and the second area.

[0215] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to transmit a signal to the external electronic device through the communication circuit, which causes the external electronic device to display the emoji graphic object in superposition on the second video images acquired through the camera.

[0216] According to one embodiment, the instructions may cause the electronic device to identify a change in the position of the visual object while the emoji graphic object is displayed superimposed on the second screen area when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to change the position of the emoji graphic object superimposed on the second screen area based on identifying the change in the position of the visual object when executed individually or collectively by the at least one processor.

[0217] According to one embodiment, the second screen area may be superimposed on the first screen area.

[0218] According to one embodiment, the size of the second screen area may be smaller than the size of the first screen area.

[0219] A method performed by an electronic device (e.g., electronic device (100)) having a communication circuit (e.g., communication circuit (205)), a display (e.g., display (208)), and a camera (e.g., camera (203)) as described above may include an operation of executing said application software to display, through said display, a user interface of said application software for a video call with an external electronic device performed using said communication circuit. The user interface of said application software may include a first screen area for displaying first video images provided from said external electronic device. The method may include an operation of displaying, through said display, a second screen area corresponding to said second video images within said user interface, based on said second video images for the video call being acquired through said camera. The method may include an operation of performing gesture recognition on said second video images for the video call acquired through said camera while displaying said second video images within said second screen area within said user interface. The above method may include an operation of identifying an emoji graphic object representing a definition gesture based on identifying a visual object representing a definition gesture according to the gesture recognition performed while displaying second video images within the second screen area within the user interface. The visual object may correspond to a part of the body associated with the definition gesture. The above method may include an operation of displaying the emoji graphic object through the display in a superimposed manner on the second screen area.The above method may include an operation of identifying a change in the size of the visual object included in the second video images acquired through the camera while the emoji graphic object is displayed superimposed on the second video images within the second screen area. Based on identifying the change in the size of the visual object, the above method may include an operation of changing the size of the emoji graphic object superimposed on the second video images within the second screen area.

[0220] According to one embodiment, the method may include the operation of transmitting a signal to an external electronic device through the communication circuit, which causes the emoji graphic object to be superimposed on second video images acquired through the camera, based on displaying the emoji graphic object.

[0221] According to one embodiment, the definition gesture may be a first definition gesture. The visual object may be a first visual object. The method may include an action of identifying a second visual object representing a second definition gesture according to the gesture recognition performed while displaying the second screen area within the user interface, before identifying the first visual object representing the first definition gesture. The method may include an action of displaying the emoji graphic object superimposed on the area of ​​the second screen area appearing by the second definition gesture through the display.

[0222] As a first example according to one embodiment, the method may include an operation of identifying whether a definition gesture for displaying an emoji graphic object is detected within a time interval based on the gesture recognition performed while displaying the second screen area within the user interface, based on identifying the second visual object expressing the second definition gesture based on the gesture recognition performed while displaying the second screen area within the user interface. The method may include an operation of displaying the emoji graphic object through the display superimposed on the area of ​​the second screen area appearing by the second definition gesture, based on identifying that the definition gesture is detected as the first definition gesture within the time interval. The method may include an operation of displaying the emoji graphic object through the display superimposed on another area different from the area, based on identifying that the definition gesture is not detected as the first definition gesture within the time interval.

[0223] As a second example according to one embodiment, the method may include an operation of identifying candidate images acquired during a time interval among the stored second video images, based on identifying the first visual object expressing the first definition gesture according to the gesture recognition performed while displaying the second screen area within the user interface after storing the second video images acquired through the camera in memory. The method may include an operation of identifying the second definition gesture according to the gesture recognition performed to identify the definition gesture indicating the area where an emoji graphic object is to be displayed for the candidate images. The method may include an operation of displaying the emoji graphic object through the display in a superposition on the area of ​​the second screen area that appears by the second definition gesture.

[0224] As a third example according to one embodiment, the method may include an operation of displaying a first visual indication associated with the area on the second screen area based on identifying the second visual object expressing the second defined gesture according to the gesture recognition performed while displaying the second screen area within the user interface. The method may include an operation of displaying the emoji graphic object superimposed on the area based on identifying the first visual object expressing the first defined gesture according to the gesture recognition performed while displaying the first visual indication.

[0225] As a fourth example according to the first example of an embodiment or the third example of an embodiment, the method may include an action of displaying a second visual indication representing a time interval associated with the area represented by the second visual object, based on identifying the second visual object. The method may include an action of displaying the emoji graphic object superimposed on the area, based on identifying the first visual object expressing the first defined gesture within the time interval represented by the second visual indication.

[0226] According to one embodiment, the definition gesture may be a first definition gesture. The method may include an operation of performing the gesture recognition on second video images acquired through the camera while displaying the emoji graphic object. The method may include an operation of maintaining the display of the emoji graphic object based on identifying a third visual object expressing a third definition gesture for displaying the emoji graphic object. The method may include an operation of changing the size of the emoji graphic object based on the size of the third visual object.

[0227] According to one embodiment, the definition gesture may be a first definition gesture. The emoji graphic object may be a first emoji graphic object. The method may include an operation of performing the gesture recognition on second video images acquired through the camera while displaying the first emoji graphic object. The method may include an operation of refraining from displaying the second emoji graphic object on the second screen area during a time interval when identifying a fourth visual object expressing a fourth definition gesture for displaying the second emoji graphic object.

[0228] According to one embodiment, the electronic device may further include a sensor. The method may include an action of identifying, through the sensor, the distance between the part of the body expressing the gesture and the electronic device when acquiring second video images through the camera. The method may include an action of determining the size of the emoji graphic object expressing the defined gesture identified based on performing gesture recognition on the second video images based on the identified distance. The method may include an action of displaying the emoji graphic object having the determined size superimposed on the second screen area through the display.

[0229] According to one embodiment, the method may include an action of identifying, through a first trained model, an ROI where a body is presumed to be identified, based on performing object detection on the second video images before performing gesture recognition on the second video images acquired through the camera. The shape of the ROI may correspond to a rectangle. The method may include an action of identifying the visual object expressing the defined gesture based on performing gesture recognition on the ROI among the ROI and a remaining area different from the ROI.

[0230] According to one embodiment, the method may include an operation of identifying a first region in which a body is identified and a second region different from the first region through a second trained model using second video images acquired through the camera. The second screen region may include the first region and the second region. The shape of the ROI may correspond to a rectangle. The method may include an operation of displaying the emoji graphic object through the display in a superimposed manner on the second region among the first region and the second region.

[0231] According to one embodiment, the method may include the operation of transmitting a signal to the external electronic device through the communication circuit, which causes the external electronic device to display the emoji graphic object superimposed on the second video images acquired through the camera when the emoji graphic object is displayed superimposed on the second screen area.

[0232] According to one embodiment, the method may include an operation of identifying a change in the position of the visual object while the emoji graphic object is displayed superimposed on the second screen area. The method may include an operation of changing the position of the emoji graphic object superimposed on the second screen area based on identifying the change in the position of the visual object.

[0233] According to one embodiment, the second screen area may be superimposed on the first screen area.

[0234] According to one embodiment, the size of the second screen area may be smaller than the size of the first screen area.

[0235] In a computer-readable storage medium in which one or more programs are stored as described above, the one or more programs may include instructions that cause the electronic device (e.g., electronic device (100)) having a communication circuit (e.g., communication circuit (205)), a display (e.g., display (208)), and a camera (e.g., camera (203)) to execute the application software to display through the display a user interface of the application software for a video call with an external electronic device performed using the communication circuit. The user interface of the application software may include a first screen area for displaying first video images provided from the external electronic device. The one or more programs may include instructions that cause the electronic device to display through the display a second screen area corresponding to the second video images within the user interface, based on the second video images for the video call being acquired through the camera, when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to perform gesture recognition on the second video images for the video call acquired through the camera while displaying the second video images within the second screen area within the user interface when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to identify an emoji graphic object representing the defined gesture based on identifying a visual object representing the defined gesture according to the gesture recognition performed while displaying the second video images within the second screen area within the user interface when executed by the electronic device.The visual object may correspond to a part of the body associated with the defined gesture. The one or more programs may include instructions that cause the electronic device to display the emoji graphic object superimposed on the second screen area via the display when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to identify a change in the size of the visual object included in the second video images acquired through the camera while the emoji graphic object is displayed superimposed on the second video images within the second screen area when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to change the size of the emoji graphic object superimposed on the second video images within the second screen area based on identifying the change in the size of the visual object when executed by the electronic device.

[0236] According to one embodiment, the one or more programs may include instructions that cause the electronic device to transmit a signal to the external electronic device through the communication circuit, which causes the emoji graphic object to be superimposed on the second video images acquired through the camera, based on displaying the emoji graphic object when executed by the electronic device.

[0237] According to one embodiment, the definition gesture may be a first definition gesture. The visual object may be a first visual object. The one or more programs may include instructions that cause the electronic device to identify a second visual object representing a second definition gesture according to the gesture recognition performed while displaying the second screen area within the user interface, before identifying the first visual object representing the first definition gesture when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to display the emoji graphic object through the display in a superposition on the area of ​​the second screen area that appears by the second definition gesture when executed by the electronic device.

[0238] As a first example according to one embodiment, the one or more programs may include instructions that cause the electronic device to identify whether a definition gesture for displaying an emoji graphic object is detected within a time interval according to the gesture recognition performed while displaying the second screen area within the user interface, based on identifying the second visual object expressing the second definition gesture according to the gesture recognition performed while displaying the second screen area within the user interface when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to display the emoji graphic object through the display in superposition on the area of ​​the second screen area appearing by the second definition gesture, based on identifying that the definition gesture is detected as the first definition gesture within the time interval when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to display the emoji graphic object through the display in a superposition on a different area different from the area, based on identifying that the defined gesture is not detected as the first defined gesture within the time interval when executed by the electronic device.

[0239] As a second example according to one embodiment, the one or more programs may include instructions that cause the electronic device to identify candidate images acquired during a time interval among the stored second video images, based on identifying the first visual object expressing the first definition gesture according to the gesture recognition performed while displaying the second screen area within the user interface, after storing the second video images acquired through the camera in memory when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to identify the second definition gesture according to the gesture recognition performed to identify the definition gesture indicating the area where an emoji graphic object is to be displayed for the candidate images when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to display, through the display, the emoji graphic object superimposed on the area of ​​the second screen region that appears by the second definition gesture when executed by the electronic device.

[0240] As a third example according to one embodiment, the one or more programs may include instructions that cause the electronic device to display a first visual indication associated with the area on the second screen area, based on identifying the second visual object expressing the second defined gesture according to the gesture recognition performed while displaying the second screen area within the user interface when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to display the emoji graphic object superimposed on the area, based on identifying the first visual object expressing the first defined gesture according to the gesture recognition performed while displaying the first visual indication when executed by the electronic device.

[0241] As a fourth example according to the first example of an embodiment or the third example of an embodiment, the one or more programs may include instructions that cause the electronic device to display a second visual indication representing a time interval associated with the area represented by the second visual object, based on identifying the second visual object when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to display the emoji graphic object superimposed on the area, based on identifying the first visual object representing the first defined gesture within the time interval represented by the second visual indication when executed by the electronic device.

[0242] According to one embodiment, the definition gesture may be a first definition gesture. The one or more programs may include instructions that cause the electronic device to perform the gesture recognition on second video images acquired through the camera while displaying the emoji graphic object when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to maintain the display of the emoji graphic object based on identifying a third visual object expressing a third definition gesture for displaying the emoji graphic object when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to change the size of the emoji graphic object based on the size of the third visual object when executed by the electronic device.

[0243] According to one embodiment, the definition gesture may be a first definition gesture. The emoji graphic object may be a first emoji graphic object. The one or more programs may include instructions that cause the electronic device to perform the gesture recognition on second video images acquired through the camera while displaying the first emoji graphic object when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to refrain from displaying the second emoji graphic object on the second screen area during a time interval when identifying a fourth visual object expressing a fourth definition gesture for displaying the second emoji graphic object when executed by the electronic device.

[0244] According to one embodiment, the electronic device may further include a sensor. The one or more programs may include instructions that cause the electronic device to identify, through the sensor, the distance between the part of the body expressing the gesture and the electronic device when the second video images are acquired through the camera when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to determine the size of the emoji graphic object expressing the defined gesture identified based on performing gesture recognition on the second video images based on the identified distance when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to display the emoji graphic object having the determined size through the display in a superimposed manner on the second screen area when executed by the electronic device.

[0245] According to one embodiment, the one or more programs may include instructions that cause the electronic device to identify, through a first trained model, an ROI where a body is presumed to be identified, based on performing object detection on the second video images before performing gesture recognition on the second video images acquired through the camera when executed by the electronic device. The shape of the ROI may correspond to a rectangle. The one or more programs may include instructions that cause the electronic device to identify the visual object expressing the defined gesture, based on performing gesture recognition on the ROI among the ROI and a remaining area different from the ROI when executed by the electronic device.

[0246] According to one embodiment, the one or more programs may include instructions that cause the electronic device to identify, through a second trained model, a first area in which a body is identified and a second area different from the first area, using second video images acquired through the camera when executed by the electronic device. The second screen area may include the first area and the second area. The one or more programs may include instructions that cause the electronic device to display, through the display, the emoji graphic object superimposed on the second area among the first area and the second area when executed by the electronic device.

[0247] According to one embodiment, the one or more programs may include instructions that cause the electronic device to transmit a signal to the external electronic device through the communication circuit, which causes the external electronic device to display the emoji graphic object superimposed on the second video images acquired through the camera when the emoji graphic object is displayed superimposed on the second screen area when executed by the electronic device.

[0248] According to one embodiment, the one or more programs may include instructions that cause the electronic device to identify a change in the position of the visual object while the emoji graphic object is displayed superimposed on the second screen area when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to change the position of the emoji graphic object superimposed on the second screen area based on identifying the change in the position of the visual object when executed by the electronic device.

[0249] According to one embodiment, the second screen area may be superimposed on the first screen area.

[0250] According to one embodiment, the size of the second screen area may be smaller than the size of the first screen area.

[0251] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0252] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0253] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0254] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may continuously store a computer-executable program, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or several combined hardware, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0255] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0256] Therefore, other implementations, other embodiments, and equivalents to the claims set forth below are also within the scope of the claims. According to one embodiment, the method according to the various embodiments disclosed herein may be provided as a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0257] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, Communication circuit; display; camera; Memory comprising one or more storage media for storing instructions; and It includes at least one processor comprising processing circuitry, and When the above instructions are executed individually or collectively by the at least one processor, The application software is executed to display, through the display, a user interface of the application software for video calling with an external electronic device performed using the communication circuit, and the user interface of the application software includes a first screen area for displaying first video images provided from the external electronic device; Based on the second video images for the video call being acquired through the camera, a second screen area corresponding to the second video images within the user interface is displayed through the display, and While displaying second video images within the second screen area of ​​the user interface, gesture recognition is performed on the second video images for the video call acquired through the camera, and Based on identifying a visual object expressing a definition gesture according to the gesture recognition performed while displaying second video images within the second screen area of ​​the user interface, an emoji graphic object expressing the definition gesture is identified, and the visual object corresponds to a part of the body associated with the definition gesture; The above emoji graphic object is displayed through the display in a superimposed manner on the second screen area, and Identifying a change in the size of the visual object included in the second video images acquired through the camera while the emoji graphic object is displayed superimposed on the second video images within the second screen area, and Based on identifying the change in the size of the visual object, to change the size of the emoji graphic object superimposed on the second video images within the second screen area, causing the above electronic device, Electronic device.

2. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor, Based on displaying the emoji graphic object, to transmit a signal to the external electronic device through the communication circuit that causes the emoji graphic object to be displayed superimposed on the second video images acquired through the camera. causing the above electronic device, Electronic device.

3. In claim 1, the above-described gesture is, It is the first definition gesture, and The above visual object is, It is the first visual object, and When the above instructions are executed individually or collectively by the at least one processor, Before identifying the first visual object expressing the first defined gesture, a second visual object expressing the second defined gesture is identified according to the gesture recognition performed while displaying the second screen area within the user interface, and To display the emoji graphic object superimposed on the area of ​​the second screen area appearing by the second definition gesture through the display. causing the above electronic device, Electronic device.

4. In Claim 3, When the above instructions are executed individually or collectively by the at least one processor, Based on identifying the second visual object expressing the second definition gesture according to the gesture recognition performed while displaying the second screen area within the user interface, identifying whether a definition gesture for displaying an emoji graphic object is detected within a time interval according to the gesture recognition performed while displaying the second screen area within the user interface, and Based on identifying that the definition gesture is detected as the first definition gesture within the above time interval, the emoji graphic object is displayed via the display in superposition on the area of ​​the second screen region appearing by the second definition gesture, and Based on identifying that the definition gesture is not detected as the first definition gesture within the above time interval, the emoji graphic object is displayed through the display in a superposition on a different area distinct from the area. causing the above electronic device, Electronic device.

5. In Claim 3, When the above instructions are executed individually or collectively by the at least one processor, After storing the second video images acquired through the camera in memory, candidate images acquired during a time interval are identified among the stored second video images based on identifying the first visual object expressing the first defined gesture according to the gesture recognition performed while displaying the second screen area in the user interface, and Identifying the second definition gesture according to gesture recognition performed to identify a definition gesture indicating an area where an emoji graphic object is to be displayed for the above candidate images, and To display the emoji graphic object superimposed on the area of ​​the second screen region appearing by the second definition gesture through the display. causing the above electronic device, Electronic device.

6. In Claim 3, When the above instructions are executed individually or collectively by the at least one processor, Based on identifying the second visual object expressing the second defined gesture according to the gesture recognition performed while displaying the second screen area within the user interface, a first visual indication associated with the area is displayed on the second screen area, and Based on identifying the first visual object expressing the first defined gesture according to the gesture recognition performed while displaying the first visual indication, to display the emoji graphic object superimposed on the area. causing the above electronic device, Electronic device.

7. In Claim 3, When the above instructions are executed individually or collectively by the at least one processor, Based on identifying the second visual object, display a second visual indication representing a time interval associated with the area represented by the second visual object, and Based on identifying the first visual object expressing the first defined gesture within the time interval indicated by the second visual indication, to display the emoji graphic object superimposed on the area. causing the above electronic device, Electronic device.

8. In Claim 1, the above-described gesture is, It is the first definition gesture, and When the above instructions are executed individually or collectively by the at least one processor, While displaying the emoji graphic object, the gesture recognition is performed on the second video images acquired through the camera, and Based on identifying a third visual object expressing a third defined gesture for displaying the emoji graphic object, maintaining the display of the emoji graphic object, and To change the size of the emoji graphic object based on the size of the third visual object. causing the above electronic device, Electronic device.

9. In Claim 1, the above-described gesture of definition is, It is the first definition gesture, and The above emoji graphic object is, The first emoji is a graphical object, and When the above instructions are executed individually or collectively by the at least one processor, Performing gesture recognition on second video images acquired through the camera while displaying the first emoji graphic object, and When identifying a fourth visual object expressing a fourth definition gesture for displaying a second emoji graphic object, refrain from displaying the second emoji graphic object on the second screen area during the time interval. causing the above electronic device, Electronic device.

10. In claim 1, the electronic device is, Includes further sensors, When the above instructions are executed individually or collectively by the at least one processor, When acquiring second video images through the camera, the distance between the body part expressing the gesture and the electronic device is identified through the sensor, and Determining the size of the emoji graphic object representing the identified defined gesture based on performing gesture recognition on the second video images based on the identified distance, and To display the emoji graphic object having the above-determined size by overlapping it on the second screen area through the display. causing the above electronic device, Electronic device.

11. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor, Before performing gesture recognition on the second video images acquired through the camera, based on performing object detection on the second video images, an ROI where a body is presumed to be identified is identified through a first trained model, and the shape of the ROI corresponds to a rectangle; and Based on performing gesture recognition on the ROI among the above ROI and remaining regions different from the above ROI, to identify the visual object expressing the defined gesture, causing the above electronic device, Electronic device.

12. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor, Using second video images acquired through the camera, a first area in which a body is identified and a second area different from the first area are identified through a second trained model, and the second screen area includes the first area and the second area; and To display the emoji graphic object above in a superimposed manner on the second area among the first area and the second area, through the display. causing the above electronic device, Electronic device.

13. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor, When displaying the emoji graphic object in a superposition on the second screen area, the external electronic device is to transmit a signal to the external electronic device through the communication circuit that causes the external electronic device to display the emoji graphic object in a superposition on the second video images acquired through the camera. causing the above electronic device, Electronic device.

14. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor, Identifying a change in the position of the visual object while the emoji graphic object is displayed superimposed on the second screen area, and Based on identifying the change in the position of the visual object, to change the position of the emoji graphic object superimposed on the second screen area, causing the above electronic device, Electronic device.

15. In claim 1, the second screen area is, Overlapping the first screen area, Electronic device.

Images