Electronic device for providing floating image and control method thereof

Abstract

This electronic device comprises: an aerial image display including an image module for forming a source image to be provided in the air and transmitting light for the source image, and a floating module for providing a floating image by forming a real image of the source image at a predetermined position in the air by using light for the source image; a communication interface; a camera; a memory for storing instructions; and at least one processor, wherein the instructions, when executed individually or collectively by the at least one processor, instruct the electronic device to: acquire an image via the camera; when a user is recognized via the image, receive context information of an external device from the external device via the communication interface; acquire information related to the context information of the external device; and provide, via the aerial image display, a floating image including the information related to the context information.

Description

Electronic device for providing a floating image and a method for controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more specifically, to an electronic device and a method for controlling the same for providing a floating image containing various information in conjunction with an external device.

[0002] Electronic devices including various types of displays are being developed. In particular, electronic devices are being developed that enable viewing floating images from mid-air (aerial) at a certain distance from the display panel.

[0003] In addition, with the recent increase in demand for AR (augmented reality) and VR (virtual reality) markets and growing interest in public health, there is an increasing demand for electronic devices capable of contactless interaction.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] According to one embodiment of the present disclosure, an electronic device comprises: an aerial imaging display (AID) including an image module that forms an original image to be provided in the air and transmits light for said original image, and a floating module that forms a real image of said original image at a predetermined location in the air through the light for said original image and provides a floating image; a communication interface; a camera; a memory that stores instructions; and at least one processor. When the instructions are executed individually or collectively by said processor, the electronic device acquires an image through said camera, and when a user is recognized through said image, receives context information of said external device from said external device through said communication interface, acquires information related to the context information of said external device, and provides a floating image including the information related to said context information through said aerial imaging display.

[0006] The above context information includes state information of the external device, and when the instructions are executed individually or collectively by the at least one processor, the electronic device may acquire information about a greeting screen when the external device is identified as being in an off state based on the state information of the external device, acquire private information about the user when the external device is identified as being in a state of displaying a daily board based on the state information of the external device, acquire information about the art image when the external device is identified as being in a state of displaying an art image based on the state information of the external device, and acquire information about an avatar UI when the external device is identified as being in a state of displaying content based on the state information of the external device.

[0007] The above context information includes information about content provided by the external device, and when the instructions are executed individually or collectively by the at least one processor, the electronic device may provide a floating image including an avatar UI through the aerial image display and, based on the information about the content received from the external device, provide (i) conversation affordances on the avatar UI or (ii) provide emotional expressions on the avatar UI.

[0008] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may provide a floating image including a content list containing a plurality of contents through the aerial image display, scroll the content list included in the floating image when a gesture for scrolling the content list is detected, and provide a floating image including the selected content through the aerial image display when a gesture for selecting one of the plurality of contents included in the content list is detected.

[0009] The above context information includes notification information, and when the above instructions are executed individually or collectively by the at least one processor, the electronic device may provide a floating image including a notification icon corresponding to the notification information, and when it is identified that the user is gazing at the aerial image display through the camera, the floating image including the notification information may be provided.

[0010] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may request user information from a wearable device worn by the user through the communication interface when the user is recognized through the image, and authenticate the user based on the user information received from the wearable device.

[0011] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may identify whether the UI element has an interaction when a gesture of selecting a UI element included in the floating image is detected after the user is authenticated, and when the UI element is identified as having an interaction, generate information about a vibration pattern and transmit the information about the vibration pattern to the wearable device.

[0012] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may identify whether the external device is placed at a preset location when the external device is recognized as being provided to the electronic device using an image acquired through the camera, provide a floating image containing augmentation information around the external device when the external device is identified as being placed at the preset location, and provide a floating image containing affordances for adjusting the position of the external device around the external device when the external device is identified as not being placed at the preset location.

[0013] When the above instructions are executed individually or collectively by the processor, the at least one electronic device may update the augmentation information based on the gesture when a gesture for controlling the augmentation information included in the floating image is detected, and transmit information about the updated augmentation information to the external device through the communication interface.

[0014] When the above instructions are executed individually or collectively by the processor, the at least one electronic device may update the augmentation information based on the received information when information for updating augmentation information is received from the external device.

[0015] Meanwhile, a control method for an electronic device including an aerial imaging display (AID) and a camera, according to one embodiment of the present disclosure, which forms an original image to be provided in the air, transmits light for the original image, and forms a real image of the original image at a predetermined location in the air through the light for the original image to provide a floating image, comprises: a step of acquiring an image through the camera; a step of receiving context information of the external device from an external device when a user is recognized through the image; a step of acquiring information related to the context information of the external device; and a step of providing a floating image including information related to the context information through the aerial imaging display.

[0016] The above context information includes state information of the external device, and the step of obtaining information related to the context information is to obtain information about a greeting screen if the external device is identified as being in an off state based on the state information of the external device, to obtain information about the user's private information if the external device is identified as being in a state of displaying a daily board based on the state information of the external device, to obtain information about the art image if the external device is identified as being in a state of displaying an art image based on the state information of the external device, and to obtain information about an avatar UI if the external device is identified as being in a state of displaying content based on the state information of the external device.

[0017] The above context information includes information about content provided by the external device, and the step of providing the floating image includes providing a floating image including an avatar UI, and the control method may include the step of (i) providing a conversation affordance on the avatar UI or (ii) providing an emotion expression on the avatar UI based on information about the content received from the external device.

[0018] The step of providing the floating image may include providing a floating image including a content list containing a plurality of contents through the aerial image display, and the control method may include: a step of scrolling the content list included in the floating image when a gesture for scrolling the content list is detected; and a step of providing a floating image including the selected content through the aerial image display when a gesture for selecting one of the plurality of contents included in the content list is detected.

[0019] The above context information includes notification information, and the step of providing the floating image may include: providing a floating image including a notification icon corresponding to the notification information; and, when it is identified that the user is gazing at the aerial image display through the camera, providing a floating image including the notification information.

[0020] The above control method may further include the step of requesting user information from a wearable device worn by the user when the user is recognized through the image; and the step of authenticating the user based on the user information received from the wearable device.

[0021] The above control method may further include: a step of identifying whether the UI element has an interaction when a gesture of selecting a UI element included in the floating image is detected after the user is authenticated; and a step of generating information about a vibration pattern and transmitting the information about the vibration pattern to the wearable device when it is identified that the UI element has an interaction.

[0022] The above control method further includes the step of identifying whether the external device is placed at a preset location when the external device is recognized as being provided to the electronic device using an image acquired through the camera; and the step of providing the floating image may provide a floating image containing augmentation information around the external device when the external device is identified as being placed at the preset location, and provide a floating image containing affordances for adjusting the position of the external device around the external device when the external device is identified as not being placed at the preset location.

[0023] The above control method may further include: a step of updating the augmentation information based on the gesture when a gesture for controlling the augmentation information included in the floating image is detected; and a step of transmitting information about the updated augmentation information to the external device.

[0024] The above control method may further include the step of updating the augmentation information based on the received information when information for updating the augmentation information is received from the external device.

[0025] The above-described aspects and other aspects, features, and advantages regarding specific embodiments of the present invention will become more apparent from the following description together with the accompanying drawings.

[0026] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0027] FIG. 1 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0028] FIGS. 2a and 2b are drawings for illustrating an aerial image display for providing a floating image according to one embodiment of the present disclosure.

[0029] FIG. 3 is a drawing for explaining a camera according to one embodiment of the present disclosure,

[0030] FIG. 4 is a sequence diagram for explaining an embodiment interacting with a wearable device according to one embodiment of the present disclosure,

[0031] FIG. 5 is a sequence diagram, and FIG. 6a, FIG. 6b, FIG. 6c, and FIG. 6d are drawings for explaining an embodiment of providing a floating image in conjunction with an external device according to an embodiment of the present disclosure.

[0032] FIG. 7 is a flowchart, and FIG. 8a, FIG. 8b, FIG. 8c, and FIG. 8d are drawings for explaining an embodiment of controlling an avatar UI according to content provided by an external device, according to one embodiment of the present disclosure.

[0033] FIG. 9 is a flowchart, and FIG. 10 is a drawing for explaining an embodiment that provides a floating image including content by providing a content list according to one embodiment of the present disclosure.

[0034] FIG. 11 is a flowchart, and FIG. 12a and FIG. 12b are drawings for illustrating an embodiment providing notification information received from an external device according to an embodiment of the present disclosure.

[0035] FIG. 13 is a flowchart, FIG. 14a and FIG. 14b are drawings for illustrating an embodiment of providing augmentation information when an external device is located on an electronic device according to one embodiment of the present disclosure, and,

[0036] FIG. 15 is a flowchart illustrating a control method for an electronic device providing a floating image according to one embodiment of the present disclosure.

[0037] The various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments.

[0038] In relation to the description of the drawings, similar reference numerals may be used for similar or related components.

[0039] The singular form of the noun corresponding to an item may include one or plural items, unless the relevant context clearly indicates otherwise.

[0040] In the present disclosure, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) only A, (2) only B, or (3) both A and B.

[0041] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in other aspects (e.g., importance or order).

[0042] Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that the component may be connected to the other component directly (e.g., via a wire), wirelessly, or through a third component.

[0043] Terms such as "include" or "have" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in this document, and do not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0044] When it is said that a component is "connected," "combined," "supported," or "in contact" with another component, this includes not only cases where the components are directly connected, combined, supported, or in contact, but also cases where they are indirectly connected, combined, supported, or in contact through a third component.

[0045] When it is said that a component is located "on" another component, this includes not only cases where one component is in contact with the other, but also cases where another component exists between the two components.

[0046] The term "and / or" includes a combination of multiple related described components or any of the multiple related described components.

[0047] In some situations, the expression “device configured to do something” may mean that the device is “capable of doing something” in conjunction with other devices or components. For example, the phrase “processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing the said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or application processor) capable of performing said operations by executing one or more software programs stored in a memory device.

[0048] In the embodiments, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor (not shown), except for a "module" or "part" that needs to be implemented in specific hardware.

[0049] Meanwhile, various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0050] An embodiment of the present disclosure will be described in more detail below with reference to the attached drawings.

[0051] FIG. 1 is a block diagram showing the configuration of an electronic device according to the present embodiment of the present disclosure. An electronic device (100) according to various embodiments of the present disclosure may include, as shown in FIG. 1, an aerial image display (110), a communication interface (120), a camera (130), a memory (140), and at least one processor (150) (hereinafter referred to as the processor (150)). An electronic device (100) according to one embodiment of the present disclosure may be an electronic device including a table-type display for providing a floating image, but is not limited thereto. Furthermore, the configuration shown in FIG. 1 is merely one embodiment, and it is understood that various components and configurations may be added or removed depending on the type of electronic device (100).

[0052] An aerial imaging display (AID) (110) can provide a floating image by forming a real image at a predetermined position in the air. Here, the floating image may represent an image that appears to be floating in the air within three-dimensional space without being fixed to a physical screen or background, and the floating image may be referred to by various terms such as a holographic image, a three-dimensional image, a virtual image, an aerial image, etc. In one or more embodiments, the aerial imaging display (110) may be implemented as a tabletop display as shown in FIG. 2a. The aerial imaging display (110) may be referred to by various terms such as a floating image display, a holographic display, etc.

[0053] In particular, referring to the aerial image display (110) of FIG. 2b, a dihedral corner reflector array (DCRA) is used to form a floating image. Light containing a source image is reflected twice from the DCRA and then gathered again on the opposite side of the DCRA, thereby forming a floating image. At this time, the position where the floating image is formed is a position located away from the DCRA by the distance between the DCRA and the source image; in other words, the height (H) at which the floating image is formed is equal to the vertical distance from the DCRA to the source image. Additionally, since the floating image must be floated in the air in a diagonal direction oblique to the DCRA, the image forming unit (3) can be positioned so that the source image is also formed in a diagonal direction symmetric to it.

[0054] An aerial image display (110) may include an image module that forms an original image to be provided in the air and transmits light for the original image, and a floating module that forms a real image of the original image at a predetermined location in the air through light for the original image and provides a floating image.

[0055] Specifically, the imaging module forms an original image to be floated mid-air / aerial and transmits light for the original image to the floating module. The imaging module includes an image forming unit and may further include a collimating optical unit. The floating module is configured to form a real image of the original image at a predetermined position in mid-air using an array of a plurality of corner reflectors. In one or more embodiments, the floating module includes a holographic optical element. Additionally, the floating module may include a waveguide that guides light for the original image and outputs it to another position, a holographic optical element including an angle-selection hologram, and a floating element in which a plurality of corner reflectors are arrayed. Additionally, the floating module may further include magnifying optics.

[0056] Accordingly, the aerial image display (110) can provide a floating image containing various information. For example, the aerial image display (110) can provide a floating image containing an avatar UI. The floating image provided by the present disclosure will be described in detail later through the drawings.

[0057] The communication interface (120) includes at least one circuit and can communicate with various types of external devices or servers. The communication interface (120) may include at least one of a BLE (Bluetooth Low Energy) module, a Wi-Fi communication module, a cellular communication module, a 3G (3rd generation) mobile communication module, a 4G (4th generation) mobile communication module, a 4th generation LTE (Long Term Evolution) communication module, and a 5G (5th generation) mobile communication module.

[0058] In one or more embodiments, the communication interface (120) can transmit and receive various information by performing a communication connection with a wearable device worn by a user or an external device (e.g., TV, laptop, monitor, etc.) using a short-range wireless communication module.

[0059] In one or more embodiments, the communication interface (120) may receive context information of an external device from an external device. Herein, the context information is information describing specific data, state, environment, or usage scenario associated with the external device, and for example, the context information may include identification information of the external device, communication connection information of the external device, user account information of the external device, status information of the external device, information about content provided by the external device, notification information obtained by the external device, etc.

[0060] The camera (130) can capture still images and video. According to various embodiments of the present disclosure, the camera (130) may include one or more lenses, an image sensor, an image signal processor, and a flash. One or more lenses may include a telephoto lens, a wide-angle lens and / or a super wide-angle lens placed on the surface of the electronic device (100), and may also include a three-dimensional depth lens. In one or more embodiments, as shown in FIG. 3, a user sitting on a sofa may be photographed by being placed on an area of ​​the electronic device (100). By doing so, the electronic device (100) can control various functions of the electronic device (100) by detecting the user's gestures (particularly hand gestures) captured by the camera (130). Additionally, the electronic device (100) can obtain information about the user's gaze by analyzing the user's gaze through the image captured by the camera (130). The camera (130) may be placed on one area of ​​the electronic device (100), but is not limited to this configuration, and various embodiments according to the present disclosure may be implemented through connection with a camera separately existing outside the electronic device (100).

[0061] Additionally, the electronic device (100) can obtain information about a user or object by receiving an image captured by a camera (310) located on an external device (300).

[0062] The memory (140) can store an operating system (OS) for controlling the overall operation of the components of the electronic device (100) and instructions or data related to the components of the electronic device (100). In particular, the memory (140) may include a plurality of modules for providing a floating image (e.g., a plurality of modules shown in FIG. 2). In particular, when the function for providing a floating image is executed, the electronic device (100) can load data for various modules stored in non-volatile memory to perform various operations into volatile memory. Here, loading means the operation of bringing data stored in non-volatile memory into volatile memory and storing it so that the processor (150) can access it.

[0063] Memory (140) can be implemented as non-volatile memory (e.g., hard disk, SSD (Solid state drive), flash memory), volatile memory (memory within the processor (150)), etc.

[0064] The processor (150) can control the electronic device (100) according to at least one instruction stored in memory (140).

[0065] In particular, at least one processor (150) may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator. One or more processors (150) may control one or any combination of other components of an electronic device and may perform operations or data processing related to communication. One or more processors (150) may execute one or more programs or instructions stored in memory (140). For example, one or more processors (150) may perform a method according to one embodiment of the present disclosure by executing one or more instructions stored in memory (140).

[0066] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. That is, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0067] One or more processors (150) may be implemented as a single-core processor including one core, or as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When one or more processors (150) are implemented as multicore processors, each of the multiple cores included in the multicore processor may include internal memory such as cache memory or on-chip memory, and a common cache shared by multiple cores may be included in the multicore processor. Additionally, each of the multiple cores included in the multicore processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one embodiment of the present disclosure.

[0068] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0069] In embodiments of the present disclosure, the processor (150) may mean a system-on-chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator or machine learning accelerator, etc., but the embodiments of the present disclosure are not limited thereto.

[0070] In particular, the processor (190) acquires an image through the camera (130) by executing at least one instruction, and when a user is recognized through the image, receives context information of an external device from an external device through the communication interface (120), acquires information related to the received context information of the external device, and provides a floating image including the information acquired through the aerial image display (110).

[0071] In one or more embodiments, the context information may include state information of an external device. Here, the processor (150) may obtain information about a greeting screen if the external device is identified as being off based on the state information of the external device, obtain private information of a user if the external device is identified as being in a state of displaying a daily board based on the state information of the external device, obtain information about an art image if the external device is identified as being in a state of displaying an art image based on the state information of the external device, and obtain information about an avatar UI if the external device is identified as being in a state of displaying content based on the state information of the external device.

[0072] In one or more embodiments, the context information may include information about content provided by an external device. Here, the processor (150) provides a floating image including an avatar UI through an aerial image display (110), and may provide conversation affordances on the avatar UI or provide emotional expressions using the avatar UI based on information about content received from an external device.

[0073] In one or more embodiments, the processor (150) provides a floating image containing a list of contents through an aerial image display (110), scrolls the list of contents contained in the floating image when a gesture for scrolling the list of contents is detected, and when a gesture for selecting one of a plurality of contents contained in the list of contents is detected, provides a floating image containing the selected content through the aerial image display (110).

[0074] In one or more embodiments, the context information may include notification information. Here, the processor (150) may provide a floating image including a notification icon corresponding to the notification information, and when it is identified that a user is looking at the aerial image display (110) through the camera (130), the floating image including the notification information may be provided.

[0075] In one or more embodiments, when a user is recognized through an image, the processor (150) may request user information from a wearable device worn by the user through a communication interface (120) and authenticate the user based on the user information received from the wearable device.

[0076] In one or more embodiments, the processor (150) identifies whether the UI element is an interactive UI element when a gesture of selecting a UI element included in the floating image is detected after the user is authenticated, and if the UI element is identified as an interactive UI element, it can generate information about a vibration pattern and transmit it to a wearable device.

[0077] In one or more embodiments, the processor (150) can identify whether the external device is placed at a preset location when it is recognized that the external device is located on the electronic device (100) using an image captured through the camera (130), and if it is identified that the external device is placed at a preset location, it can provide a floating image containing augmentation information around the external device, and if it is identified that the external device is not placed at a preset location, it can provide a floating image containing affordances around the external device for adjusting the position of the external device.

[0078] In one or more embodiments, when a gesture for controlling augmented information included in a floating image is detected, the processor (150) can update the augmented information based on the gesture and transmit information about the updated augmented information to an external device through a communication interface.

[0079] In one or more embodiments, when the processor (150) receives information for updating augmentation information from an external device, it can update augmentation information based on the received information.

[0080] Of course, in addition to the configuration shown in FIG. 1, the electronic device (100) may have various additional configurations such as a microphone for receiving user voice, a plurality of sensors for detecting the external environment, and an input / output interface.

[0081] FIG. 4 is a sequence diagram for explaining an embodiment that interacts with a wearable device according to one embodiment of the present disclosure. Here, the wearable device (400) is a device worn by a user and may be a smart watch. However, the embodiment of the present disclosure is not limited thereto and may be implemented with various wearable devices such as smart rings, smart glasses, etc.

[0082] The electronic device (100) can acquire an image (410). Here, the electronic device (100) can acquire an image through the camera (130) of the electronic device (100), but this is merely one embodiment, and the electronic device (100) can acquire an image taken by an external device (300) through a communication interface (120).

[0083] The electronic device (100) can detect a user through an acquired image (420). In particular, the electronic device (100) can analyze the image to detect whether the user is located at a preset location (e.g., a sofa located around the electronic device (100)).

[0084] The electronic device (100) can request user information from the wearable device (400) (430). Here, the electronic device (100) can request user information by broadcasting. However, this is merely one embodiment, and the electronic device (100) can request user information using a pre-established communication method. Here, the user information may be user account information, but this is merely one embodiment, and may include various information such as user image information.

[0085] The wearable device (400) can transmit user information to the electronic device (100) (440).

[0086] The electronic device (100) can recognize a user (450). In one or more embodiments, the electronic device (100) can identify whether the user included in the image is a registered user based on user information. That is, the electronic device (100) can recognize a user if the user account information received from the wearable device (400) matches the account information of a registered user.

[0087] The electronic device (100) can recognize hand proximity (460). Here, hand proximity may mean that the user's hand moves toward the electronic device (100) and is located within a preset distance from the electronic device (100). In one or more embodiments, the electronic device (100) can recognize the user's hand proximity by analyzing an image captured through a camera (130). In one or more embodiments, the electronic device (100) can recognize hand proximity based on a sensing value received from a wearable device (400). When hand proximity is recognized, the electronic device (100) can provide a floating image containing UI elements. However, this is merely one embodiment, and the electronic device (100) may first provide a floating image containing UI elements before hand proximity is recognized, and after hand proximity is recognized, provide a cursor, etc., for selecting UI elements.

[0088] The electronic device (100) can detect a gesture for selecting a UI element (470). In one or more embodiments, the electronic device (100) can detect a gesture for selecting a UI element when a preset gesture (e.g., a double tap gesture or a fist clenching gesture) is detected while the UI element is highlighted. Here, the electronic device (100) can detect the gesture by analyzing an image captured through a camera (130). However, this is only one embodiment, and the electronic device (100) can detect the gesture based on a sensing value received from a wearable device (400).

[0089] In one or more embodiments, the electronic device (100) can perform a function corresponding to a UI element. For example, if the UI element is an item corresponding to content, the electronic device (100) can provide a floating image containing content corresponding to the UI element.

[0090] The electronic device (100) can generate a vibration pattern based on a UI element (480). In one or more embodiments, the electronic device (100) can generate a vibration pattern based on the type of UI element. For example, if the UI element is an object that can be used, such as a button or an avatar UI, the electronic device (100) can generate a vibration pattern corresponding to the UI element. In one embodiment, the electronic device (100) can generate different types of vibration patterns depending on the UI element. For example, the electronic device (100) can generate a first vibration pattern in the case of a first type of UI element, and can generate a second vibration pattern different from the first vibration pattern in the case of a second type of UI element. Here, the vibration pattern may include various information such as vibration intensity, vibration time, vibration period, etc.

[0091] One or more electronic devices (100) may provide a vibration pattern to correspond to an emotional expression (e.g., expression of surprise or liking) provided by the avatar UI through a touch gesture, such as in the case of an avatar UI capable of emotional interaction.

[0092] The electronic device (100) can transmit information about the generated vibration pattern to the wearable device (400) (490).

[0093] The wearable device (400) can provide a vibration pattern based on information about the vibration pattern received from the electronic device (100) (495).

[0094] Accordingly, the electronic device (100) can recognize the user through the wearable device (400) that the user is wearing, and the user can receive feedback based on interaction with the electronic device (100) based on vibration patterns, thereby enabling more vivid and intuitive control of the electronic device (100).

[0095] FIG. 5 is a sequence diagram illustrating an embodiment of providing a floating image in conjunction with an external device according to one embodiment of the present disclosure. Here, the external device (300) is a device located outside the electronic device (100) that can provide content, and may be, for example, a TV, but is not limited thereto, and may be implemented as a user terminal device such as a laptop PC, desktop PC, tablet PC, etc., may be implemented as a home appliance such as a refrigerator, washing machine, etc., and may be implemented as a device such as an AI speaker, etc.

[0096] The electronic device (100) can acquire an image (510). Here, the electronic device (100) can acquire an image through the camera (130) of the electronic device (100), but this is merely one embodiment, and the electronic device (100) can acquire an image taken by an external device (300) through a communication interface (120).

[0097] The electronic device (100) can detect a user through an acquired image (520). In particular, the electronic device (100) can analyze the image to detect whether the user is located at a preset location (e.g., a sofa located around the electronic device (100)).

[0098] The electronic device (100) may request status information of the external device (300) from the external device (300) (530). Here, the status information of the external device (300) may include the current operating status of the external device, the screen status displayed by the external device, the setting status of the external device, etc., but is not limited thereto.

[0099] The external device (300) can transmit status information of the external device (300) to the electronic device (100) in response to a request (540).

[0100] The electronic device (100) can obtain information related to the status information of the external device (300) based on the status information of the external device (300) (550). In one or more embodiments, if the external device (300) is identified as being in an off state based on the status information of the external device (300), the electronic device (100) can obtain information regarding a greeting screen. Here, the greeting screen is an initial interface provided to the user when the electronic device (100) is activated or booted, and may include information for guiding the initial setup of the electronic device (100), information for guiding basic information of the electronic device (100), or previous usage information.

[0101] In one or more embodiments, if the external device (300) is identified as being in a state where it displays a daily board based on the status information of the external device (300), the electronic device (100) may obtain the user's private information. Here, the term "daily board" refers to a personalized dashboard of the user and may include information that the user frequently checks in daily life. The user's private information may include the user's schedule information, the user's preference information, the user's setting information, etc.

[0102] In one or more embodiments, if the external device (300) is identified as being in a state where it displays an art image based on the state information of the external device (300), the electronic device (100) can obtain information about the art image. Here, the state where the external device (300) displays an art image may be called an art mode, a gallery mode, a screen saver mode, etc. Additionally, the information about the art image may include the title of the art image, the artist of the art image, detailed information about the art image, etc.

[0103] In one or more embodiments, if the external device (300) is identified as being in a state where it displays content based on the state information of the external device (300), the electronic device (100) can obtain information about the avatar UI. Here, the avatar UI is a UI element capable of interacting with the user and may be called a character UI, a virtual agent UI, a persona UI, etc. While viewing content through the avatar UI, the user can interact with the avatar UI to receive various user experiences. Meanwhile, the electronic device (100) can obtain information about the content currently being displayed, as well as information about the avatar UI. For example, various information such as the title, characters, director, playback time, and plot of the content currently being displayed can be obtained.

[0104] The electronic device (100) may provide a floating image containing relevant information (560). In one or more embodiments, if the external device (300) is identified as being off based on the status information of the external device (300), the electronic device (100) may provide a floating image (610) including a greeting screen to guide the user's previous usage information and the function to be performed now, as shown in FIG. 6a.

[0105] In one or more embodiments, if the external device (300) is identified as being in a state where it displays a daily board based on the state information of the external device (300), the electronic device (100) may provide a floating image (620) including user schedule information and message information, etc., as shown in FIG. 6d.

[0106] In one or more embodiments, if the external device (300) is identified as being in a state where it displays an art image based on the state information of the external device (300), the electronic device (100) may provide a floating image (630) including information and details about the artist of the currently displayed art image, as illustrated in FIG. 6c.

[0107] In one or more embodiments, if the external device (300) is identified as being in a state where it displays content based on the state information of the external device (300), the electronic device (100) may provide a floating image (640) including an avatar UI that can interact with the user, as shown in FIG. 6d.

[0108] In the above-described embodiment, it was explained that when the external device (300) displays content, it displays an avatar UI. However, this is merely one embodiment, and the electronic device (100) may provide a floating image containing an avatar UI when it receives a preset trigger voice. For example, when a preset trigger voice, such as "Elijah," is detected, the electronic device (100) may provide a floating image (640) containing an avatar UI that allows interaction with the user, as illustrated in FIG. 6d. While the avatar UI is displayed, the electronic device (100) may provide various functions of the electronic device (100) through the avatar UI. For example, the electronic device (100) may change the currently displayed content or art image through the user's voice or gesture, and the avatar UI may provide information about the changed content or art image. By doing so, the user can control various functions of the electronic device (100) while performing interaction with a virtual avatar, thereby providing a new user environment.

[0109] FIG. 7 is a flowchart, and FIGS. 8a through 8d are drawings for explaining an embodiment of controlling an avatar UI according to content provided by an external device, according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the external device (300) may be in a state of providing content.

[0110] The electronic device (100) can receive content information (710). Here, the content information may include identification information of the content, information about the current screen that the content is displaying, and a captured image of the screen currently being displayed.

[0111] The electronic device (100) can determine the type of content (or screen) based on content information (720). Here, the type of content (or screen) may include a first type and a second type, and the content (or screen) of the first type may be content (or screen) capable of conversing with the avatar UI. The content (or screen) of the second type may be content (or screen) for providing emotional expression.

[0112] When identified as content or screen of the first type (730-Y), the electronic device (100) can provide a floating image containing conversation affordances using an avatar UI (740). Here, conversation affordances are UI elements intended to guide a user to intuitively understand a conversation situation by looking at interface elements, and may be referred to by various terms such as conversation-inducing UI elements, conversation guide UI elements, etc.

[0113] In one embodiment, as illustrated in FIG. 8a, while an external device (300) is displaying content, an electronic device (100) may provide a floating image including an avatar UI (810) of a first brightness. At this time, when the content (or screen) displayed by the external device (300) is recognized as a first type of content (or screen), the electronic device (100) may provide a conversation affordance that changes the brightness of the avatar UI (810) from a first brightness to a second brightness, as illustrated in FIG. 8b. That is, the electronic device (100) may provide a conversation affordance that induces the user to participate in a conversation by gradually activating the avatar UI (810). By doing so, the user may start a conversation by uttering a user voice (820), such as "Elijah, why," as illustrated in FIG. 8b.

[0114] And, the electronic device (100) can perform a conversation (830) with the user using the activated avatar UI (810) as illustrated in FIG. 8c. For example, the electronic device (100) can provide a query through the avatar UI (810) such as "Who do you think Gayoung will win?" and when a response such as "I am the Chinese Goddess Chef" is received from the user, the electronic device (100) can provide a response such as "Me too~ You are a very famous person!". The user can perform a conversation (830) with the electronic device (100) as illustrated in FIG. 8c by recognizing the conversation affordances as illustrated in FIG. 8a and FIG. 8b.

[0115] If it is identified that the content or screen is not of the first type (730-N) (i.e., if the content or screen of the second type is identified), the electronic device (100) may provide a floating image containing an emotional expression on the avatar UI (740). Here, the electronic device (100) may use the avatar UI to express an emotion corresponding to the current content or screen. For example, if the currently displayed content screen is a quiz show screen, the electronic device (100) may use the avatar UI (840) to provide a curious emotional expression, as shown in FIG. 8d. As another example, if the currently displayed content screen is a sad screen, the electronic device (100) may use the avatar UI to express a sad emotion.

[0116] FIG. 9 is a flowchart, and FIG. 10 is a drawing for explaining an embodiment that provides a floating image including content by providing a content list according to one embodiment of the present disclosure.

[0117] The electronic device (100) can detect user commands related to content exploration (910). Here, user commands related to content exploration are user commands for generating a content list, and can be obtained through gestures, user voice, etc.

[0118] The electronic device (100) may provide a floating image containing a list of contents (920). For example, the electronic device (100) may provide a floating image containing a list of contents (1010) containing a plurality of UI elements corresponding to a plurality of contents, as shown in FIG. 10. Here, when the user raises their hand, the electronic device (100) may provide a highlight (or cursor) on the list of contents.

[0119] The electronic device (100) can detect a gesture for scrolling (930). In one or more embodiments, the electronic device (100) can detect a gesture for scrolling based on a sensing value received from a camera (130) or a wearable device (400). Here, the gesture for scrolling may include a gesture for moving left and right or a gesture for moving up and down.

[0120] The electronic device (100) can scroll the content list in response to a gesture for scrolling (940). For example, the electronic device (100) can scroll the content list to the right when a gesture for scrolling to the right is detected. Here, if it moves more than a preset value to the right, the electronic device (100) can move the UI elements of the content list one step at a time.

[0121] The electronic device (100) can detect a gesture for selecting one of a plurality of contents included in a content list (950). Here, the gesture for selecting the content may be a gesture of pressing in the Z-axis direction while a highlight or focus is placed on one of the plurality of UI elements, but is not limited thereto. For example, a gesture for selecting the content may be detected while a highlight is placed on a UI element corresponding to a third content in the content list.

[0122] The electronic device (100) can provide a floating image containing selected content (960). For example, when a gesture to select content is detected while a highlight is positioned on a UI element corresponding to a third content in a list of content, the electronic device (100) can provide a floating image containing the selected third content.

[0123] According to one embodiment of the present disclosure, an electronic device (100) can provide an emotional expression of the avatar UI according to a user gesture input on the avatar UI while the avatar UI is displayed. For example, when the avatar UI is looking back and the avatar UI is touched, the electronic device (100) can provide an avatar UI with a surprised emotion. As another example, when the head of the avatar UI is stroked, the electronic device (100) can provide an avatar UI with a happy emotion.

[0124] According to one embodiment of the present disclosure, an electronic device (100) may provide a floating image including notification information received from an external device.

[0125] FIG. 11 is a flowchart, and FIG. 12a and FIG. 12b are drawings for explaining an embodiment of providing notification information received from an external device according to one embodiment of the present disclosure.

[0126] First, the electronic device (100) can receive notification information (1110). Here, the notification information is notification information received by an external device (300), and may be various notification information such as message reception, email reception, phone reception, schedule alarm, application alarm, etc.

[0127] The electronic device (100) can generate a notification icon (1120). Here, the notification icon is an icon for guiding that notification information has been received. As shown in FIG. 12a, it may be a geometric icon (1210), but this is merely one embodiment and can be implemented in various forms such as text. Additionally, the electronic device (100) can provide voice guidance through a speaker along with the notification icon that notification information has been received. Here, the notification icon may be displayed for a preset period, but this is merely one embodiment, and it is obvious that it may continue to be displayed until a user is recognized or user input is received. Additionally, after the preset period has passed, the notification icon may be displayed with adjusted color or brightness.

[0128] The electronic device (100) can recognize a user (1130). Specifically, the electronic device (100) can identify whether a user currently located near the electronic device (100) is a registered user. For example, the electronic device (100) can identify whether a user is a registered user by analyzing an image captured through a camera (130). For another example, the electronic device (100) can identify whether a user is a registered user based on user account information received from a wearable device (400) worn by the user. However, this is merely one embodiment, and it is obvious that the user can be recognized by various methods (e.g., fingerprint, iris, password, gaze, etc.).

[0129] The electronic device (100) may provide a floating image containing notification information (1140). For example, the electronic device (100) may provide notification information to guide "baby waking up." And, as shown in FIG. 12b, when a user voice (1220) such as "show me" is input, the electronic device (100) may provide a floating image containing a baby cam screen (1230) corresponding to the notification information.

[0130] By doing so, the user can check notification information (or a screen corresponding to the notification information) using the electronic device (100) without interfering with the viewing of content displayed by the external device (300).

[0131] According to one embodiment of the present disclosure, an electronic device (100) can display an image on an external device (300) or on the electronic device (100) based on a gesture.

[0132] In one or more embodiments, when a preset gesture toward an external device (300) is detected while the electronic device (100) is providing a floating image containing content, the electronic device (100) transmits information about the content currently displayed by the electronic device (100) to the external device (300), and the external device (300) can provide content using the information about the content currently displayed by the electronic device (100).

[0133] In one or more embodiments, when a user’s gaze is directed toward an external device (300) while the electronic device (100) provides a floating image containing content, the electronic device (100) transmits information about the content currently displayed by the electronic device (100) to the external device (300), and the external device (300) can provide content using the information about the content currently displayed by the electronic device (100). That is, the electronic device (100) can detect the user’s gaze or the direction of the user’s head and provide content on the device that the user is looking at, either the electronic device (100) or the external device (300).

[0134] In one or more embodiments, the electronic device (100) can change or exchange the location of information displayed by the electronic device (100) and information displayed by the external device (300) through a user gesture or user voice.

[0135] According to one embodiment of the present disclosure, an electronic device (100) may recognize an external device (300) located on the electronic device (100) and provide augmented information. Here, augmented information may be information related to the external device (300) that is augmented and provided around the external device (300).

[0136] FIG. 13 is a flowchart, and FIG. 14a and FIG. 14b are drawings for illustrating an embodiment of providing augmented information when an external device is located on an electronic device according to one embodiment of the present disclosure.

[0137] The electronic device (100) can acquire an image (1310). Here, the electronic device (100) can acquire an image through a camera (130).

[0138] The electronic device (100) can recognize that an external device (300) is located on the electronic device (100) (1320). That is, if the electronic device (100) is a table-type device, the electronic device (100) can identify whether an external device (300) is placed on the electronic device (100).

[0139] The electronic device (100) can identify whether the external device (300) is placed at a preset location (1330). Here, the preset location may be a location corresponding to the central area of ​​the electronic device (100).

[0140] When placed at a preset location (1330-Y), the electronic device (100) can provide a floating image containing augmentation information (1340). Here, the augmentation information is information related to an external device (300) and may include information about the external device (300), information about a user logged into the external device (300), alarm information, etc.

[0141] In one embodiment, as illustrated in FIG. 14a, when the external device (300) is a notebook PC, the electronic device (100) may include a first augmentation information (1410) corresponding to schedule information received from the external device (300), a second augmentation information (1420) corresponding to a map application received from the external device (300), and a third augmentation information (1430) corresponding to a mail application received from the external device (300). Additionally, when a user terminal (1400) is detected, the electronic device (100) may provide a fourth augmentation information (1440) corresponding to an application icon provided by the user terminal (1400) around the user terminal (1400).

[0142] In another embodiment, as illustrated in FIG. 14b, when the external device (300) is a monitor, the electronic device (100) may provide a fifth augmented information (1450) corresponding to a calendar application received from the external device (300), a sixth augmented information (1460) corresponding to a schedule application, and a seventh augmented information (1470) corresponding to a clock application.

[0143] When a user gesture is input on the augmented information illustrated in FIG. 14a or FIG. 14b, the electronic device (100) can update the augmented information on which the gesture was input based on the user gesture. Additionally, when the augmented information is updated, the electronic device (100) can transmit augmented information corresponding to the update to an external device (300), and the external device (300) can update the stored augmented information.

[0144] If the external device (300) is not placed in a preset position (1330-N), the electronic device (100) may provide a floating image including affordances for adjusting the position of the external device (300) (1350). For example, if the external device (300) is placed to the right rather than in the central area of ​​the electronic device (100), the electronic device (100) may provide a floating image including affordances for guiding the external device (300) to move to the left.

[0145] As illustrated in FIG. 14a in one or more embodiments, when a user cup is placed on the electronic device (100), the electronic device (100) can detect the amount of water in the cup through a captured image and calculate the amount of water the user is drinking. By doing so, the electronic device (100) can update the user's health information.

[0146] In one or more embodiments, the electronic device (100) can transmit and receive information with an external device (300) or a wearable device (400) worn by a user to share information of each device (e.g., charging information, schedule information, etc.). At this time, when the information of the electronic device (100), the external device (300), or the wearable device (400) is updated, the electronic device (100), the external device (300), and the wearable device (400) can share the updated information.

[0147] FIG. 15 is a flowchart illustrating a control method for an electronic device providing a floating image according to one embodiment of the present disclosure.

[0148] The electronic device (100) captures an image (1510) through a camera (130). Here, the image may include a user.

[0149] The electronic device (100) recognizes the user through an image (1520). That is, the electronic device (100) can recognize the user by identifying whether the user included in the image is a registered user.

[0150] The electronic device (100) can receive context information of the external device (300) from the external device (300) (1530). Here, the context information is information describing specific data, state, environment, or usage scenario associated with the external device (300). For example, the context information may include identification information of the external device (300), communication connection information of the external device (300), user account information of the external device (300), state information of the external device (300), information about content provided by the external device (300), notification information obtained by the external device (300), etc.

[0151] The electronic device (100) obtains information related to context information (1540). In one or more embodiments, the context information may include state information of an external device (300). Based on the state information of the external device (300), the electronic device (100) may obtain information about a greeting screen if it identifies that the external device (300) is in an off state. Based on the state information of the external device (300), the electronic device (100) may obtain information about a user's private information if it identifies that the external device (300) is in a state of displaying a daily board. Based on the state information of the external device (300), the electronic device (100) may obtain information about an art image if it identifies that the external device (300) is in a state of displaying an art image. Based on the state information of the external device (300), the electronic device (100) may obtain information about an avatar UI if it identifies that the external device (300) is in a state of displaying content.

[0152] The electronic device (100) provides a floating image containing acquired information (1550). In one or more embodiments, the context information may include information about content provided by an external device (300). The electronic device (100) provides a floating image containing an avatar UI through an aerial image display (110), and may provide conversation affordances on the avatar UI or provide emotional expressions using the avatar UI based on information about content received from the external device (300).

[0153] Methods according to various embodiments of the present disclosure may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0154] A method according to various embodiments of the present disclosure may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., a TV) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions.

[0155] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0156] When the above instruction is executed individually or collectively by one or more processors, one or more processors may perform the function corresponding to the above instruction directly or by using other components under the control of said processors. The instruction may include code generated or executed by a compiler or an interpreter.

[0157] Although exemplary embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In an electronic device, An aerial imaging display (AID) comprising an image module that forms an original image to be provided in the air and transmits light for said original image, and a floating module that forms a real image of said original image at a predetermined position in the air through the light for said original image to provide a floating image; Communication interface; camera; Memory for storing instructions; and Includes at least one processor; and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An image is acquired through the above camera, and When a user is recognized through the above image, context information of the external device is received from the external device through the communication interface, and Acquire information related to the context information of the above external device, and An electronic device that provides a floating image containing information related to the context information through the above-described aerial image display.

2. In Paragraph 1, The above context information includes state information of the external device, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, If the external device is identified as being in an off state based on the status information of the external device, information regarding the greeting screen is obtained, and If, based on the status information of the external device, it is identified that the external device is in a state of displaying a daily board, the user's private information is obtained, and If it is identified that the external device is in a state of displaying an art image based on the status information of the external device, information regarding the art image is obtained, and An electronic device that obtains information about an avatar UI when it is identified that the external device is in a state of displaying content based on the state information of the external device.

3. In Paragraph 1, The above context information includes information about the content provided by the external device, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, A floating image including an avatar UI is provided through the above aerial image display, and An electronic device that, based on information about the content received from the external device, (i) provides conversation affordances on the avatar UI or (ii) provides emotional expressions on the avatar UI.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, A floating image including a content list containing multiple contents is provided through the above aerial image display, and When a gesture for scrolling the above content list is detected, the above content list included in the floating image is scrolled, and An electronic device that provides a floating image including the selected content through the aerial image display when a gesture for selecting one of the plurality of contents included in the above content list is detected.

5. In Paragraph 1, The above context information includes notification information, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, A floating image including a notification icon corresponding to the above notification information is provided, and An electronic device that provides a floating image containing the notification information when it is identified that the user is gazing at the aerial image display through the camera.

6. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When the user is recognized through the above image, user information is requested to the wearable device worn by the user through the communication interface, and An electronic device that authenticates the user based on the user information received from the wearable device.

7. In Paragraph 6, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When a gesture selecting a UI element included in the floating image is detected after the user is authenticated, it identifies whether the UI element has an interaction, and An electronic device that, when the above UI element is identified as having an interaction, generates information about a vibration pattern and transmits the information about the vibration pattern to the wearable device.

8. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, If the external device is recognized as being provided to the electronic device using an image acquired through the camera, it identifies whether the external device is placed at a preset location, and When the above external device is identified as being placed at the above preset location, a floating image containing augmentation information is provided around the external device, and An electronic device that provides a floating image including affordances for adjusting the position of the external device around the external device when it is identified that the external device is not placed at the preset position.

9. In Paragraph 8, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When a gesture for controlling the augmented information included in the above floating image is detected, the augmented information is updated based on the gesture, and An electronic device that transmits information about the updated augmented information to the external device through the communication interface.

10. In Paragraph 8, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that updates the augmentation information based on the received information when information for updating augmentation information is received from the above external device.

11. A control method for an electronic device including an aerial imaging display (AID) and a camera, wherein the device forms an original image to be provided in the air, transmits light for said original image, and forms a real image of said original image at a predetermined position in the air through the light for said original image to provide a floating image, wherein Step of acquiring an image through a camera; When a user is recognized through the above image, a step of receiving context information of the external device from the external device; A step of obtaining information related to the context information of the external device; and A control method comprising the step of providing a floating image containing information related to the context information through the above-described aerial image display.

12. In Paragraph 11, The above context information includes state information of the external device, The step of obtaining information related to the above context information is, If the external device is identified as being in an off state based on the status information of the external device, information regarding the greeting screen is obtained, and If, based on the status information of the external device, it is identified that the external device is in a state of displaying a daily board, the user's private information is obtained, and If it is identified that the external device is in a state of displaying an art image based on the status information of the external device, information regarding the art image is obtained, and A control method for obtaining information about an avatar UI when the external device is identified as being in a state of displaying content based on the state information of the external device.

13. In Paragraph 11, The above context information includes information about the content provided by the external device, and The step of providing the above-mentioned floating image is, It provides a floating image containing an avatar UI, and The above control method is, A control method further comprising the step of (i) providing conversation affordances on the avatar UI or (ii) providing emotional expressions on the avatar UI based on information about the content received from the external device.

14. In Paragraph 11, The step of providing the above-mentioned floating image is, A floating image including a content list containing multiple contents is provided through the above aerial image display, and The above control method is, When a gesture for scrolling the above content list is detected, the step of scrolling the content list included in the floating image; and A control method comprising the step of providing a floating image including the selected content through the aerial image display when a gesture for selecting one of the plurality of contents included in the above content list is detected.

15. In Paragraph 11, The above context information includes notification information, The step of providing the above-mentioned floating image is, A step of providing a floating image including a notification icon corresponding to the above notification information; and A control method comprising the step of providing a floating image including notification information when it is identified that the user is gazing at the aerial image display through the camera.

Images