Smart glasses for transmitting information based on object indicated by user to display device, and control method thereof

Abstract

Smart glasses are disclosed. The smart glasses comprise: a memory storing instructions; a camera; a communication circuit; and at least one processor including processing circuitry, wherein the instructions, when executed individually or collectively by the at least one processor, may: acquire images by capturing the real world around the smart glasses via the camera; detect a display device connected via the communication circuit in the images acquired via the camera; acquire first information on an object in the real world on the basis of a gesture of a user in the images acquired via the camera; acquire second information by processing the first information on the object; and transmit, to the display device via the communication circuit, a control signal for allowing the second information to be displayed on the display device.

Description

Smart glasses for transmitting information based on an object directed by a user to a display device and a control method thereof

[0001] The present disclosure relates to smart glasses and a method for controlling the same, and more specifically, to smart glasses and a method for controlling the same that transmit information based on an object indicated by a user to a display device.

[0002] Thanks to the advancement of electronic technology, various types of electronic devices are being developed. In particular, recently, various wearable devices are being developed beyond smartphones.

[0003] In particular, smart glasses can synchronize with the user's gaze, offering different usage methods compared to smartphones.

[0004] According to one embodiment of the present disclosure for achieving the above objectives, smart glasses include at least one processor comprising a memory for storing instructions, a camera, a communication circuit, and a processing circuitry. When the instructions are executed individually or collectively by the at least one processor, the smart glasses may capture images of the real world surrounding the smart glasses through the camera, detect a display device connected via the communication circuitry from the images acquired through the camera, acquire first information about an object in the real world based on a user's gesture from the images acquired through the camera, process the first information about the object to acquire second information, and transmit a control signal to the display device via the communication circuitry to display the second information on the display device.

[0005] Additionally, when the above instructions are executed individually or collectively by the at least one processor, a function to be processed in the smart glasses is determined based on the attributes of the first information regarding the object, and the second information can be obtained by processing the first information based on the determined function.

[0006] And, when the above instructions are executed individually or collectively by the at least one processor, if the first information is not of a preset language type, the first information is translated into the preset language type to obtain the second information, and if the first information is an image or a question, a description of the image or an answer to the question can be obtained as the second information.

[0007] Additionally, when the above instructions are executed individually or collectively by the at least one processor, if interpretation of the first information is impossible, the first information can be copied and obtained as the second information.

[0008] And, when the above instructions are executed individually or collectively by the at least one processor, they can identify information to be obtained from the object in the real world based on the user's gesture in images obtained through the camera, and obtain the first information based on the identified information.

[0009] Additionally, when the above instructions are executed individually or collectively by the at least one processor, if the gesture is a touch on a point on the object, a word corresponding to the touch is identified among the content included in the object; if the gesture is a drag in a first direction on the object, a sentence or phrase corresponding to the beginning to the end of the drag is identified among the content included in the object; and if the gesture is a drag in a second direction on the object, sentences, paragraphs, or images corresponding to the beginning to the end of the drag are identified among the content included in the object.

[0010] And, when the above instructions are executed individually or collectively by the at least one processor, a feedback signal can be transmitted to the display device through the communication circuit to provide different feedback to the display device according to the type of the identified information.

[0011] In addition, when the above instructions are executed individually or collectively by the at least one processor, images can be acquired through the camera when a preset command is received from the display device through the communication circuit while a communication channel is formed with the display device.

[0012] In addition, the memory further stores a plurality of appearance images corresponding to each of the plurality of display devices including the display device, and when the instructions are executed individually or collectively by the at least one processor, the display device having a corresponding appearance image among the plurality of display devices located around the user in the images acquired through the camera can be detected as the display device.

[0013] Additionally, when the above instructions are executed individually or collectively by the at least one processor, while transmitting the second information to the display device based on the control signal, images are acquired by capturing the real world around the smart glasses through the camera, and if it is identified from the images acquired through the camera that the display device displays the second information through the display of the display device, alignment information can be transmitted to the display device through the communication circuit to control the second information to be aligned and displayed according to the content included in the object of the real world.

[0014] And, when the above instructions are executed individually or collectively by the at least one processor, if the first information or the second information includes a musical score, a playback signal to be played based on the first information or the second information can be transmitted to a sound device through the communication circuit.

[0015] Additionally, the display device is a smart watch, and when the instructions are executed individually or collectively by the at least one processor, they can identify the hand of the user wearing the smart watch in images acquired through the camera and acquire the first information based on the position of the user's hand or finger on the object in the real world.

[0016] Meanwhile, according to one embodiment of the present disclosure, a control method for smart glasses may include the steps of: capturing the real world surrounding the smart glasses through a camera included in the smart glasses to obtain images; detecting a display device connected to the smart glasses from the images obtained through the camera; obtaining first information about an object in the real world based on a user's gesture from the images obtained through the camera; processing the first information about the object to obtain second information; and transmitting a control signal to the display device to cause the second information to be displayed on the display device.

[0017] Additionally, the step of acquiring the second information may determine a function to be processed in the smart glasses based on the attributes of the first information regarding the object, and acquire the second information by processing the first information based on the determined function.

[0018] And, in the step of acquiring the second information, if the first information is not of a preset language type, the first information is translated into the preset language type to acquire the second information, and if the first information is an image or a question, a description of the image or an answer to the question can be acquired as the second information.

[0019] In addition, in the step of obtaining the second information, if interpretation of the first information is impossible, the first information may be copied and obtained as the second information.

[0020] And, the step of acquiring the first information may identify information to be acquired from the object in the real world based on the user's gesture in images acquired through the camera, and acquire the first information based on the identified information.

[0021] Additionally, the step of acquiring the first information may identify a word corresponding to the touch among the content contained in the object if the gesture is a touch on a point on the object, identify a sentence or phrase corresponding to the beginning to the end of the drag among the content contained in the object if the gesture is a drag in the first direction on the object, and identify the type of information to be acquired as sentences, paragraphs, or images corresponding to the beginning to the end of the drag among the content contained in the object if the gesture is a drag in the second direction on the object.

[0022] And, it may further include the step of transmitting a feedback signal to the display device to provide feedback that provides different feedback from the display device according to the type of the identified information.

[0023] In addition, the step of acquiring the above images can be performed by acquiring images through the camera when a preset command is received from the display device while a communication channel is formed with the display device.

[0024] In addition, the smart glasses store a plurality of exterior images corresponding to each of the plurality of display devices including the display device, and the detection step can detect a display device having a corresponding exterior image among the plurality of display devices located around the user from the images obtained through the camera as the display device.

[0025] Additionally, while transmitting the second information to the display device based on the control signal, the method may further include the step of acquiring images by capturing the real world around the smart glasses through the camera, and, when it is identified from the images acquired through the camera that the display device displays the second information through the display of the display device, the method may further include the step of transmitting alignment information to the display device to control the second information to be aligned and displayed according to the content included in the object of the real world.

[0026] And, if the first information or the second information includes a musical score, the method may further include the step of transmitting a playback signal to a sound device through the communication circuit to enable the musical score to be played based on the first information or the second information.

[0027] Additionally, the display device is a smart watch, and the step of acquiring the first information may involve identifying the hand of the user wearing the smart watch in images acquired through the camera, and acquiring the first information based on the position of the user's hand or finger on the object in the real world.

[0028] FIG. 1 is a block diagram showing the configuration of an electronic system according to one embodiment of the present disclosure.

[0029] FIG. 2 is a block diagram showing the configuration of smart glasses according to one embodiment of the present disclosure.

[0030] FIG. 3 is a block diagram showing the detailed configuration of smart glasses according to one embodiment of the present disclosure.

[0031] FIGS. 4 to 6 are drawings for explaining the operation according to a user's touch method according to one embodiment of the present disclosure.

[0032] FIG. 7 is a drawing for explaining a processing mode of smart glasses according to one embodiment of the present disclosure.

[0033] FIG. 8 is a drawing for explaining an initialization method according to one embodiment of the present disclosure.

[0034] FIG. 9 is a drawing for explaining a method for determining a function to be processed in smart glasses (100) according to one embodiment of the present disclosure.

[0035] FIG. 10 is a drawing for explaining a method for controlling the display state of a display device according to one embodiment of the present disclosure.

[0036] FIG. 11 is a drawing for explaining an operation according to the type of second information according to one embodiment of the present disclosure.

[0037] FIG. 12 is a drawing for explaining a method of specifying a display device according to one embodiment of the present disclosure.

[0038] FIG. 13 is a drawing for explaining the operation when a plurality of display devices exist according to one embodiment of the present disclosure.

[0039] FIG. 14 is a flowchart illustrating a method for controlling smart glasses according to one embodiment of the present disclosure.

[0040] The purpose of the present disclosure is to provide smart glasses and a method for controlling the same for transmitting information based on an object indicated by a user to a display device.

[0041] The present disclosure will be described in detail below with reference to the attached drawings.

[0042] The terms used in the embodiments of this disclosure have been selected to be as widely used as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section of this disclosure. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0043] In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, operations, or components such as parts) and do not exclude the presence of additional features.

[0044] The expression "at least one of A or / and B" should be understood as representing either "A" or "B" or "A and B".

[0045] Expressions such as "first," "second," "first," or "second" used in this specification may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0046] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "consisting of" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0047] In this specification, the term "user" may refer to a person using smart glasses or a device using smart glasses (e.g., artificial intelligence smart glasses).

[0048] Various embodiments of the present disclosure will be described in more detail below with reference to the attached drawings.

[0049] FIG. 1 is a block diagram showing the configuration of an electronic system (1000) according to one embodiment of the present disclosure. As shown in FIG. 1, the electronic system (1000) may include smart glasses (100) and a display device (200).

[0050] The smart glasses (100) may be a device that captures images by photographing the front of the smart glasses (100) and transmits information based on objects included in the images to a display device (200). For example, the smart glasses (100) may be a device that captures images by photographing the front of the smart glasses (100) and transmits information based on objects indicated by a user's finger in the images to a display device (200).

[0051] The smart glasses (100) may be a device that does not have a function to display text or images, or does not include a component that implements a function to display text or images. Alternatively, the smart glasses (100) may have a function to display text or images, or include a component that implements a function to display text or images, but in this case, the smart glasses (100) may have the function turned off or be set to use an external display device (200) first according to user settings. If the external display device (200) is set to use first, the function of the smart glasses (100) may be disabled.

[0052] The display device (200) is a device that displays information received from smart glasses (100), and may be a smart watch, smartphone, tablet PC, desktop PC, laptop, TV, projector, etc. However, it is not limited thereto, and any device that includes a display and can display information received from smart glasses (100) through the display may be used.

[0053] FIG. 2 is a block diagram showing the configuration of smart glasses (100) according to one embodiment of the present disclosure.

[0054] According to FIG. 2, smart glasses (100) include a memory (110), a camera (120), a communication circuit (130), and a processor (140).

[0055] Memory (110) may refer to hardware that stores information, such as data, in an electrical or magnetic form so that a processor (140), etc. can access it. To this end, memory (110) may be implemented as at least one piece of hardware among non-volatile memory, volatile memory, flash memory, hard disk drive (HDD) or solid-state drive (SSD), RAM, ROM, etc.

[0056] At least one instruction required for the operation of the smart glasses (100) or the processor (140) may be stored in the memory (110). Here, the instruction is a unit of code that directs the operation of the smart glasses (100) or the processor (140), and may be written in machine language, which is a language that a computer can understand. Alternatively, a plurality of instructions that perform a specific task of the smart glasses (100) or the processor (140) may be stored in the memory (110) as an instruction set.

[0057] Data that is information in bit or byte units capable of representing characters, numbers, images, etc. can be stored in the memory (110). For example, images, neural network models, and multiple appearance images corresponding to each of multiple display devices can be stored in the memory (110).

[0058] The memory (110) is accessed by the processor (140), and the processor (140) may perform read / write / modify / delete / update, etc. on instructions, instruction sets, or data.

[0059] The camera (120) is configured to capture still images or video. The camera (120) can capture a still image at a specific point in time, but can also capture a series of still images.

[0060] The camera (120) can capture the front of the smart glasses (100) to capture the actual environment in front of the smart glasses (100). The processor (140) can identify an object indicated by the user from the images obtained through the camera (120).

[0061] The camera (120) includes a lens, a shutter, an aperture, a solid-state image sensor, an AFE (Analog Front End), and a TG (Timing Generator). The shutter controls the time when light reflected from a subject enters the camera (120), and the aperture controls the amount of light incident on the lens by mechanically increasing or decreasing the size of the opening through which light enters. When light reflected from a subject accumulates as photocharge, the solid-state image sensor outputs an image based on the photocharge as an electrical signal. The TG outputs a timing signal for reading out pixel data from the solid-state image sensor, and the AFE samples and digitizes the electrical signal output from the solid-state image sensor.

[0062] The communication circuit (130) is configured to communicate with various types of external devices according to various types of communication methods. For example, smart glasses (100) can communicate with a display device (200), a sound device, etc. through the communication circuit (130).

[0063] The communication circuit (130) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, and a wireless communication module, etc. Here, each communication module may be implemented in the form of at least one hardware chip.

[0064] The Wi-Fi module and Bluetooth module perform communication using the Wi-Fi and Bluetooth methods, respectively. When using the Wi-Fi or Bluetooth module, various connection information, such as the SSID and session key, is transmitted and received first; after establishing a communication connection using this information, various data can be transmitted and received. The infrared communication module performs communication according to infrared communication (IrDA, Infrared Data Association) technology, which wirelessly transmits data over short distances using infrared rays that lie between visible light and millimeter waves.

[0065] In addition to the communication method described above, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), and 5G (5th Generation).

[0066] Alternatively, the communication circuit (130) may include a wired communication circuit such as HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, etc.

[0067] In addition, the communication circuit (130) may include at least one of a LAN (Local Area Network) module, an Ethernet module, or a wired communication module that performs communication using a pair cable, a coaxial cable, or a fiber optic cable.

[0068] The processor (140) controls the overall operation of the smart glasses (100). Specifically, the processor (140) is connected to each component of the smart glasses (100) to control the overall operation of the smart glasses (100). For example, the processor (140) is connected to components such as memory (110), camera (120), and communication circuit (130) to control the operation of the smart glasses (100).

[0069] At least one processor (140) may include one or more of a CPU, a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), a MIC (Many Integrated Core), a NPU (Neural Processing Unit), a hardware accelerator, or a machine learning accelerator. At least one processor (140) may control one or any combination of other components of the smart glasses (100) and may perform operations or data processing related to communication. At least one processor (140) may execute one or more programs or instructions stored in memory. For example, at least one processor (140) may perform a method according to one embodiment of the present disclosure by executing one or more instructions stored in memory.

[0070] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0071] At least one processor (140) may be implemented as a single-core processor including one core, or as one or more multi-core processors including multiple cores (e.g., homogeneous multi-core or heterogeneous multi-core). When at least one processor (140) is implemented as a multi-core processor, each of the multiple cores included in the multi-core processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by multiple cores may be included in the multi-core processor. Additionally, each of the multiple cores included in the multi-core processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one embodiment of the present disclosure.

[0072] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0073] In the embodiments of the present disclosure, at least one processor (140) may refer to a system-on-chip (SoC) in which at least one processor and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. However, for convenience of explanation, the operation of the smart glasses (100) is described below using the expression "processor (140)."

[0074] The processor (140) can acquire images by capturing the real world around the smart glasses (100) through the camera (120) and detect a display device connected via the communication circuit (130) from the images acquired through the camera (120).

[0075] For example, the processor (140) may acquire images of the front of the smart glasses (100) captured through the camera (120) and detect a device having a preset motion in the images as a display device. For instance, the processor (140) may receive motion information from the display device and identify the display device in the images based on the motion information. Alternatively, the processor (140) may identify the device as a display device if it is identified in the images that the device is displaying a preset image. For instance, the display device may display an image, video, barcode, or QR code that is not identifiable by the user, and the processor (140) may identify the device displaying the image, video, barcode, or QR code in the images as a display device.

[0076] The processor (140) can obtain first information about real-world objects based on user gestures from images obtained through the camera (120).

[0077] For example, the processor (140) can obtain first information about an object indicated by the user's finger in images. For instance, the processor (140) can obtain "go" indicated by the user's finger in a captured image as first information.

[0078] The processor (140) may input images into a first neural network model to identify an object indicated by the user's finger as first information. Alternatively, the processor (140) may input images into a first neural network model to identify an area indicated by the user's finger and identify text contained in the identified area as first information. Here, the first neural network model may be a model trained to identify an object indicated by the user's finger in images. However, it is not limited thereto, and the processor (140) may obtain an object indicated by the user's finger in images as first information based on rules.

[0079] The processor (140) can process first information about an object to obtain second information and transmit a control signal to the display device through the communication circuit (130) to display the second information on the display device.

[0080] For example, the processor (140) can determine a function to be processed in the smart glasses (100) based on the attributes of the first information about the object, process the first information based on the determined function to obtain second information, and transmit a control signal through the communication circuit (130) to display the second information on the display device.

[0081] If the first information is not of a preset language type, the processor (140) may obtain second information by translating the first information into a preset language type, and if the first information is an image or a question, may obtain a description of the image or an answer to the question as second information. Alternatively, if the processor (140) cannot interpret the first information, it may obtain second information by copying the first information.

[0082] However, it is not limited thereto, and the processor (140) may determine the function to be processed based on user commands. Alternatively, the processor (140) may receive information about the function to be processed from the display device (200).

[0083] The processor (140) can identify information to be obtained from real-world objects based on user gestures in images obtained through the camera (120), and can obtain first information based on the identified information.

[0084] For example, the processor (140) can identify a word corresponding to the touch among the contents included in the object if the gesture is a touch on a point on the object, identify a sentence or phrase corresponding to the beginning to the end of the drag among the contents included in the object if the gesture is a drag in a first direction on the object, and identify sentences, paragraphs, or images corresponding to the beginning to the end of the drag among the contents included in the object if the gesture is a drag in a second direction on the object.

[0085] However, it is not limited to this, and the types of gestures can vary widely. For example, a gesture may include at least one of touch (tap, drag, tap & hold, tap, double tap, press, flick, swipe, pan, pinch, or rotate).

[0086] Additionally, the processor (140) may transmit a feedback signal to the display device through the communication circuit (130) so that different feedback is provided in the display device depending on the type of identified information.

[0087] When the processor (140) receives a preset command from the display device (200) through the communication circuit (130) while a communication channel is formed with the display device (200), it can acquire images through the camera (120). For example, when a user executes a preset application on the display device (200), the display device (200) can form a communication channel with the smart glasses (100) and transmit a preset command to the smart glasses (100). When the processor (140) receives a preset command from the display device (200) through the communication circuit (130) while a communication channel is formed with the display device (200), it can control the communication circuit (130) to acquire images through the camera (120), acquire first information, and transmit second information and a command to display the second information to the display device (200).

[0088] However, it is not limited to this, and even when a communication channel is not formed between the smart glasses (100) and the display device (100), the smart glasses (100) may acquire images through the camera (120) and acquire first information as a preset application of the smart glasses (100) is executed.

[0089] For example, the memory (110) further stores a plurality of appearance images corresponding to each of a plurality of display devices including a display device (200), and the processor (140) may detect a display device having a corresponding appearance image among the plurality of display devices located around the user from images obtained through the camera (120).

[0090] Additionally, the processor (140) may perform additional shooting to acquire additional images while the second information is being displayed on the detected display device, and when the display device having a corresponding exterior image among the plurality of display devices is changed, transmit a control signal to the changed display device through the communication circuit (130) to display the second information. In this case, the changed display device may display the second information.

[0091] In one embodiment, the processor (140) may control the communication circuit (130) to transmit a control signal to a modified display device through the communication circuit (130) to cause the second information to be displayed, and to transmit a display stop signal to an identified display device through the communication circuit (130). In this case, the modified display device may display the second information, and the identified display device may stop displaying the second information.

[0092] The processor (140) can acquire images by capturing the real world around the smart glasses (100) through the camera (120) while transmitting second information to the display device (200) based on a control signal, and when it is identified from the images acquired through the camera (120) that the display device (200) displays the second information through the display of the display device (200), it can transmit alignment information to the display device (200) through the communication circuit (130) to control the second information to be aligned and displayed according to the content included in the real-world object. The display device (200) can rotate the display screen of the second information based on the alignment information.

[0093] Alternatively, the processor (140) may acquire images by capturing the real world around the smart glasses (100) through the camera (120) while transmitting the second information to the display device (200) based on a control signal, and if it is identified from the images acquired through the camera (120) that the display device (200) displays the second information through the display of the display device (200), it may transmit alignment information to the display device (200) through the communication circuit (130) to control the content included in the real world object and the second information to be displayed horizontally aligned in the user's field of view.

[0094] The processor (140) may identify a device to transmit a display signal of the first information or the second information based on the type of the first information or the second information. For example, if the first information or the second information is text or a picture, the processor (140) may identify a display device (200) as the transmission target device and transmit a display signal of the first information or the second information to the display device (200) through the communication circuit (130). Alternatively, if the first information or the second information includes a musical score, the processor (140) may transmit a playback signal to a sound device through the communication circuit (130) to cause the musical score to be played based on the first information or the second information. Alternatively, if the first information or the second information includes a musical score, the processor (140) may identify a sound device as the transmission target device and transmit sound information corresponding to the musical score to the sound device through the communication circuit (130).

[0095] The display device (200) is a smart watch, and the processor (140) may identify the hand of a user wearing the smart watch in images through the camera (120) and obtain first information based on the position of the user's hand or finger on a real-world object. Through this operation, the processor (140) can identify the user's finger of the smart watch even if multiple users' fingers are identified.

[0096] Meanwhile, the artificial intelligence-related functions according to the present disclosure can be operated through the processor (140) and memory (110).

[0097] The processor (140) may be composed of one or more processors. In this case, the one or more processors may be a general-purpose processor such as a CPU, AP, DSP, etc., a graphics-dedicated processor such as a GPU, VPU (Vision Processing Unit), or an artificial intelligence-dedicated processor such as an NPU.

[0098] One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory (110). Alternatively, if one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model. The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0099] Here, "created through learning" means that a basic artificial intelligence model is trained using multiple learning data by a learning algorithm, thereby creating a predefined rule of operation or an artificial intelligence model configured to perform a desired characteristic (or objective). Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0100] An artificial intelligence model can be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through calculations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights can be updated during the learning process so that the loss or cost values ​​obtained by the artificial intelligence model are reduced or minimized.

[0101] Artificial neural networks may include deep neural networks (DNNs), such as, but are not limited to, Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), Generative Adversarial Networks (GANs), or Deep Q-Networks.

[0102] FIG. 3 is a block diagram showing the detailed configuration of smart glasses (100) according to one embodiment of the present disclosure. The smart glasses (100) may include a memory (110), a camera (120), a communication circuit (130), and a processor (140). Additionally, according to FIG. 3, the smart glasses (100) may further include a user interface (150) and a microphone (160). According to one embodiment, the smart glasses (100) may further include a display (170) and a speaker (180).

[0103] The user interface (150) may be implemented as a button, touchpad, mouse, and keyboard, or as a touch screen capable of performing display functions and operation input functions. Here, the button may be a various type of button, such as a mechanical button, touchpad, or wheel, formed in any area of ​​the exterior of the main body of the smart glasses (100), such as the front, side, or back.

[0104] When a user command is received through the user interface (150), the processor (140) may control the communication circuit (130) to acquire images, acquire first information from the images, and transmit second information based on the first information and a command to display the second information to the display device (200).

[0105] The microphone (160) is configured to receive sound input and convert it into an audio signal. The microphone (160) is electrically connected to the processor (140) and can receive sound under the control of the processor (140).

[0106] For example, the microphone (160) may be formed as an integrated unit on the upper side, front side, or side side of the smart glasses (100). Alternatively, the microphone (160) may be provided in a remote control or the like, separate from the smart glasses (100). In this case, the remote control may receive sound through the microphone (160) and provide the received sound to the smart glasses (100).

[0107]

[0108] * The microphone (160) may include various configurations such as a microphone that collects analog sound, an amplifier circuit that amplifies the collected sound, an A / D conversion circuit that samples the amplified sound and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0109] Meanwhile, the microphone (160) may be implemented in the form of a sound sensor, and any configuration capable of collecting sound is acceptable.

[0110] When a preset user voice is received through a microphone (160), the processor (140) may control a communication circuit (130) to acquire images, acquire first information from the images, and transmit second information based on the first information and a command to display the second information to a display device (200).

[0111] The smart glasses (100) may be implemented in a form that does not include a display (170) and a speaker (180). However, they are not limited thereto, and the smart glasses (100) may be implemented in a form that includes at least one of a display (170) or a speaker (180).

[0112] When the smart glasses (100) include a display (170), the processor (140) may display the second information through the display (170) and may control the communication circuit (130) to transmit the second information and a command to display the second information to the display device (200). For example, the processor (140) may control the communication circuit (130) to display the second information through the display (170) when a communication channel with the display device (200) is not formed, and to transmit it to the display device (200) when a communication channel with the display device (200) is formed.

[0113] Here, the display (170) is configured to display an image and can be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, a PDP (Plasma Display Panel), etc. The display (170) may also include a driving circuit, a backlight unit, etc., which can be implemented in forms such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (170) can be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0114] When the smart glasses (100) include a speaker (180), the processor (140) may output sound information corresponding to the musical score through the speaker (180) if the second information includes a musical score, or may control the communication circuit (130) to transmit the sound information corresponding to the musical score to a sound device. For example, if the processor (140) is not in a state where a communication channel with the sound device is not formed, it may output sound information corresponding to the musical score through the speaker (180), and if a communication channel with the sound device is formed, it may control the communication circuit (130) to transmit sound information corresponding to the musical score to the sound device.

[0115] Here, the speaker (180) is a component that outputs various audio data processed by the processor (140), as well as various notification sounds or voice messages.

[0116] As described above, the smart glasses (100) can improve user convenience by capturing images of the front of the smart glasses (100) and transmitting information based on an object indicated by the user in the images to a display device.

[0117] The operation of the smart glasses (100) will be explained in more detail below through FIGS. 4 to 12. FIGS. 4 to 12 describe individual embodiments for the convenience of explanation. However, the individual embodiments of FIGS. 4 to 12 may be implemented in any combination.

[0118] FIGS. 4 to 6 are drawings for explaining the operation according to a user's touch method according to one embodiment of the present disclosure.

[0119] The processor (140) can acquire images by capturing the real world around the smart glasses (100) through the camera (120), identify the type of information to be acquired based on the user's gestures in the images, and acquire the first information based on the type.

[0120] For example, as illustrated in FIG. 4, if the gesture is a touch to a point on an object, the processor (140) can identify the type of information to be acquired as the word "go (410)" and translate "go (410)" to transmit the display signals of "go" and "go" to the display device (200) through the communication circuit (130). In this case, the display device (200) can display a screen (420) containing "go".

[0121] Alternatively, the processor (140) may, as illustrated in FIG. 5, if the gesture is a first-direction drag on an object, identify the type of information to be acquired as the sentence "I go to school (510)" and translate "I go to school (510)" to transmit the display signals "I go to school" and "I go to school" to the display device (200) through the communication circuit (130). In this case, the display device (200) may display a screen (520) containing "I go to school".

[0122] Alternatively, if the gesture is a drag in the second direction on an object as illustrated in FIG. 6, the processor (140) may identify the type of information to be acquired as the paragraph “What are you doing?! Let go of me! (610)” and translate “What are you doing?! Let go of me! (610)” to transmit display signals of “What are you doing?! Let me go!” and “What are you doing?! Let me go!” to the display device (200) through the communication circuit (130). In this case, the display device (200) may display a screen (620) containing “What are you doing?! Let me go!”.

[0123] Here, the first direction drag is a drag within a preset angle range from the horizontal line, and the second direction drag may be a drag within the remaining angle range. For example, the first direction drag may be a drag between -10 degrees and 10 degrees from the horizontal line, and the second direction drag may be a drag within the remaining angle range.

[0124] However, it is not limited thereto, and the processor (140) may identify the type of information to be acquired based on the user's touch or drag and the object at the corresponding location. For example, the processor (140) may identify the type of information to be acquired as a word when the user touches a point on the object, identify the type of information to be acquired as a sentence when the user drags the bottom of the object, and identify the type of information to be acquired as a paragraph or an image when the user drags across the object.

[0125] FIG. 7 is a drawing for explaining the processing mode of smart glasses (100) according to one embodiment of the present disclosure.

[0126] When a portion of the images is identified by a user's drag input from the images, the processor (140) can input the portion into a second neural network model to obtain a description of the portion. For example, when a portion of the images (710) is identified as illustrated in FIG. 7, the processor (140) can input the portion (710) into a second neural network model to obtain information that the portion (710) contains a puppy, and transmit display signals of "It is a puppy" and "It is a puppy" to a display device (200) through a communication circuit (130). In this case, the display device (200) can display a screen (720) containing "It is a puppy". Here, the second neural network model may be a model trained to output a description of the image. That is, the processor (140) can obtain a description of the image if the first information obtained based on the position of the user's finger in the images is an image, and transmit the description of the image to the display device (200) through the communication circuit (130).

[0127] Alternatively, the processor (140) may change the processing mode based on the text if the first information obtained based on the position of the user's finger in the images is text. For example, if the first information is not of a preset language type, the processor (140) may obtain second information by translating the first information into a preset language type, and if the first information is a question, it may obtain a description of the image or an answer to the question as second information. Alternatively, if the first information is not of a preset language type, the processor (140) may obtain second information by translating the first information into a preset language type, and if the second information translated into the preset language type is a question, it may update the second information by adding an answer to the question to the second information. In this case, the display device (200) may display the second information translated into a preset language type and the answer thereto.

[0128] If the processor (140) is unable to interpret the first information, it may copy the first information to obtain the second information.

[0129] FIG. 8 is a drawing for explaining an initialization method according to one embodiment of the present disclosure.

[0130] When a preset user input is identified, the processor (140) can transmit a display interruption signal to the display device (200) through the communication circuit (130).

[0131] For example, the processor (140) can acquire a plurality of field images of the front of the smart glasses (100) through the camera (120) as shown in FIG. 8, and if the touch method of the finger in the images is a double tap, it can transmit a display stop signal to the display device (200) through the communication circuit (130). In this case, the display device (200) can display a screen (810) that does not display any information.

[0132] FIG. 9 is a drawing for explaining a method for determining a function to be processed in smart glasses (100) according to one embodiment of the present disclosure.

[0133] The user can determine the function to be processed on the smart glasses (100) through the display device (200). For example, the display device (200) can identify whether a connected device exists (920) when an icon (910) corresponding to a preset application is selected, as shown in FIG. 9.

[0134] The display device (200) can display a screen (930) containing information about the connected device (smart glasses of AA) if a connected device exists, and can display a screen (940) for selecting a mode. Each mode can represent a function to be processed by the smart glasses (100).

[0135] The display device (200) displays a screen (950) that indicates that there is no connected device and induces pairing when there is no connected device, and when a command to connect to smart glasses (960) is received, it displays a screen (930) containing information about the connected device (smart glasses of AA) and can display a screen (940) for selecting a mode.

[0136] When one of the multiple modes is selected, the display device (200) can transmit information about the selected mode to the smart glasses (100).

[0137] However, it is not limited to this, and the smart glasses (100) may determine the mode by receiving a user command. Alternatively, the smart glasses (100) may change the mode by receiving a user command. When the mode is determined or changed, the smart glasses (100) may provide information about the mode to the display device (200).

[0138] FIG. 10 is a drawing for explaining a method of controlling the display state of a display device (200) according to one embodiment of the present disclosure.

[0139] The processor (140) transmits second information to the display device (200) and captures the real world around the smart glasses (100) through the camera (120) to obtain other images, and when it is identified from the other images obtained through the camera (120) that the display device (200) displays the second information through the display of the display device (200), it can transmit alignment information through the communication circuit (130) to control the second information to be aligned and displayed according to the content included in the real world object.

[0140] For example, the processor (140) transmits second information, such as "I go to...", to the display device (200), and the display device (200) can display a screen (1010) containing "I go to..." as shown on the left side of FIG. 10. The processor (140) acquires other images by continuously capturing the front of the smart glasses (100) through the camera (120), and if tilting is identified in one of the other images, such as a screen (1020) containing "I go to..." as shown on the right side of FIG. 10, it can transmit alignment information to the display device (200) through the communication circuit (130) to control the display so that "I go to..." is horizontally aligned.

[0141] However, it is not limited to this, and the display device (200) may perform an alignment operation on its own. For example, the display device (200) may include an acceleration sensor, an angular velocity sensor, etc., and may identify the tilt of the screen (1020) containing "I go to..." based on the orientation information of the display device (200) obtained from the sensor, and correct the tilt.

[0142] FIG. 11 is a drawing for explaining an operation according to the type of second information according to one embodiment of the present disclosure.

[0143] The processor (140) can identify a device to transmit the second information and a command to display the second information based on the type of the second information. For example, if the second information is text or a picture, the processor (140) may identify a display device (200) as the transmission target device and transmit the second information to the display device (200) through the communication circuit (130). Alternatively, as illustrated in FIG. 11, if the second information includes a musical score (1110), the processor (140) may identify a sound device as the transmission target device and transmit the musical score (1110) to the sound device through the communication circuit (130). Alternatively, if the second information includes a musical score (1110), the processor (140) may identify a sound device as the transmission target device and transmit sound information corresponding to the musical score (1110) to the sound device through the communication circuit (130). In this case, the sound device can output sound based on the musical score (1110) or sound information corresponding to the musical score (1110).

[0144] FIG. 12 is a drawing for explaining a method of specifying a display device according to one embodiment of the present disclosure.

[0145] As shown in the upper part of FIG. 12, the processor (140) can acquire images of the front of the smart glasses (100) through the camera (120) while a communication channel with other electronic devices is not formed, acquire first information based on the position of the user's finger in the images, and acquire second information by processing the first information.

[0146] As illustrated in the middle of FIG. 12, when the second information is acquired, the processor (140) acquires other images of the front of the smart glasses (100) through the camera (120), forms a communication channel with the other display device (300) based on the position of the finger in the other images, and can transmit a control signal to the other display device (300) through the communication circuit (130) to display the second information.

[0147] However, if there are multiple display devices around the smart glasses (100), the processor (140) may not be able to identify the other display device (300) corresponding to the position of the finger in the other images. In this case, the processor (140) may identify the other display device (300) corresponding to the position of the finger in the other images based on multiple appearance images corresponding to each of the multiple display devices stored in the memory (110).

[0148] As shown in the lower part of FIG. 12, the processor (140) can transmit a control signal to the display device (200) via the communication circuit (130) to display second information when the position of the finger changes from another display device (300) to the display device (200). Additionally, the processor (140) can transmit a display interruption signal to the other display device (300) via the communication circuit (130).

[0149] FIG. 13 is a drawing for explaining the operation when a plurality of display devices exist according to one embodiment of the present disclosure.

[0150] The processor (140) can acquire images by capturing the real world around the smart glasses (100) through the camera (120), acquire first information based on the user's gestures in the images, process the first information to acquire second information, and transmit the second information to the display device (200).

[0151] For example, the processor (140) can acquire images by capturing the real world around the smart glasses (100) through the camera (120) as shown in FIG. 13, identify "go (1310)" based on the user's gesture in the images, translate "go (1310)" and transmit the display signal of "go" and "go" through the communication circuit (130) to at least one of the smart watch (210) or smartphone (220). In this case, at least one of the smart watch (210) or smartphone (220) can display a screen (1320) containing "go".

[0152] The processor (140) may determine the display device (200) to which the second information is to be transmitted based on at least one of the location of the user's gesture, the state of forming a communication channel with the surrounding display device (200), or the hardware specifications of the surrounding display device (200). For example, the processor (140) may transmit the second information to a smart watch (210), which is the display device (200) closest to the location of the user's gesture. Alternatively, the processor (140) may transmit the second information to a smart watch (210) with which a communication channel has been formed. Alternatively, if a communication channel has been formed with both the smart watch (210) and the smartphone (220), the processor (140) may transmit the second information to the smart watch (210) and the smartphone (220). Alternatively, the processor (140) may transmit the second information to a smartphone (220) with a relatively large display size.

[0153] FIG. 14 is a flowchart illustrating a method for controlling smart glasses according to one embodiment of the present disclosure.

[0154] First, images are acquired by capturing the real world around the smart glasses through a camera included in the smart glasses (S1410). Then, a display device connected to the smart glasses is detected from the images acquired through the camera (S1420). Then, first information about a real-world object is acquired from the images acquired through the camera based on the user's gesture (S1430). Then, second information is acquired by processing the first information about the object (S1440). Then, a control signal is transmitted to the display device to display the second information (S1450).

[0155] Additionally, the step of acquiring the second information (S1440) can determine a function to be processed in the smart glasses based on the attributes of the first information about the object, and acquire the second information by processing the first information based on the determined function.

[0156] And, in the step of acquiring the second information (S1440), if the first information is not of a preset language type, the first information is translated into a preset language type to acquire the second information, and if the first information is an image or a question, a description of the image or an answer to the question can be acquired as the second information.

[0157] Additionally, in the step of acquiring the second information (S1440), if interpretation of the first information is impossible, the first information may be copied and acquired as the second information.

[0158] And, the step of acquiring the first information (S1430) can identify information to be acquired from a real-world object based on a user's gesture in images acquired through a camera, and acquire the first information based on the identified information.

[0159] Additionally, the step of acquiring the first information (S1430) can identify a word corresponding to the touch among the contents contained in the object if the gesture is a touch on a point on the object, identify a sentence or phrase corresponding to the beginning to the end of the drag among the contents contained in the object if the gesture is a drag in the first direction on the object, and identify sentences, paragraphs, or images corresponding to the beginning to the end of the drag among the contents contained in the object as the type of information to be acquired if the gesture is a drag in the second direction on the object.

[0160] And, it may further include the step of transmitting a feedback signal to a display device to provide feedback that provides different feedback on the display device depending on the type of identified information.

[0161] Additionally, in the step of acquiring images (S1410), when a communication channel is formed with the display device and a preset command is received from the display device, images can be acquired through the camera.

[0162] In addition, the smart glasses store and detect a plurality of appearance images corresponding to each of a plurality of display devices including a display device (S1420), and can detect a display device having a corresponding appearance image among the plurality of display devices located around the user from images acquired through a camera.

[0163] Additionally, while transmitting second information to a display device based on a control signal, the method may further include the step of capturing the real world around the smart glasses through a camera to acquire images, and, when it is identified from the images acquired through the camera that the display device displays the second information through the display of the display device, the step of transmitting alignment information to the display device to control the second information to be aligned and displayed according to the content included in the real-world object.

[0164] And, if the first information or the second information includes a musical score, the method may further include the step of transmitting a playback signal to a sound device through a communication circuit to enable the musical score to be played based on the first information or the second information.

[0165] Additionally, the display device is a smart watch, and the step of acquiring first information (S1430) can identify the hand of a user wearing a smart watch from images acquired through a camera, and acquire first information based on the position of the user's hand or fingers on a real-world object.

[0166] According to an embodiment of the present disclosure for achieving the above-described purpose, the smart glasses include at least one processor comprising a memory for storing instructions, a camera, a communication circuit, and a processing circuitry. When the instructions are executed individually or collectively by the at least one processor, the smart glasses may capture images of the real world surrounding the smart glasses through the camera, detect a display device connected via the communication circuitry from the images acquired through the camera, acquire first information about an object in the real world based on a user's gesture from the images acquired through the camera, process the first information about the object to acquire second information, and transmit a control signal to the display device via the communication circuitry to display the second information on the display device.

[0167] According to one example, when the instructions are executed individually or collectively by the at least one processor, the function to be processed in the smart glasses is determined based on the attributes of the first information regarding the object, and the second information can be obtained by processing the first information based on the determined function.

[0168] According to one example, when the instructions are executed individually or collectively by at least one processor, if the first information is not of a preset language type, the first information is translated into the preset language type to obtain the second information, and if the first information is an image or a question, a description of the image or an answer to the question can be obtained as the second information.

[0169] According to one example, when the instructions are executed individually or collectively by the at least one processor, if interpretation of the first information is impossible, the first information can be copied and obtained as the second information.

[0170] According to one example, when the instructions are executed individually or collectively by the at least one processor, they can identify information to be obtained from the object in the real world based on the user's gesture in images obtained through the camera, and obtain the first information based on the identified information.

[0171] According to one example, when the instructions are executed individually or collectively by at least one processor, if the gesture is a touch on a point on the object, a word corresponding to the touch is identified among the contents included in the object; if the gesture is a first-direction drag on the object, a sentence or phrase corresponding to the beginning to the end of the drag is identified among the contents included in the object; and if the gesture is a second-direction drag on the object, sentences, paragraphs, or images corresponding to the beginning to the end of the drag are identified among the contents included in the object.

[0172] According to one example, when the instructions are executed individually or collectively by the at least one processor, a feedback signal can be transmitted to the display device through the communication circuit to provide different feedback to the display device according to the type of the identified information.

[0173] According to one example, when the instructions are executed individually or collectively by the at least one processor, and a communication channel is formed with the display device, images can be acquired through the camera when a preset command is received from the display device through the communication circuit.

[0174] According to one example, the memory further stores a plurality of appearance images corresponding to each of a plurality of display devices including the display device, and when the instructions are executed individually or collectively by the at least one processor, the display device having a corresponding appearance image among the plurality of display devices located around the user in the images acquired through the camera can be detected as the display device.

[0175] According to one example, when the instructions are executed individually or collectively by the at least one processor, while transmitting the second information to the display device based on the control signal, images are acquired by capturing the real world around the smart glasses through the camera, and if it is identified from the images acquired through the camera that the display device displays the second information through the display of the display device, alignment information can be transmitted to the display device through the communication circuit to control the second information to be aligned and displayed according to the content included in the object of the real world.

[0176] According to one example, when the instructions are executed individually or collectively by the at least one processor, if the first information or the second information includes a musical score, a playback signal can be transmitted to a sound device through the communication circuit to play the musical score based on the first information or the second information.

[0177] According to one example, the display device is a smart watch, and the instructions, when executed individually or collectively by the at least one processor, can identify the hand of the user wearing the smart watch in images acquired through the camera and acquire the first information based on the position of the user's hand or finger on the object in the real world.

[0178] A control method for smart glasses according to one embodiment may include the steps of: capturing the real world surrounding the smart glasses through a camera included in the smart glasses to obtain images; detecting a display device connected to the smart glasses from the images obtained through the camera; obtaining first information about an object in the real world based on a user's gesture from the images obtained through the camera; processing the first information about the object to obtain second information; and transmitting a control signal to the display device to cause the second information to be displayed on the display device.

[0179] According to one example, the step of acquiring the second information may determine a function to be processed in the smart glasses based on the attributes of the first information regarding the object, and acquire the second information by processing the first information based on the determined function.

[0180] According to one example, the step of acquiring the second information may be to acquire the second information by translating the first information into the pre-set language type if the first information is not of a pre-set language type, and if the first information is an image or a question, to acquire a description of the image or an answer to the question as the second information.

[0181] According to one example, in the step of obtaining the second information, if interpretation of the first information is impossible, the first information may be copied and obtained as the second information.

[0182] According to one example, the step of acquiring the first information may identify information to be acquired from the object in the real world based on the user's gesture in images acquired through the camera, and acquire the first information based on the identified information.

[0183] According to one example, the step of acquiring the first information may identify a word corresponding to the touch among the content contained in the object if the gesture is a touch on a point on the object, identify a sentence or phrase corresponding to the beginning to the end of the drag among the content contained in the object if the gesture is a drag in a first direction on the object, and identify sentences, paragraphs, or images corresponding to the beginning to the end of the drag among the content contained in the object as the type of information to be acquired if the gesture is a drag in a second direction on the object.

[0184] According to one example, the method may further include the step of transmitting a feedback signal to the display device to provide feedback that provides different feedback from the display device depending on the type of the identified information.

[0185] According to one example, the step of acquiring the images can be performed by acquiring the images through the camera when a preset command is received from the display device while a communication channel is formed with the display device.

[0186] According to one example, the smart glasses store a plurality of appearance images corresponding to each of the plurality of display devices including the display device, and the detection step can detect a display device having a corresponding appearance image among the plurality of display devices located around the user from images obtained through the camera as the display device.

[0187] According to one example, while transmitting the second information to the display device based on the control signal, the method may further include the step of acquiring images by capturing the real world around the smart glasses through the camera, and, when it is identified from the images acquired through the camera that the display device displays the second information through the display of the display device, the method may further include the step of transmitting alignment information to the display device to control the second information to be aligned and displayed according to the content included in the object of the real world.

[0188] According to one example, if the first information or the second information includes a musical score, the method may further include the step of transmitting a playback signal to a sound device through the communication circuit to enable the musical score to be played based on the first information or the second information.

[0189] According to one example, the display device is a smart watch, and the step of acquiring the first information may identify the hand of the user wearing the smart watch in images acquired through the camera, and acquire the first information based on the position of the user's hand or finger on the object in the real world.

[0190] According to various embodiments of the present disclosure as described above, smart glasses can improve user convenience by capturing the real world around the smart glasses to acquire images and transmitting information based on an object indicated by the user in the images to a display device.

[0191] The electronic device according to one or more embodiments disclosed in this disclosure may be a device of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this disclosure is not limited to the devices described above.

[0192] One or more embodiments of the present disclosure and the terms used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In the present disclosure, each of phrases such as “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationally,” it means that said component may be connected to said other component directly (e.g., wired), wirelessly, or through a third component.

[0193] The term “module” as used in one or more embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0194] One or more embodiments of the present disclosure may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0195] According to one embodiment, the method according to one or more embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0196] According to one or more embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one or more embodiments, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one or more embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. Regarding smart glasses, Memory for storing instructions; camera; Communication circuit; and at least one processor including processing circuitry; and When the above instructions are executed individually or collectively by the at least one processor, Images are obtained by capturing the real world around the smart glasses through the camera, and Detecting a display device connected via the communication circuit from images acquired through the camera, and First information about the real-world object is obtained based on the user's gesture from images obtained through the camera, and Process the first information regarding the above object to obtain second information, and Smart glasses that transmit a control signal to the display device through the communication circuit to display the second information on the display device.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, Determining the function to be processed in the smart glasses based on the attributes of the first information regarding the object, and Smart glasses that process the first information based on the above-determined function to obtain the second information.

3. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, If the above first information is not of a preset language type, the above first information is translated into the preset language type to obtain the above second information, and Smart glasses that, if the first information is an image or a question, obtain a description of the image or an answer to the question as the second information.

4. In Paragraph 3, When the above instructions are executed individually or collectively by the at least one processor, Smart glasses that, when interpretation of the first information is impossible, copy the first information and obtain it as the second information.

5. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, Identifying information to be obtained from the object in the real world based on the user's gesture in images obtained through the camera, and Smart glasses that acquire the first information based on the identified information.

6. In Paragraph 5, When the above instructions are executed individually or collectively by the at least one processor, If the above gesture is a touch on a point on the object, identify the word corresponding to the touch among the contents included in the object, and If the above gesture is a drag in the first direction on the object, identify a sentence or phrase among the contents included in the object that corresponds to the start to the end of the drag, and Smart glasses that identify sentences, paragraphs, or images among the contents included in the object that correspond to the start to the end of the drag, if the gesture is a drag in a second direction on the object.

7. In Paragraph 5, When the above instructions are executed individually or collectively by the at least one processor, Smart glasses that transmit a feedback signal to a display device through a communication circuit, so that different feedback is provided to the display device according to the type of information identified above.

8. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, Smart glasses that acquire images through the camera when a preset command is received from the display device through the communication circuit while a communication channel is formed with the display device.

9. In Paragraph 1, The above memory is, A plurality of exterior images corresponding to each of the plurality of display devices including the above-mentioned display device are further stored, and When the above instructions are executed individually or collectively by the at least one processor, Smart glasses that detect a display device having a corresponding appearance image among a plurality of display devices located around the user in images obtained through the camera.

10. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, While transmitting the second information to the display device based on the above control signal, images are acquired by capturing the real world around the smart glasses through the camera, and Smart glasses that transmit alignment information to the display device through the communication circuit to control the display device to be aligned and displayed in accordance with the content included in the object in the real world, when it is identified that the display device displays the second information through the display of the display device in images acquired through the camera.

11. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, Smart glasses that transmit a playback signal to a sound device through the communication circuit to play the musical score based on the first information or the second information, if the first information or the second information includes a musical score.

12. In Paragraph 1, The above display device is, It is a smartwatch, and When the above instructions are executed individually or collectively by the at least one processor, Identifying the hand of the user wearing the smartwatch in images acquired through the camera, and Smart glasses that acquire the first information based on the position of the user's hand or finger on the object in the real world.

13. In a method for controlling smart glasses, A step of acquiring images by capturing the real world around the smart glasses through a camera included in the smart glasses; A step of detecting a display device connected to the smart glasses in images obtained through the camera; A step of obtaining first information about the real-world object based on the user's gesture in images obtained through the camera; A step of obtaining second information by processing the first information regarding the object; and A control method comprising the step of transmitting a control signal to the display device to cause the second information to be displayed on the display device.

14. In Paragraph 13, The step of obtaining the above second information is, Determining the function to be processed in the smart glasses based on the attributes of the first information regarding the object, and A control method for obtaining the second information by processing the first information based on the above-determined function.

15. In Paragraph 13, The step of obtaining the above second information is, If the above first information is not of a preset language type, the above first information is translated into the preset language type to obtain the above second information, and A control method for obtaining a description of the image or an answer to the question as the second information, wherein if the first information is an image or a question.

Images