Artificial intelligence-based smart glass and control method therefor

Abstract

The present disclosure relates to an artificial intelligence-based smart glass and a control method therefor. The smart glass according to the present disclosure comprises: a camera module; an output module for outputting information related to an object captured by means of the camera module; a memory for storing at least one process for performing an operation of outputting the information related to the object; and a processor for performing the operation of outputting the information related to the object on the basis of the at least one process. The processor: recognizes the object from an image captured by means of the camera module; determines information related to the recognized object by using a pre-trained artificial intelligence model; and controls the output module to output the information related to the determined object.

Description

AI-based smart glasses and a method for controlling the same

[0001] The present disclosure relates to smart glasses and a method for controlling the same. More specifically, the present disclosure relates to artificial intelligence-based smart glasses and a method for controlling the same that can be utilized in the field of research.

[0002] Basic research plays an essential role in advancing scientific knowledge and developing new technologies. However, conventional basic research requires significant time and effort, and collaboration and information sharing among researchers have often been limited. Furthermore, there was a risk of safety accidents and errors occurring during the experimental process.

[0003] Recently, research, interest, and commercialization of wearable computing systems that users can directly attach to their bodies have been actively underway. In particular, there is very high interest in so-called "smart glasses," which can provide various data through the lenses while worn like eyeglasses, and offer features such as mobile phone and camera functions.

[0004] Conventional smart glasses display information on a transparent display and acquire information about the user's front through a front-facing camera attached to the glasses. However, there are limitations to the information that can be acquired using only the front camera, and there are limitations to the information that can be provided to the user based on that information.

[0005] Accordingly, there is a growing need to combine artificial intelligence with smart glasses and utilize them in the field of basic research. There is also a need to apply smart glasses to basic research so that research can be conducted at a certain level regardless of the researcher's proficiency during the experiment, and to reduce the possibility of safety accidents and errors that may occur during the experiment.

[0006] The purpose of the embodiments disclosed in this disclosure is to provide smart glasses and a method for controlling the same that are capable of providing optimized information in the field of basic research based on artificial intelligence.

[0007] The problems that this disclosure aims to solve are not limited to those mentioned above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below.

[0008] A smart glass according to the present disclosure for achieving the above-described technical problem comprises a camera module, an output module that outputs information related to an object captured through the camera module, a memory in which at least one process for performing an operation of outputting information related to the object is stored, and a processor that performs an operation of outputting information related to the object based on the at least one process, wherein the processor recognizes the object in an image captured through the camera module, determines information related to the recognized object using a pre-trained artificial intelligence model, and controls the output module so that information related to the determined object is output.

[0009] In an embodiment, the information related to the object includes at least one of manual information used in the experiment, experimental protocol information, experimental data, experimental data analysis information, risk factor information, information related to rearing management, and status information.

[0010] In an embodiment, when the object is maintained in a state where it is included for a predetermined time in an image received through the camera module, the processor inputs the image containing the object into the artificial intelligence model and receives information related to the object as an output value from the artificial intelligence model.

[0011] In an embodiment, the previously trained artificial intelligence model includes a plurality of artificial intelligence models, and the processor recognizes the type of object included in the image received through the camera module, and based on the recognized type of object, receives information related to the object using different artificial intelligence models among the plurality of artificial intelligence models.

[0012] In an embodiment, the processor controls the output module to output information related to the first type of object using a first artificial intelligence model when a first type of object is recognized in the image, and controls the output module to output information related to the second type of object using a second artificial intelligence model different from the first artificial intelligence model when a second type of object different from the first type is recognized in the image.

[0013] In an embodiment, the system further includes a communication module that receives location information, and the processor controls the output module based on the location information received through the communication module so that information related to the output object changes even if the same object is recognized through the camera module.

[0014] In an embodiment, the processor outputs first information related to the object to the output module when the object is recognized at a first location, and outputs second information related to the object, which is different from the first information, to the output module when the object is recognized at a second location different from the first location.

[0015] In an embodiment, the processor further includes a microphone, and when a preset control command is received through the microphone while information related to the object is output, the processor stores at least two of the following in combination in the memory: an image captured through the camera module, information related to the object output to the output module, and information input through the microphone.

[0016] In an embodiment, the system further includes a sensor module formed to sense a risk factor, and the processor controls the output module to output a warning alarm based on the detection of a risk factor related to an object through the sensor module.

[0017] In addition, the control method of smart glasses according to the present disclosure includes the step of recognizing an object in an image captured through a camera module, the step of determining information related to the recognized object using a pre-trained artificial intelligence model, and the step of outputting information related to the determined object to an output module.

[0018] In addition to this, a computer program stored on a computer-readable recording medium for implementing the present disclosure may be further provided.

[0019] In addition to this, a computer-readable recording medium for recording a computer program for implementing the present disclosure may be further provided.

[0020] According to the aforementioned means for solving the problem of the present disclosure, the present disclosure makes it possible to provide optimized data in real time in the field of basic research using smart glasses.

[0021] In addition, the present disclosure can provide customized information based on the object being viewed by the user, can provide an experiment protocol while performing an object experiment, and can efficiently manage data related to the object while performing object management.

[0022] In addition, the present disclosure provides the effect of guiding experienced and novice practitioners to perform treatments at a similar level by providing information related to the object in real time.

[0023] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below.

[0024] FIG. 1 is an overall system diagram of the present disclosure.

[0025] FIG. 2 is a block diagram of smart glasses of the present disclosure.

[0026] FIG. 3 is a flowchart for explaining a control method of smart glasses of the present disclosure.

[0027] Figures 4 through 7 are each conceptual diagrams for explaining the control method examined in Figure 3.

[0028] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and general content in the art to which this disclosure pertains or content that overlaps between embodiments is omitted. The terms 'part, module, component, block' as used in the specification may be implemented in software or hardware, and depending on the embodiments, a plurality of 'parts, modules, components, blocks' may be implemented as a single component, or a single 'part, module, component, block' may include a plurality of components.

[0029] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.

[0030] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0031] Throughout the specification, when it is stated that a component is located "on" another component, this includes not only cases where a component is in contact with another component, but also cases where another component exists between the two components.

[0032] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0033] Singular expressions include plural expressions unless there is an obvious exception in the context.

[0034] In each step, identification codes are used for convenience of explanation and do not describe the order of the steps; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.

[0035] The operating principles and embodiments of the present disclosure will be described below with reference to the attached drawings.

[0036] In this specification, ‘the device,’ ‘the device according to the present disclosure,’ or ‘smart glasses’ includes all various devices capable of performing computational processing and providing results to a user. For example, the device according to the present disclosure may include all of a computer, a server device, and a portable terminal, or may take the form of any one of them.

[0037] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0038] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0039] The above portable terminal may include, for example, all types of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, smartphones, etc., as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0040] Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0041] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform a desired characteristic (or objective) are created by a basic artificial intelligence model being trained using a number of training data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0042] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained by the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0043] According to an exemplary embodiment of the present disclosure, a processor can implement artificial intelligence. Artificial intelligence refers to a machine learning method based on an artificial neural network that enables a machine to learn by mimicking human biological neurons. Methodologies of artificial intelligence can be classified according to the learning method into supervised learning, where input and output data are provided together as training data and the solution (output data) to the problem (input data) is predetermined; unsupervised learning, where only input data is provided without output data and the solution (output data) to the problem (input data) is not predetermined; and reinforcement learning, where a reward is given from an external environment whenever an action is taken from the current state, and learning proceeds in a direction that maximizes such reward. In addition, artificial intelligence methodologies can be classified according to the architecture, which is the structure of the learning model. The architectures of widely used deep learning technologies can be classified into Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Transformers, and Generative Adversarial Networks (GAN).

[0044] The device and system may include an artificial intelligence model. The artificial intelligence model may be a single model or may be implemented as multiple models. The artificial intelligence model may be composed of a neural network (or artificial neural network) and may include statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network may refer to a model that possesses problem-solving capabilities by having artificial neurons (nodes) that form a network through synaptic connections and change the strength of synaptic connections through learning. The neurons of a neural network may include combinations of weights or biases. A neural network may include one or more layers composed of one or more neurons or nodes. For example, the device may include an input layer, a hidden layer, and an output layer. The neural network constituting the device can infer a result (output) to be predicted from an arbitrary input by changing the weights of the neurons through learning.

[0045] The processor can create a neural network, train or learn a neural network, perform computations based on received input data, generate an information signal based on the results of the computation, or retrain the neural network. The neural network models may include, but are not limited to, various types of models such as Convolutional Neural Networks (CNN), Region with Convolutional Neural Networks (R-CNN), Region Proposal Networks (RPN), Recurrent Neural Networks (RNN), Stacking-based Deep Neural Networks (S-DNN), State-Space Dynamic Neural Networks (S-SDNN), Deconvolution Networks, Deep Belief Networks (DBN), Restructured Boltzmann Machines (RBM), Fully Convolutional Networks, Long Short-Term Memory Networks (LSTM), and Classification Networks, such as GoogleNet, AlexNet, and VGG Network. The processor may include one or more processors to perform computations according to the neural network models. For example, the neural network is a deep neural network It may include a (Deep Neural Network).

[0046] Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto) Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), GAN (Generative Adversarial Network), LSM (Liquid State Machine), ELM (Extreme Learning Machine), ESN (Echo It will be understood by a person skilled in the art that any neural network may be included, but is not limited to, State Network, Deep Residual Network, Differential Neural Computer, Neural Turning Machine, Capsule Network, Kohonen Network, and Attention Network.

[0047] According to exemplary embodiments of the present disclosure, the processor comprises a Convolutional Neural Network (CNN) such as GoogleNet, AlexNet, VGG Network, Region with Convolutional Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based Deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restructured Boltzmann Machine (RBM), Fully Convolutional Network, Long Short-Term Memory (LSTM) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA, Text Analysis, Dialog System, GPT-3, GPT-4 for Natural Language Processing, Visual Analytics, Visual Understanding, Video Synthesis for Vision Processing, Anomaly Detection, Prediction, Time-Series Forecasting, Optimization for ResNet Data Intelligence, Various artificial intelligence structures and algorithms, such as recommendation and data creation, may be used, but are not limited thereto. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0048] FIG. 1 is an overall system diagram of the present disclosure.

[0049] The smart glasses (100) according to the present disclosure can be utilized in the development of education and management systems in basic research fields (animal experiments and breeding management, etc.). Specifically, the present disclosure may include a configuration that enables the visualization of research data, provision of experimental protocols, data management, and collaboration among researchers by utilizing smart glasses in research areas such as laboratories.

[0050] Referring to FIG. 1, the present disclosure includes smart glasses (100), and information regarding objects (200) (e.g., animals, equipment, visitors, places, etc.) in the field of basic research can be output using the output modules (151, 152) of the smart glasses (100). At this time, the output information may be output in an augmented reality (AR), virtual reality (VR), mixed reality (MR), or extended reality (XR) manner. As an example, the information output from the output modules (151, 152) of the smart glasses (100) may be output on a display area (152a). The display area (152a) may be located on the display (151) of the smart glasses (100) or may be located in a predetermined space between the smart glasses (100) and the objects (200). When a display area (152a) is output in the above predetermined space, a holographic method may be applied.

[0051] The above information may include various information related to the object (200). For example, the information (210) related to the object may include at least one of manual information used in the experiment, experiment protocol information, experiment data, experiment data analysis information, risk factor information, information related to breeding management, and status information.

[0052] The information (210) related to the object may further include experimental data, experimental results, analysis information, experimental process information, experimental protocol information, information related to learning (virtual teaching materials, experimental procedures, equipment usage methods, etc.), information on risk factors that may occur or have occurred during the experiment, object status information, inventory information of experimental-related equipment, information related to laboratory personnel, etc.

[0053] The present disclosure enables the visualization of data regarding objects in front of a user in the field of basic research through smart glasses, and can provide automation and guidance of experimental processes, thereby enabling experts or beginners to perform experiments of a similar level.

[0054] Hereinafter, with reference to the attached drawings, we will examine in more detail a method for providing optimized information in the field of basic research using smart glasses according to the present disclosure.

[0055] FIG. 2 is a block diagram of smart glasses of the present disclosure.

[0056] The smart glasses (100) according to the present disclosure may include a communication module (110), a camera module (120), a sensor module (130), an interface module (140), an output module (150), an input module (160), a memory (170), and a processor (180). The components illustrated in FIG. 2 are not essential for implementing the smart glasses (100) according to the present disclosure, so the smart glasses (100) described herein may have more or fewer components than those listed above.

[0057] The smart glasses (100) according to the present disclosure may be formed to be driven or implemented independently, or may be implemented to be operated by being connected to an external device via wired or wireless connection.

[0058] Among the above components, the communication module (110) may include one or more components that enable communication with an external device, and may include, for example, at least one of a broadcast receiving module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

[0059] A broadcast receiving module receives broadcast signals and / or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include satellite channels and terrestrial channels. Two or more broadcast receiving modules may be provided in the device according to the present disclosure for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

[0060] The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

[0061] In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, the wireless communication module may include wireless communication modules that support various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G.

[0062] The wireless communication module may include a wireless communication interface comprising an antenna and a transmitter that transmit a mobile communication signal. Additionally, the wireless communication module may further include a signal conversion module that modulates a digital control signal output from the processor through the wireless communication interface into an analog wireless signal under the control of the processor.

[0063] The wireless communication module may include a wireless communication interface comprising an antenna and a receiver for receiving mobile communication signals. Additionally, the wireless communication module may further include a signal conversion module for demodulating an analog wireless signal received through the wireless communication interface into a digital control signal.

[0064] A short-range communication module is for short-range communication and can support short-range communication by using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0065] The location information module is a module for obtaining the location (or current location) of the device according to the present disclosure, and representative examples thereof include a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, if a GPS module is utilized, the location of the device can be obtained using signals sent from GPS satellites. As another example, if a Wi-Fi module is utilized, the location of the device can be obtained based on information from a Wireless Access Point (AP) that transmits or receives wireless signals from the Wi-Fi module. If necessary, the location information module may perform any of the functions of other modules of the communication unit to obtain data regarding the location of the device, either substituted or additionally. The location information module is a module used to obtain the location (or current location) of the device, and is not limited to a module that directly calculates or obtains the location of the device.

[0066] The camera module (120) processes image frames, such as still images or video, obtained by an image sensor in a video call mode or a shooting mode. The processed image frames can be displayed in a display area (152a) through a display unit (151) or an augmented reality output module (152). The image frames processed by the camera (121) can be stored in a memory (170) or transmitted to an external device through a communication module (110). Additionally, user location information, etc., can be calculated from the image frames obtained by the camera module (120).

[0067] Meanwhile, the camera module (120) may include a plurality of cameras, and in this case, they may be arranged to form a matrix structure. Through the cameras forming such a matrix structure, a plurality of image information having various angles or focal points may be input. In addition, the cameras may be arranged in a stereo structure to acquire left and right images for realizing a three-dimensional stereoscopic image.

[0068] The camera module (120) (or the first camera) can be formed to capture an image of the front of the smart glasses (100). Additionally, the camera module (120) can be rotated up, down, left, and right at a certain angle.

[0069] The smart glasses (100) according to the present disclosure may further include a camera (not shown) (or a second camera) formed to capture an image of the eye of a user wearing the smart glasses (100).

[0070] For example, the processor (180) can perform an analysis of images captured by the first camera and the second camera. Additionally, the processor (180) can perform an operation to acquire information about a forward object to which the user directs their gaze based on the analysis results, and can output the acquired information and information related to the object through the output module (150).

[0071] Additionally, the processor (180) can analyze an image of the user's eye captured by the second camera (120) and perform a specific function corresponding to the user's eye gesture recognized as a result of the analysis (e.g., a capture function, a function to change to other information, an artificial intelligence on / off function, etc.).

[0072] The sensor module (130) senses at least one of internal information of the device (smart glasses (100)), surrounding environment information surrounding the device, and user information, and generates a corresponding sensing signal. Based on this sensing signal, the processor (180) can control the operation or function of the device, or perform data processing, functions, or operations related to an application installed on the device.

[0073] The sensor module (130) described above may include at least one of a proximity sensor, an illumination sensor, a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor (G-sensor), a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor (IR sensor: infrared sensor), a fingerprint sensor, an ultrasonic sensor, an optical sensor (e.g., a camera), a microphone, an environmental sensor (e.g., including at least one of a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, a gas detection sensor), and a chemical sensor (e.g., a healthcare sensor, a biometric sensor, etc.). Meanwhile, the device may utilize information sensed from at least two of these sensors in combination.

[0074] The smart glasses (100) according to the present disclosure can detect (determine) risk factors in a basic research experiment or laboratory using a sensor module (130).

[0075] The interface module (140) serves as a passage for various types of external devices connected to the smart glasses (100) according to the present disclosure. This interface module (140) may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module (SIM), an audio I / O (Input / Output) port, a video I / O (Input / Output) port, and an earphone port. The device can perform appropriate control related to the external device connected to the interface section.

[0076] For example, an external display device (not shown) may be connected to the interface module (140) to output information related to an image captured from the camera module (120) of the smart glasses (100) and an object included in the image.

[0077] The output module (150) is for generating output related to sight, hearing, or touch, and may include at least one of a display (151), an augmented reality output module (152), a sound output module (153), and a haptic module (154).

[0078] The display (151) displays (outputs) information processed by the device. For example, the display (151) may display execution screen information of an application (e.g., an application) running on the device, or UI (User Interface) and GUI (Graphic User Interface) information based on such execution screen information.

[0079] The display (151) can be implemented as a transparent display, for example, and in this case, information related to the object can be output to the display area of ​​the display (151) in an AR manner.

[0080] The processor (180) can determine the output location based on the user's gaze direction so that information related to the object is displayed in an area overlapping with the object or in an area surrounding the object when the user looks at the object (200).

[0081] However, it is not limited to this, and the display (151) may be implemented as a display panel that prevents the real world from being seen through it. In this case, the display (151) may output an image received through the camera module (120) in real time under the control of the processor (180), and information related to an object included in the image may be overlaid on the image and output in a virtual reality (VR) manner.

[0082] Meanwhile, the augmented reality output module (152), referring to FIG. 1, can be formed to output information related to an object in a display area (152a) located in a space between the smart glasses (100) and the object (200), and, for example, can be output in a holographic manner. Additionally, the augmented reality output module (152) may be a projector that outputs light so that information related to the object is projected onto the actual object (200) or the area around the object.

[0083] The sound output module (153) can output audio data received through the communication module (110) or stored in the memory (170), or output sound signals related to functions performed by the device. The sound output module (153) may include a receiver, a speaker, a buzzer, etc.

[0084] A haptic module (154) generates various tactile effects that can be felt by the user. A typical example of a tactile effect generated by a haptic module is vibration. The intensity and pattern of vibration generated by the haptic module can be controlled by the user's selection or by the settings of the processor (180). In addition, the haptic module can generate various tactile effects in addition to vibration, such as effects caused by stimulation including a pin array moving vertically to the contact skin surface, air jet or suction force through a nozzle or suction port, brushing against the skin surface, contact with an electrode, electrostatic force, and effects caused by the reproduction of cold and hot sensations using an element capable of endothermic or exothermic heat.

[0085] The haptic module (154) can not only transmit tactile effects through direct contact, but can also be implemented so that the user can feel tactile effects through the sense of touch of fingers or arms. Two or more haptic modules may be provided depending on the configuration of the device.

[0086] The processor (180) can output information related to an object in the form of sound or vibration through the sound output module (153) or the haptic module (154), and can be configured to output a warning when a risk factor is detected through the sensor module (130).

[0087] The input module (160) is for inputting video information (or signal), audio information (or signal), data, or information input from a user, and may include at least one of at least one microphone (161) and at least one user input unit (162). Voice data collected by the input module (160) can be analyzed and processed into a user control command.

[0088] The microphone (161) processes an external acoustic signal into electrical voice data. The processed voice data can be utilized in various ways depending on the function (or application running) being performed on the device. Meanwhile, the microphone (161) may implement various noise removal algorithms to remove noise generated during the process of receiving an external acoustic signal.

[0089] The user input unit (162) is for receiving information from a user, and when information is input through the user input unit (162), the processor (180) can control the operation of the smart glasses (100) to correspond to the input information. Such user input units may include hardware physical keys (e.g., buttons, dome switches, jog wheels, jog switches, etc. located on at least one of the front, rear, and side of the smart glasses (100)) and software touch keys. As an example, the touch keys may consist of virtual keys, soft keys, or visual keys displayed on a touchscreen-type display (151) through software processing, or touch keys placed on a part other than the touchscreen. Meanwhile, the virtual keys or visual keys may have various forms and may be displayed on the touchscreen, for example, as graphics, text, icons, videos, or a combination thereof.

[0090] The memory (170) can store data supporting various functions of the device and programs for the operation of the processor, and can store input / output data (e.g., music files, still images, videos, etc.), and can store a number of application programs (or applications) running on the device, data for the operation of the device, and instructions. At least some of these application programs can be downloaded from an external server via wireless communication.

[0091] Additionally, the memory (170) may store at least one process (or task, operation, function, control method, process, data, algorithm, program, etc.) or processor for performing the method according to the present disclosure. Such at least one process may be performed under the control of the processor (180) and may refer to information used by the processor (180) to implement the method according to the present disclosure. For example, the memory may store at least one process for performing an operation that outputs information related to an object.

[0092] Such memory (170) may include at least one type of storage medium among flash memory type, hard disk type, SSD type (Solid State Disk type), SSD type (Silicon Disk Drive type), multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (random access memory; RAM), SRAM (static random access memory), ROM (read-only memory; ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk. Additionally, the memory may be a database that is separated from the device but connected via wired or wireless connection.

[0093] The processor (180) may be implemented with a memory (170) that stores data for an algorithm or a program that reproduces the algorithm for controlling the operation of components within the device, and at least one processor (not shown) that performs the process described in the present disclosure using the data stored in the memory. In this case, the memory (170) and the processor (180) may each be implemented as separate chips (or modules). However, not limited thereto, the memory (170) and the processor (180) may be implemented as a single chip (or single module).

[0094] The processor (180) of the smart glasses (100) according to the present disclosure may include an artificial intelligence model (182) as previously described. The artificial intelligence model (182) may be the artificial intelligence model described in the present disclosure. The artificial intelligence model (182) may be trained to recognize an object included in an image received through a camera module (120) in the field of basic research, determine the type of the object, and determine (determine) the state of the object. Additionally, the artificial intelligence model (182) may be trained to provide an experiment protocol according to the type of object and the type of experiment when conducting an experiment on the object, and may be trained to provide management information for the management of the object.

[0095] The processor (180) may include at least one (or multiple) artificial intelligence models (182), and in the case of multiple artificial intelligence models (182), they may be specialized or lightweight artificial intelligence models for the purpose of providing information (experimentation or management or risk factor judgment, etc.) and for the type of object.

[0096] FIG. 3 is a flowchart for explaining a control method of smart glasses of the present disclosure, and FIGS. 4 to 7 are each conceptual diagrams for explaining the control method examined in FIG. 3.

[0097] In the present disclosure, a step of recognizing an object in an image captured through a camera module (120) is performed (S310).

[0098] Subsequently, in the present disclosure, a step is performed to determine information related to the recognized object using a pre-trained artificial intelligence model (S320).

[0099] Subsequently, in the present disclosure, a step of outputting information related to the determined object through an output module is performed (S330).

[0100] The processor (180) can recognize an object in a video captured through the camera module (120) and determine the type of the object. At this time, the processor (180) can extract one image (or frame) from the real-time video received through the camera module (120), input the extracted image into an artificial intelligence model (182), and receive information related to the object included in the video as an output value from the artificial intelligence model (182).

[0101] Referring to FIG. 4, information related to an object may include at least one of manual information (410) (or experimental protocol information) used in the experiment, experimental data (411), experimental data analysis information (412), risk factor information (413), information related to breeding management (414), and status information (415).

[0102] Additionally, information related to the object may be determined based on the object (200) itself included in the image captured through the camera module (120), or may be determined based on separate identification information (e.g., information used for management such as a QR code) (220).

[0103] The processor (180) can input the image containing the object into the artificial intelligence model when the object is maintained in a state where it is included for a predetermined time in the image received through the camera module (120), and receive information related to the object as an output value from the artificial intelligence model.

[0104] That is, the processor (180) can proceed with a process of outputting information related to the object by recognizing the object (200) being photographed for a predetermined period of time through the camera module (120) without using a separate hand as a control command.

[0105] Meanwhile, the smart glasses (100) of the present disclosure may include a plurality of previously trained artificial intelligence models as described above.

[0106] Multiple artificial intelligence models may be installed in the processor (180) or implemented to be accessible from an external device through the communication module (110).

[0107] The processor (180) can recognize the type of object included in the image received through the camera module (120). Based on the type of the recognized object, the processor (180) can receive information related to the object using different artificial intelligence models among the plurality of artificial intelligence models.

[0108] For example, as illustrated in FIG. 5(a), the processor (180) can control the output module (150) to output information related to the first type of object using the first artificial intelligence model (A) when the first type of object (501) is recognized in the image. Additionally, as illustrated in FIG. 5(b), the processor (180) can control the output module (150) to output information related to the second type of object (520) using the second artificial intelligence model (B) different from the first artificial intelligence model (A) when the second type of object (502) different from the first type is recognized in the image.

[0109] Meanwhile, the smart glasses (100) of the present disclosure may include a communication module (110) as previously described. The communication module (110) may be formed to receive location information of the smart glasses (100).

[0110] The processor (180) can control the output module (150) so that the information related to the output object changes even if the same object is recognized through the camera module (120), based on the location information received through the communication module (110).

[0111] The processor (180) can determine the location of the smart glasses (100) by receiving GPS information through a location information module included in the communication module (110), or determine the location information of the laboratory through a short-range communication module based on a BLE (Bluetooth Low Energy) transmission signal installed in each laboratory.

[0112] As illustrated in FIG. 6(a), the processor (180) can output first information (610) related to the object (600) through the output module (150) when the object (600) is recognized at a first location (or location A). As illustrated in FIG. 6(b), the processor (180) can output second information (620) different from the first information (610) to the output module (150) when the object (600) is recognized at a second location (or location B) different from the first location.

[0113] That is, the processor (180) can control the output module (150) so that different information is output depending on the location (or place) of the smart glasses, even if the same object (600) is recognized in the image received through the camera module (120).

[0114] For example, if the smart glasses (100) are located in a laboratory where an experiment on an object (e.g., a rabbit) is performed, the processor (180) can output information necessary to perform the experiment on the object (such as manual information used in the experiment or experiment protocol information) to the output module (150).

[0115] As another example, if the smart glasses (100) are located in a place where management of an object (e.g., a rabbit) is performed, the processor (180) can output information necessary for the management of the object (information related to breeding management or status information, etc.) to the output module (150).

[0116] The smart glasses (100) according to the present disclosure can provide an optimized hands-free function to provide optimized utilization in the field of research of the object.

[0117] To this end, the smart glasses (100) according to the present disclosure may include a microphone (161).

[0118] When the processor (180) receives a preset control command through the microphone (161) while information related to an object is output, it can store at least two of the following in combination in the memory (170): an image captured through the camera module (120), information related to the object output to the output module, and information input through the microphone.

[0119] For example, as illustrated in FIG. 7(a), the processor (180) can receive information (A information) input through the microphone after receiving a preset control command (e.g., a control command set to start recording, “record”) through the microphone, while information related to the object is output.

[0120] The processor (180) can store at least one of the following in memory (170): information input through a microphone (voice signal or information converted therefrom into text), video being captured through a camera module, and information output to an output module (information related to an object) after the above-mentioned preset control command (“record”) is set. As illustrated in FIG. 7(b), the processor (180) can store at least two of the following in combination in memory (170): information input through a microphone (710), video being captured through a camera module (700), and information output to an output module (information related to an object).

[0121] However, not limited to this, the processor (180) may store at least two of the image, information related to the object, and information input through the microphone in combination in memory (170) when a preset control command is received through the microphone (161), even if information related to the object is not output.

[0122] The processor (180) can execute a control command corresponding to a received voice when a voice corresponding to a plurality of preset control commands (e.g., literature search, input, automatic saving, transmission, sharing, deletion, etc.) is received through the microphone (161). Through this, the present invention allows various functions to be controlled via voice without using hands.

[0123] The smart glasses (100) according to the present disclosure can provide an interface that can immediately visualize and provide risk factors detected at the current location to a user wearing the smart glasses.

[0124] To this end, the processor (180) may include a sensor module (130) formed to sense risk factors, as previously described.

[0125] Here, risk factors may include all elements that pose a risk to the management of the experiment or object, and may refer to environmental factors that may exceed standards, such as gas, temperature, humidity, light intensity, noise, vibration, and odor.

[0126] As seen in FIG. 4, the processor (180) can control the output module (150) to output a warning alarm (413) based on the detection of a risk factor related to an object through the sensor module (130).

[0127] Sensing a risk factor related to the above object may mean that a value exceeding the standard range set for each environmental factor is detected.

[0128] The warning alarm may include not only warning notification information being output on the display (151), but also a warning alarm generated by sound or vibration through a sound output module or a haptic module.

[0129] Additionally, the smart glasses (100) of the present disclosure may control the communication module (110) so that information being output through the output module (150) is output to another smart glasses (100) located nearby. For example, the processor (180) may control the communication module (110) to transmit information related to the object currently being output to another smart glasses (100) when a preset gesture (or user input) is detected through the sensor module (130) while information related to the object is being output to the output module (150).

[0130] Through the configurations described in the present disclosure, the smart glasses (100) according to the present disclosure can improve experimental accuracy and efficiency. Specifically, by visualizing research data in real time, the accuracy of the experimental process can be increased and errors minimized. Specifically, the present disclosure can shorten the experimental period and maximize efficiency by providing an automated protocol guide. In addition, the present disclosure can support the learning and improvement of researchers. Specifically, by guiding complex experimental procedures step by step, it supports the learning of novice researchers and students, and enables actual field practice, thereby helping them effectively complete research skills and related education. In addition, the present disclosure can provide ethical alternatives for animal experiment research and work. Specifically, the present disclosure enables field practice and education for researchers, managers, etc., to be conducted remotely, thereby enabling the realization of the 3Rs (Replacement, Reduction, Refinement) of animal experiment research. In addition, the present disclosure can strengthen real-time collaboration and data sharing. Specifically, the present disclosure enables researchers to increase collaboration efficiency by sharing data with colleagues in real time, and provides the ability to check research progress and immediately incorporate feedback even from remote locations. Furthermore, the present disclosure has the effect of improving safety and reducing experimental failure rates. Specifically, the present disclosure can prevent safety accidents by providing real-time warnings and guidance to researchers in hazardous experimental environments, and can reduce experimental failure rates by accurately guiding and reproducing experimental conditions. Additionally, the present disclosure can support the systematic management and analysis of research data. Specifically, the present disclosure can support data-driven research decision-making by collecting, storing, and analyzing research data in real time, and can ensure experimental reproducibility and transparency through the automation of research records.

[0131] According to the present disclosure, smart glasses (100) can provide automatic data voice recognition, literature search, input and automatic saving, transmission and sharing functions when performing animal experiments, animal experiment technical support and AI coaching functions, real-time coaching, real-time verification of animal experiment manuals, autopsy, administration, blood collection, etc. in the basic research area.

[0132] According to the present disclosure, the smart glasses (100) can provide surgical and procedural guides in the field of clinical research. Specifically, they can improve the precision of surgical procedures by augmenting anatomical structures and procedural procedures in real time, and can provide step-by-step guides to researchers or medical staff practicing. Additionally, the present disclosure can perform clinical trial data management using the smart glasses (100). Specifically, they can collect and analyze accurate data by checking the biometric data of clinical participants in real time, and can be utilized as a tool to efficiently manage clinical trial procedures. Furthermore, the present disclosure can perform patient and guardian education and counseling using the smart glasses (100). Specifically, they can improve understanding by visually explaining the treatment process or drug mechanism of action to patients and guardians, improve the efficiency of education related to all of the above matters, and maximize intuitive efficiency and reduce the workload of clinical research and medical care through automatic storage, sharing, and automatic analysis of all data.

[0133] According to the present disclosure, smart glasses (100) can provide hazardous material management and laboratory safety functions in the research safety domain. Specifically, the present disclosure can use smart glasses (100) to provide guidance and education on the safe use of chemical and biological hazardous materials during experiments, and can provide safety warnings and real-time response guidelines during the use of experimental equipment. The present disclosure can use smart glasses (100) to respond to emergency situations and can provide immediate emergency response guides and a communication system in the event of an accident in the laboratory. In addition, the present disclosure can use smart glasses (100) to monitor the location and status of researchers in real time to request rescue quickly, and can visually warn of radiation exposure levels in relation to radiation and infectious material safety, provide protection guidelines when handling infectious materials, and provide education related to all of the above matters.

[0134] According to the aforementioned means for solving the problem of the present disclosure, the present disclosure makes it possible to provide optimized data in real time in the field of basic research using smart glasses.

[0135] In addition, the present disclosure can provide customized information based on the object being viewed by the user, can provide an experiment protocol while performing an object experiment, and can efficiently manage data related to the object while performing object management.

[0136] In addition, the present disclosure provides the effect of guiding experienced and novice practitioners to perform treatments at a similar level by providing information related to the object in real time.

[0137] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below.

[0138] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. The instructions may be stored in the form of program code and, when executed by a processor, may generate a program module to perform the operation of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0139] Computer-readable recording media include all types of recording media that store instructions that can be decoded by a computer. Examples include ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, optical data storage devices, etc.

[0140] As described above, the disclosed embodiments have been explained with reference to the attached drawings. Those skilled in the art will understand that the present disclosure may be practiced in forms different from the disclosed embodiments without changing the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be interpreted restrictively.

Claims

1. Camera module; An output module that outputs information related to an object captured through the above camera module; Memory storing at least one process for performing an operation to output information related to the above object; and It includes a processor that performs an operation of outputting information related to the object based on at least one process, and The above processor is, Recognizing the object in the image captured through the camera module above, and Information related to the above-mentioned recognized object is determined using a pre-trained artificial intelligence model, and Smart glasses that control the output module so that information related to the determined object is output.

2. In Paragraph 1, Information related to the above object includes smart glasses comprising at least one of manual information used in the experiment, experimental protocol information, experimental data, experimental data analysis information, risk factor information, information related to rearing management, and status information.

3. In Paragraph 1, The above processor is, A smart glass that, when the object is maintained in a state of being included for a predetermined time in an image received through the camera module, inputs the image containing the object into the artificial intelligence model and receives information related to the object as an output value from the artificial intelligence model.

4. In Paragraph 3, The aforementioned previously trained artificial intelligence model includes a plurality of artificial intelligence models, and The above processor is, Recognizing the type of object included in the image received through the above camera module, and Smart glasses that receive information related to an object using different artificial intelligence models among the plurality of artificial intelligence models based on the type of the recognized object.

5. In Paragraph 4, The above processor is, When a first type of object is recognized in the above image, the output module is controlled to output information related to the first type of object using a first artificial intelligence model, and Smart glasses that control the output module to output information related to the second type of object using a second artificial intelligence model different from the first artificial intelligence model when a second type of object different from the first type is recognized in the above image.

6. In Paragraph 1, It further includes a communication module that receives location information, and The above processor is, Smart glasses that control the output module so that, based on location information received through the communication module, information related to the output object changes even if the same object is recognized through the camera module.

7. In Paragraph 6, The above processor is, When an object is recognized at a first location, first information related to the object is output to the output module, and A smart glass that outputs second information related to the object, different from the first information, to the output module when the object is recognized at a second location different from the first location.

8. In Paragraph 1, Includes additional microphones, The above processor is, A smart glass that stores at least two of the following in combination in the memory: an image captured through the camera module, information related to the object output to the output module, and information input through the microphone, when a preset control command is received through the microphone while information related to the object is output.

9. In Paragraph 1, It further includes a sensor module formed to sense risk factors, and The above processor is, Smart glasses that control the output module to output a warning alarm based on the detection of a risk factor related to an object through the sensor module.

10. In a method for controlling smart glasses, A step of recognizing an object in an image captured through a camera module; A step of determining information related to the above-mentioned recognized object using a pre-trained artificial intelligence model; and A method for controlling smart glasses, comprising the step of outputting information related to the determined object to an output module.

11. In Paragraph 10, The above-mentioned recognition step is, If the object is maintained in a state where it is included for a predetermined period of time in the image received through the camera module, the image containing the object is input into the artificial intelligence model, and The above-mentioned determining step is, A method for controlling smart glasses that receives information related to the object as an output value from the artificial intelligence model.

12. In Paragraph 11, The aforementioned previously trained artificial intelligence model includes a plurality of artificial intelligence models, and The above-mentioned recognition step recognizes the type of object included in the image received through the camera module, and The above-mentioned determining step is, A method for controlling smart glasses that receives information related to an object using different artificial intelligence models among the plurality of artificial intelligence models based on the type of the recognized object.

13. In Paragraph 12, The above outputting step is, When a first type of object is recognized in the above image, the output module is controlled to output information related to the first type of object using a first artificial intelligence model, and A method for controlling smart glasses, wherein when a second type of object different from the first type is recognized in the above image, the output module is controlled to output information related to the second type of object using a second artificial intelligence model different from the first artificial intelligence model.

14. In Paragraph 10, The method further includes a step of controlling the output module such that, based on location information received through the communication module, the information related to the output object changes even if the same object is recognized through the camera module. The above-mentioned controlling step is, When an object is recognized at a first location, first information related to the object is output to the output module, and A control method for smart glasses that outputs second information related to the object, different from the first information, to the output module when the object is recognized at a second location different from the first location.

15. In Paragraph 10, When information related to the object is output, and a preset control command is received through a microphone, a step of storing at least two of the image captured through the camera module, the information related to the object output to the output module, and the information input through the microphone in combination in memory; and A method for controlling smart glasses, further comprising the step of controlling the output module to output a warning alarm based on the detection of a risk factor related to an object through the sensor module.

Images