Electronic device and method for providing alarm

Abstract

A method for providing an alarm message, according to the present disclosure, comprises an operation of acquiring an image by using a first camera of an HMD device. The image includes at least one object including a first electronic device. The method according to the present disclosure comprises the operations of: acquiring situation information related to an HMD device by using at least one sensor of the HMD device; acquiring alarm data related to an alarm generated by a first electronic device; applying the situation information and the alarm data to a first artificial intelligence model so as to acquire alarm information labeled with additional information related to the alarm; applying the situation information and the alarm data to a second artificial intelligence model trained to determine whether to output the alarm message, thereby determining whether to output an alarm message related to the alarm information; applying the alarm information labeled with the additional information to a third artificial intelligence model trained to generate the alarm message, thereby generating the alarm message related to the alarm information; and outputting the alarm message on the basis of determining to output the alarm message.

Description

Electronic device and method for providing an alarm

[0001] The present disclosure relates to an electronic device and method for providing an alarm.

[0002] Recently, wearable electronic devices that provide extended reality (XR) services, including augmented reality (AR), virtual reality (VR), or mixed reality (MR), are being developed. For example, users can enjoy various XR services such as cameras, games, video streaming, or navigation while wearing a head-mounted display (HMD) type wearable electronic device on their head or XR glasses on their face.

[0003] Recently, artificial intelligence systems capable of achieving human-level intelligence are being utilized in various fields. Unlike conventional rule-based smart systems, artificial intelligence systems are systems in which machines learn, make judgments, and become smarter on their own. As artificial intelligence systems improve in recognition accuracy and gain a more accurate understanding of user preferences with continued use, existing rule-based smart systems are gradually being replaced by deep learning-based artificial intelligence systems.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art in relation to the present disclosure.

[0005] A method for a head-mounted display (HMD) device to provide an alarm message according to the present disclosure may include an operation of acquiring an image using a first camera of the HMD device. The image may include at least one object including a first electronic device. The method may include an operation of acquiring situation information related to the HMD device using at least one sensor of the HMD device. The method may include an operation of acquiring alarm data regarding an alarm generated by the first electronic device, and may include an operation of acquiring alarm information labeled with additional information related to the alarm by applying the situation information and the alarm data to a first artificial intelligence model, and may include an operation of determining whether to output an alarm message related to the alarm information by applying the situation information and the alarm data to a second artificial intelligence model trained to determine whether to output an alarm message, and may include an operation of generating an alarm message related to the alarm information by applying the alarm information labeled with additional information to a third artificial intelligence model trained to generate an alarm message, and may include an operation of outputting the alarm message based on the decision to output the alarm message.

[0006] A head-mounted display (HMD) device according to the present disclosure may include a first camera, at least one sensor, at least one processor, and a memory for storing instructions. The instructions may be executed individually or collectively by the at least one processor to enable the HMD device to acquire an image using the first camera. The image may include at least one object including a first electronic device. The instructions may be executed individually or collectively by the at least one processor to enable the HMD device to acquire situational information related to the HMD device using the at least one sensor, and to acquire alarm data regarding an alarm generated for the first electronic device. By applying the situational information and the alarm data to a first artificial intelligence model, the HMD device may acquire alarm information with additional information related to the alarm labeled. By applying the situational information and the alarm data to a second artificial intelligence model trained to determine whether to provide an alarm message, the HMD device may determine whether to output an alarm message related to the alarm information. The above commands are executed individually or collectively by the at least one processor, so that the HMD device can generate an alarm message related to the alarm information by applying the alarm information labeled with the additional information to a third artificial intelligence model trained for generating an alarm message, and can output the alarm message based on a decision to output the alarm message.

[0007] FIG. 1 is a diagram illustrating an overview of a method for an HMD device to provide an alarm to a user according to the present disclosure.

[0008] FIG. 2 is a diagram illustrating the structure of an HMD device according to one embodiment.

[0009] FIG. 3 is a flowchart illustrating the process of generating and outputting an alarm message in an HMD device according to one embodiment.

[0010] FIG. 4 is a flowchart illustrating the process of acquiring situational information using a camera and a sensor in an HMD device according to one embodiment.

[0011] FIG. 5 is a diagram showing an example of acquiring situational information using a camera and a sensor in an HMD device according to one embodiment.

[0012] FIG. 6 is a block diagram illustrating the process of acquiring situational information by inputting image and sensing values ​​into a fourth artificial intelligence model in an HMD device according to one embodiment.

[0013] FIG. 7 is a flowchart illustrating the process of acquiring alarm data based on data received from an external electronic device in an HMD device according to one embodiment.

[0014] FIG. 8 is a diagram illustrating an example of a situation in which alarm data is acquired based on data received from an external electronic device in an HMD device according to one embodiment.

[0015] FIG. 9 is a block diagram illustrating the process of acquiring alarm data by inputting data received from an external electronic device into a fifth artificial intelligence model in an HMD device according to one embodiment.

[0016] FIG. 10 is a flowchart illustrating the process of labeling additional information related to an alarm into alarm data in an HMD device according to one embodiment.

[0017] FIG. 11 is a block diagram illustrating the process of labeling additional information related to an alarm into the alarm data by inputting the alarm data into a first artificial intelligence model in an HMD device according to one embodiment.

[0018] FIG. 12 is a flowchart illustrating the process of determining whether to output an alarm message in an HMD device according to one embodiment.

[0019] FIG. 13 is a block diagram illustrating the process of determining whether to output an alarm message by inputting situation information and alarm data into a second artificial intelligence model in an HMD device according to one embodiment.

[0020] FIG. 14 is a flowchart illustrating the process of generating an alarm message in an HMD device according to one embodiment.

[0021] FIG. 15 is a block diagram illustrating the process of generating an alarm message by inputting alarm information labeled with additional information into a third artificial intelligence model in an HMD device according to one embodiment.

[0022] FIG. 16 is a flowchart illustrating the process of obtaining user feedback information regarding an output alarm message in an HMD device according to one embodiment, and adjusting weights within a network of a third artificial intelligence model based on the obtained feedback information.

[0023] FIG. 17 is a flowchart illustrating a specific process of generating and outputting an alarm message in an HMD device according to one embodiment.

[0024] FIG. 18 is an example drawing for explaining a specific embodiment of generating and outputting an alarm message in an HMD device according to one embodiment.

[0025] FIG. 19 is an example drawing for explaining a specific embodiment of generating and outputting an alarm message in an HMD device according to one embodiment.

[0026] FIG. 20 is an example drawing for explaining a specific embodiment of generating and outputting an alarm message in an HMD device according to one embodiment.

[0027] FIG. 21 is an example drawing for explaining a specific embodiment of generating and outputting an alarm message in an HMD device according to one embodiment.

[0028] FIG. 22 is a block diagram showing artificial intelligence models of an HMD device, an external electronic device, and a server that can be used to generate and output an alarm message in one embodiment of the present disclosure.

[0029] FIG. 23 is a block diagram of an electronic device in a network environment according to various embodiments.

[0030] FIG. 24 is a diagram showing a system including a generative artificial intelligence model according to one embodiment.

[0031] Hereinafter, embodiments are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.

[0032] The terms used in this disclosure are described in the form of general terms currently in use, considering the functions mentioned in this disclosure; however, they may mean various other terms depending on the intent of those skilled in the art, case law, or the emergence of new technologies. Therefore, the terms used in this disclosure should not be interpreted solely by their names, but should be interpreted based on the meaning of the terms and the overall content of this disclosure.

[0033] Additionally, terms such as the first, second, third, ..., Nth may be used to describe various components, but the components should not be limited by these terms. These terms are used for the purpose of distinguishing one component from another.

[0034] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "electrically connected" with other components interposed between them. Furthermore, when a part is described as "including" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0035] Phrases such as "in one embodiment" appearing in various places in this disclosure do not necessarily refer to the same embodiment.

[0036] One embodiment of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a specific function. Additionally, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented as algorithms executed on one or more processors. Furthermore, the present disclosure may employ prior art for electronic configuration, signal processing, and / or data processing. Terms such as “mechanism,” “element,” “means,” and “configuration” may be used broadly and are not limited to mechanical and physical configurations.

[0037] Furthermore, the connecting lines or connecting members between the components depicted in the drawings are merely illustrative of functional connections and / or physical or circuit connections. In the actual device, connections between components may be represented by various alternative or added functional connections, physical connections, or circuit connections.

[0038] The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding is a technology that recognizes, applies, and processes human language and text, and includes natural language processing, machine translation, dialogue systems, question answering, and speech recognition / synthesis. Visual understanding is a technology that perceives and processes objects like human vision, and includes object recognition, object tracking, image search, person recognition, scene understanding, spatial understanding, and image enhancement. Inference and prediction is a technology that judges information to logically infer and predict, and includes knowledge / probability-based inference, optimization prediction, preference-based planning, and recommendation. Knowledge representation is a technology that automatically processes human experiential information into knowledge data, and includes knowledge construction (data generation / classification) and knowledge management (data utilization). Motion control is a technology that controls the autonomous driving of vehicles and the movement of robots, and includes motion control (navigation, collision, driving) and manipulation control (behavior control).

[0039] Functions related to artificial intelligence according to the present disclosure may be operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0040] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a basic artificial intelligence model is trained using a number of training data by a learning algorithm, thereby creating predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives). Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0041] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0042] The three-dimensional XR space provided by the HMD device in the present disclosure may be implemented by displaying a virtual image containing virtual objects, but is not limited thereto. For example, the three-dimensional XR space may be implemented by displaying a combination of a real image and a virtual image of a real scene containing real-world objects. For example, the three-dimensional XR space may be implemented by overlaying a virtual image onto a physical environment space of the real world that is viewed through the lens of an electronic device.

[0043] In the present disclosure, the head-mounted display (HMD) device (100) may be a wearable device. The HMD device (100) may be, for example, a head-mounted display (HMD) worn on the head or XR glasses worn on the face, but is not limited thereto.

[0044] In the present disclosure, an external electronic device (e.g., the vacuum cleaner (800) of FIG. 8) may be an electronic device other than the HMD device (100). The external electronic device may be a device capable of transmitting and receiving data in relation to the HMD device (100). However, it is not limited thereto. The external electronic device may be a plurality of devices or a single device. For example, the external electronic device may include a smartphone, a home appliance, or a smart home device.

[0045] In the present disclosure, alarm data may be data containing information related to an alarm. Alarm data may be data representing an alarm related to an external electronic device. Alarm data may include data representing an alarm generated by an external electronic device and may be generated by an external electronic device. Alarm data may be data generated by an HMD device based on the state of an external electronic device and may be data representing an alarm related to an external electronic device. However, it is not limited thereto. By labeling additional information to the alarm data, alarm information labeled with additional information may be generated.

[0046] In the present disclosure, context information may include information regarding a situation related to an electronic device and / or an external electronic device. Context information may include various information including the user's environment, behavior, and the state of the device.

[0047] Contextual information may include information related to physical environment information. Physical environment information may include information related to the surrounding environment detected through cameras, sensors, or other devices. For example, it may include the location and status of sofas, chairs, tables, and home appliances (air conditioners, refrigerators, vacuum cleaners, etc.), and whether these objects are in use (e.g., whether a washing machine is operating). In addition, lighting conditions (distinction between day and night, whether lights are on) and GPS and Wi-Fi-based location information (e.g., whether the user is located in the living room, kitchen, or bathroom) may also be utilized as physical environment information.

[0048] Contextual information may include information related to the user's state information. The user's state information may include information indicating the user's current activity and health status. The user's state information may include whether the user is performing specific actions such as cleaning, cooking, exercising, or reading, as well as hand movements, gaze direction, body posture (standing, sitting, lying down), and movements; it may also include information related to the user's current state identified by analyzing such data. Furthermore, biometric information measured by sensors, such as heart rate, body temperature, and step count, as well as the user's stress index and fatigue level, and scheduled events (meetings, appointments, etc.) linked via calendars or applications, may also be included in the contextual information as user state information.

[0049] Context information may include information related to the state of the device. Information related to the state of the device may include information indicating the state of the device used by the user. For example, the context information may include the battery level of an XR device or smartphone, whether it is being charged, network connection status (whether Wi-Fi, Bluetooth, or 5G is connected and its quality), and environmental information such as temperature, humidity, and noise detected by sensors built into the device.

[0050] Contextual information may include time and weather information. Time and weather information can be used to understand the user's temporal context. For example, time information such as the current time (morning, noon, evening, dawn), day of the week (weekday, weekend, holiday), and the timing of specific events (birthday, anniversary, etc.) may be included in the contextual information. Time information can be utilized to predict the user's activity patterns, and information related to the predicted activity patterns may also be included in the contextual information. Additionally, weather information such as outdoor temperature, humidity, rain, snow, strong winds, and fine dust concentration may also be included in the contextual information.

[0051] Contextual information may include user preferences and past behavior information. User preferences and past behavior information may be used to provide a personalized user experience. For example, an HMD device (100) may analyze devices or services frequently used by the user to identify personalized preferences and evaluate appropriateness by utilizing past response records to specific notification types (e.g., exercise notifications, health notifications, etc.). For example, notification types that the user has previously ignored or closed may be used to exclude them from the next notification or to lower their importance.

[0052] Context information may include data related to an external electronic device (e.g., the vacuum cleaner (800) of FIG. 8). Data related to the external electronic device may include information regarding interaction with IoT (Internet of Things) devices and external information. For example, context information may include data such as the status and usage information of smart home devices (smart lights, smart speakers, thermostats, etc.), the inventory status of a refrigerator (e.g., whether milk or vegetables are out of stock), delivery notifications, traffic conditions, whether public transportation is delayed, breaking news, or disaster alerts.

[0053] Contextual information may include data related to sound and acoustics. Sound and acoustic information can be used to analyze the environment based on ambient sound and noise levels. For example, if specific sounds such as ambient conversation, TV sounds, music, doorbell sounds, or crying infants are detected, information related to these sounds may be included in the contextual information. For example, information regarding the level of ambient noise may be included in the contextual information.

[0054] The HMD device and method according to the present disclosure can provide the most suitable alarm to the user by comprehensively analyzing and combining situational information. For example, when the user is cleaning, an alert regarding the replacement of an air conditioner filter or the replacement cycle of a vacuum cleaner filter may be provided, and while cooking, the cooking time, appropriate temperature alert, refrigerator inventory information, etc., may be displayed.

[0055] The present disclosure will be described in detail below with reference to the attached drawings.

[0056] FIG. 1 is a diagram illustrating an overview of a method in which an HMD device (100) according to the present disclosure provides an alarm to a user.

[0057] The present disclosure may provide a method for providing an alarm to a user and an HMD device (100) for providing an alarm. In one embodiment, the HMD device (100) may display an XR space through a display (101).

[0058] An HMD device (100) according to one embodiment can acquire image and situation information using a camera and a sensor. The HMD device (100) can acquire alarm data related to an alarm based on the acquired image and situation information. The HMD device (100) can acquire alarm information labeled with additional information related to an alarm by inputting the acquired alarm data and situation information into a first artificial intelligence model (110), and can store the same. The HMD device (100) can generate an alarm message (103) to be provided to a user by inputting the alarm information labeled with additional information into a third artificial intelligence model (130) to be described later. The HMD device (100) can determine whether to output an alarm message (103) by inputting alarm data related to an external electronic device (e.g., (800) of FIG. 8) into a second artificial intelligence model (120), and can output an alarm message (103) based on the decision to output an alarm message (103).

[0059] An HMD device (100) according to one embodiment may obtain user feedback information regarding an output alarm message (103) and apply the user feedback information to a second artificial intelligence model (120). The second artificial intelligence model (120) may determine whether to output an alarm message that is subsequently generated based on the user feedback information received. For example, if the HMD device (100) obtains user feedback information that ignores or deletes the output alarm message (103), the obtained feedback information may be applied to the second artificial intelligence model (120), and accordingly, the probability that an alarm message that is highly related to the output alarm message (103) will be output subsequently may be reduced.

[0060] The HMD device (100) and method according to the present disclosure can provide the most suitable alarm to the user by comprehensively analyzing and combining situational information. In addition, the appropriateness of the next notification can be continuously improved by receiving positive or negative feedback regarding the notification from the user. The present disclosure can provide an HMD device (100) and a notification provision method that enhance user experience satisfaction through this process.

[0061] FIG. 2 is a drawing illustrating the structure of an HMD device (100) according to one embodiment.

[0062] Referring to FIG. 2, the HMD device (100) may include a main frame (210), a display (211, 212), and a wearable part (221, 222).

[0063] The main frame (210) may be worn on at least a part of the user's head (e.g., face) and may be supported on the user's face by various components. The main frame (210) may include a first surface facing a first direction and a second surface facing a second direction opposite to the first direction. According to one embodiment, the main frame (210) may be made of a material light enough to allow the user to feel a comfortable fit. For example, the main frame (210) may be made of plastic. Additionally, the main frame (210) may further include at least one of various materials, e.g., glass, ceramic, metal (e.g., aluminum), or metal alloy (e.g., steel, stainless steel, titanium, or magnesium alloy), for strength or aesthetics. FIG. 2 shows an HMD device (100) in the form of glasses, and the main frame (210) shows a structure including a rim, a bar, and a bridge of glasses, and such an HMD device (100) can be referred to as AR glasses.

[0064] The above displays (211, 212) may be exposed through the second surface of the main frame (210). According to one embodiment, the first display (211) among the displays (211, 212) may be placed on the right edge of the main frame (210) so as to face the user's right eye, and the second display (212) among the displays (211, 212) may be placed on the left edge of the main frame (210) so as to face the user's left eye. According to one embodiment, either the first display (211) or the second display (212) may be omitted.

[0065] The above display (211, 212) may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), a light emitting diode (LED) on silicon (LEDoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). Although not illustrated, if the display (211, 212) is one of a liquid crystal display, a digital mirror device, or a liquid crystal display, the HMD device (100) may include a light source that irradiates light onto a screen output area of ​​the display (211, 212). In another embodiment, if the display (211, 212) can generate light on its own, for example, one of an organic light emitting diode or a micro LED, the HMD device (100) may provide a good quality image to the user without including a separate light source. In one embodiment, if the display (211, 212) is implemented as an organic light-emitting diode or a micro LED, a light source is unnecessary, so the HMD device (100) can be made lighter.

[0066] A transparent member (213, 214) may be disposed on the first surface of the main frame (210). The transparent member (213, 214) may be formed from a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent. According to one embodiment, the first transparent member (213) among the transparent members (213, 214) may be disposed on the right edge of the main frame (210), and the second transparent member (214) among the transparent members (213, 214) may be disposed on the left edge of the main frame (210). According to various embodiments, when the display (211, 212) is transparent, the transparent member (213, 214) may be disposed in a position facing the user's eyes to form a screen display. According to one embodiment, either the first transparent member (213) or the second transparent member (214) may be omitted.

[0067] The above displays (211, 212) may include an optical waveguide (215). The optical waveguide (215) may serve to transmit a light source generated by the displays (211, 212) to the user's eyes. The optical waveguide (215) may be made of glass, plastic, or polymer and may include a nano pattern formed on some internal or external surface, for example, a grating structure in the shape of a polygon or curve. According to one embodiment, light incident on one end of the waveguide may propagate within the optical waveguide (215) by the nano pattern and be provided to the user. Additionally, the optical waveguide (215), composed of a free-form prism, may provide the incident light to the user through a reflective mirror. The optical waveguide (215) may include at least one diffractive element (e.g., DOE (Diffractive Optical Element) or HOE (Holographic Optical Element)) or a reflective element (e.g., a reflective mirror). The optical waveguide (215) may guide display light emitted from a light source to the user's eye using at least one diffractive element or reflective element included in the optical waveguide (215).

[0068] According to one embodiment, the diffraction element may include an input optical member (216) and an output optical member (not shown). For example, the input optical member (216) may represent an input grating area, and the output optical member may represent an output grating area. The input grating area may serve as an input terminal that diffracts (or reflects) light output from the display (211, 212) (e.g., Micro LED) to transmit light to the transparent member (213, 214) of the screen display unit. The output grating area may serve as an outlet that diffracts (or reflects) light transmitted to the transparent member (213, 214) of the optical waveguide (215) to the user's eye.

[0069] According to one embodiment, the reflection element may include a total internal reflection optical element or a total internal reflection waveguide for total internal reflection (TIR). For example, total internal reflection is a method of inducing light, which may mean creating an angle of incidence such that light (e.g., an image) input through the input grating area is reflected 100% from one side (e.g., a specific side) of the optical waveguide (215) and transmitted 100% to the output grating area.

[0070] According to one embodiment, light emitted from the display (211, 212) can be guided along a light path to the optical waveguide (215) through the input optical member (216). Additionally, light traveling within the optical waveguide (215) can be guided toward the user's eyes through the output optical member. The screen display can be determined based on the light emitted toward the eyes.

[0071] The above-mentioned wearable parts (221, 222) may be connected to a part of the main frame (210) so that a user can wear the HMD device (100). According to one embodiment, the first wearable part (221) of the wearable parts (221, 222) may be connected to the end of the right frame of the main frame (210) and extend in one direction (e.g., the second direction), and the second wearable part (222) of the wearable parts (221, 222) may be connected to the end of the left frame of the main frame (210) and extend in one direction (e.g., the second direction). According to one embodiment, the wearable parts (221, 222) may be connected to the main frame (210) through hinges (223, 224). FIGS. 2 and 3 show an HMD device (100) in the form of glasses, and the wearable parts (221, 222) may represent the temples of the glasses.

[0072] Each of the above-mentioned wearable parts (221, 222) may include an inner surface (221b, 222b) that contacts a part of the head (e.g., temples) when the HMD device (100) is worn on the user's head, and an outer surface (221a, 222a) that is opposite to the inner surface (221b, 222b).

[0073] According to one embodiment, the HMD device (100) may further include a lens (not shown). The lens may serve to adjust the focus so that the screen output to the display (211, 212) can be seen by the user's eyes. The lens may be composed of, for example, a Fresnel lens, a pancake lens, or a multichannel lens.

[0074] According to one embodiment, the HMD device (100) may include at least one camera. The at least one camera may include at least one of a first camera unit (231, 232), a second camera unit (233, 234), or a third camera unit (235).

[0075] The first camera unit (231, 232) may be used for 3 degrees of freedom (Dof) or 6 degrees of freedom (Dof) head tracking, hand detection and tracking, and spatial recognition. For example, the first camera unit (231, 232) may perform at least one of a spatial recognition function for 6 degrees of freedom, a simultaneous localization and mapping (SLAM) function through depth capture, or a user gesture recognition function. The first camera unit (231, 232) may be placed on the first surface of the main frame (210). According to one embodiment, one camera (231) of the first camera unit (231, 232) may be placed on the right edge of the main frame (210), and the other camera (232) of the first camera unit (231, 232) may be placed on the left edge of the main frame (210). According to one embodiment, the first camera unit (231, 232) may include a global shutter camera.

[0076] The second camera unit (233, 234) may be used to detect and track the user's pupils. For example, the second camera unit (233, 234) may be used to position the center of the image projected onto the AR glasses according to the direction the wearer's pupils are gazing. The second camera unit (233, 234) may be placed on the second surface of the main frame (210). According to one embodiment, one camera (233) of the second camera unit (233, 234) may be placed on the right edge of the main frame (210), and the other camera (234) of the second camera unit (233, 234) may be placed on the left edge of the main frame (210). According to one embodiment, the second camera unit (233, 234) may include a global shutter camera.

[0077] The third camera unit (235) may be used to photograph an external subject. The third camera unit (235) may be placed on the first surface of the main frame (210). According to one embodiment, the third camera unit (235) may be placed on the bridge of the main frame (210). According to one embodiment, the third camera unit (235) may include a high-resolution camera, referred to as HR (high resolution) or PV (photo video). According to one embodiment, the third camera unit (235) may include a color camera equipped with at least one function for acquiring high-quality images, such as an auto focus (AF) function and / or an optical image stabilization (OIS) function. According to one embodiment, the third camera unit (235) may include a global shutter camera or a rolling shutter camera.

[0078] According to one embodiment, the at least one camera may further include a fourth camera (not shown). The fourth camera may be used to detect and track a user's facial expression. The fourth camera may be positioned on the second surface of the main frame (210).

[0079] According to one embodiment, the HMD device (100) may include a light-emitting unit (241, 242) (e.g., an LED). The light-emitting unit (241, 242) may have different uses depending on its placement location. For example, when the light-emitting unit (241, 242) is placed around the main frame (210), the light-emitting unit (241, 242) may be used as an auxiliary means to facilitate gaze detection when tracking the user's eye movements with the second camera unit (233, 234), and in this case, an IR LED of infrared wavelength may be mainly used. As another example, when the light-emitting part (241, 242) is positioned adjacent to a camera (e.g., the first camera part (231, 232)) positioned around a hinge (223, 224) connecting the main frame (210) and the wearable part (221, 222) or a camera (e.g., the third camera part (235)) positioned around the bridge of the main frame (210), the light-emitting part (241, 242) can be used as a means to supplement ambient brightness when the camera is taking a picture. For example, the light-emitting part (241, 242) can emit light when it is not easy to detect a subject in a dark environment.

[0080] According to one embodiment, the HMD device (100) may include a microphone (251, 252, 253) and a speaker (254, 255). The microphone (251, 252, 253) may convert sound into an electrical signal and transmit the converted electrical signal to a processor. The speaker (254, 255) may output a voice signal. For example, the speaker (254, 255) may convert an electrical signal generated inside the HMD device (100) into sound and output it externally. According to one embodiment, the microphone (251, 252, 253) may be placed on the main frame (210) to be positioned close to the user's mouth, and the speaker (254, 255) may be placed on the wearable part (221, 222) to be positioned close to the user's ear.

[0081] According to one embodiment, the HMD device (100) may include a printed circuit board (261, 262). The printed circuit board (261, 262) may be placed inside the wearable part (221, 222) and may have various electronic components mounted on it. The electronic components mounted on the printed circuit board (261, 262) may include, for example, at least one of a processor, a memory, or a communication circuit. According to one embodiment, the printed circuit board (261, 262) may include a flexible printed circuit board. Additionally, the printed circuit board (261, 262) may transmit an electrical signal to at least one component of the HMD device (100) (e.g., a display (211, 212) or a camera (231, 232, 233, 234, 235)) through the flexible printed circuit board. According to one embodiment, the printed circuit board (261, 262) may be in the form of including a first substrate, a second substrate, and an interposer disposed between the first substrate and the second substrate.

[0082] The above processor (e.g., the processor (2320) of FIG. 23) can control at least one component of the HMD device (100) and can perform various data processing or operations. According to one embodiment, the processor can perform a function related to detecting the wearing state of the HMD device (100) by executing instructions stored in the memory included in the HMD device (100).

[0083] The memory (e.g., memory (2330) of FIG. 23) can store various data used by at least one component of the HMD device (100). According to one embodiment, the memory can store commands and data related to detecting the wearing state of the HMD device (100). In this case, the commands can be executed by the processor.

[0084] The communication circuit (e.g., the communication module (2390) of FIG. 23) can support communication between the HMD device (100) and an external electronic device. For example, the communication circuit can establish wired or wireless communication with the external electronic device according to a defined communication protocol and transmit or receive signals or data.

[0085] According to one embodiment, the HMD device (100) may include a battery (271, 272). The battery (271, 272) may supply power to at least one component of the HMD device (100). According to one embodiment, the battery (271, 272) may be disposed on the inner side of the end of the wearable part (221, 222).

[0086] The configuration and shape of the HMD device (100) are not limited thereto. According to various embodiments, the HMD device (100) may omit at least one of the components described above and may include at least one additional component. For example, the HMD device (100) may include at least one of an antenna or a sensor.

[0087] FIG. 3 is a flowchart illustrating the process of generating and outputting an alarm message in an HMD device (100) according to one embodiment.

[0088] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0089] Referring to identification number 310, an HMD device (100) according to one embodiment can acquire an image (e.g., image (105) of FIG. 1) including a first electronic device (e.g., vacuum cleaner (800) of FIG. 8) by using a camera (e.g., first camera unit (231, 232) of FIG. 2).

[0090] In one embodiment, an image may be obtained using a camera (e.g., the first camera unit (231, 232) of FIG. 2). An image may be obtained by photographing the front of the user using a camera (e.g., the first camera unit (231, 232) of FIG. 2). An image may be obtained by photographing the area where the user's gaze is directed through a camera of the HMD device (100). However, this is not limited thereto, and for example, an image may be obtained by photographing a plurality of directions including the front of the user using a camera (e.g., the first camera unit (231, 232) of FIG. 2).

[0091] In one embodiment, the image may include at least one object (e.g., the pot in FIG. 5). For example, at least one object included in the image may include at least one electronic device (e.g., the vacuum cleaner (800) in FIG. 8). For example, at least one electronic device included in the image (e.g., the vacuum cleaner (800) in FIG. 8) may be connected to the HMD device (100). For example, at least one electronic device included in the image (e.g., the vacuum cleaner (800) in FIG. 8) may be connected to the HMD device (100) so as to transmit and receive certain data with the HMD device (100). However, at least one object included in the image is not limited thereto and may include an object that is not an electronic device (e.g., the pot in FIG. 5).

[0092] Referring to identification number 320, an HMD device (100) according to one embodiment can acquire situation information using at least one sensor.

[0093] According to one embodiment, the HMD device (100) can acquire a sensed value through at least one sensor (e.g., the sensor module (2376) of FIG. 23). The HMD device (100) can acquire situational information based on the acquired sensed value.

[0094] In one embodiment, the HMD device (100) may acquire situational information. The situational information may include information related to the user's environment, behavior, the state of the HMD device (100), and an external electronic device (e.g., the vacuum cleaner (800) of FIG. 8).

[0095] In one embodiment, the HMD device (100) can acquire situational information based on an image acquired using a camera (e.g., the first camera unit (231, 232) of FIG. 2). The HMD device (100) can acquire situational information based on a sensing value acquired using a sensor (e.g., the sensor module (2376) of FIG. 23). The HMD device (100) can acquire situational information based on an image acquired using a camera (e.g., the first camera unit (231, 232) of FIG. 2) and a sensing value acquired using a sensor (e.g., the sensor module (2376) of FIG. 23). For example, the HMD device (100) can acquire situational information by applying at least one of the acquired image or the acquired sensing value to the fourth artificial intelligence model (140) to be described later.

[0096] Referring to identification number 330, an HMD device (100) according to one embodiment can obtain alarm data related to an alarm for a first electronic device (e.g., a vacuum cleaner (800) of FIG. 8).

[0097] In one embodiment, the HMD device (100) can acquire alarm data related to an alarm to be provided to a user. In one embodiment, the HMD device (100) can acquire alarm data related to an alarm to be provided to a user for a predetermined object. In one embodiment, the HMD device (100) can acquire alarm data related to an object included in an image acquired through a camera (e.g., the first camera unit (231, 232) of FIG. 2). In one embodiment, the HMD device (100) can acquire alarm data indicating an alarm generated by an external electronic device (e.g., the vacuum cleaner (800) of FIG. 8) which is an object included in the image. For example, the alarm data may include data generated by the external electronic device. However, it is not limited thereto.

[0098] In one embodiment, the HMD device (100) can receive alarm data generated from an external electronic device (e.g., a vacuum cleaner (800) of FIG. 8) included in an image acquired through a camera (e.g., a first camera unit (231, 232) of FIG. 2).

[0099] In one embodiment, the HMD device (100) may receive data related to the state of an external electronic device (e.g., the vacuum cleaner (800) of FIG. 8) from an external electronic device (e.g., the vacuum cleaner (800) of FIG. 8). The HMD device (100) may generate alarm data based on the received data related to the state of the external electronic device (e.g., the vacuum cleaner (800) of FIG. 8). For example, the HMD device (100) may input the data related to the state of the received external electronic device (e.g., the vacuum cleaner (800) of FIG. 8) into a fifth artificial intelligence model (150). Accordingly, the HMD device (100) may obtain alarm data output from the fifth artificial intelligence model (150).

[0100] Referring to identification number 340, an HMD device (100) according to one embodiment can obtain alarm information labeled with additional information related to the alarm by inputting situation information and alarm data into a first artificial intelligence model (110).

[0101] An HMD device (100) according to one embodiment can label additional information related to an alarm into the alarm data based on situation information and alarm data. An HMD device (100) according to one embodiment can input situation information and alarm data into a first artificial intelligence model (110). The first artificial intelligence model (110), having received the situation information and alarm data, can label additional information related to the alarm into the alarm data. For example, the additional information related to the alarm may include keywords related to the alarm data.

[0102] In one embodiment, the HMD device (100) may obtain content previously stored by the user in relation to alarm data. For example, the HMD device (100) may input situation information, alarm data, and content previously stored by the user into the first artificial intelligence model (110). For example, the first artificial intelligence model (110), which receives the situation information, alarm data, and content previously stored by the user, may label additional information related to the alarm into the alarm data. Labeling additional information into the alarm data may include, for example, tagging additional information into the notification data or matching and storing additional information into the notification data. Additionally, for example, additional information may be labeled into the notification data in the form of metadata, but is not limited thereto.

[0103] Referring to identification number 360, an HMD device (100) according to one embodiment can generate an alarm message by inputting alarm information labeled with additional information into a third artificial intelligence model (130).

[0104] An HMD device (100) according to one embodiment may input alarm information labeled with additional information into a third artificial intelligence model (130). In one embodiment, the third artificial intelligence model (130) may be trained to receive alarm information labeled with additional information through a first artificial intelligence model (110) and to generate notification text or content in a form that a user can understand. The third artificial intelligence model (130) may generate only an alarm message, or it may analyze data and generate results to deliver useful and meaningful information to the user along with the alarm message.

[0105] Referring to identification number 350, an HMD device (100) according to one embodiment can determine whether to output an alarm message by inputting alarm data into a second artificial intelligence model (120).

[0106] The second artificial intelligence model (120) may be trained to analyze the user's needs and determine whether to output a predetermined alarm message based on the analysis results. The HMD device (100) may apply the user's state information, situation information, and alarm data to the second artificial intelligence model (120). The second artificial intelligence model (120) may receive the user's state information, situation information, and alarm data and decide whether to output an alarm message or not to output it.

[0107] Referring to identification number 370, an HMD device (100) according to one embodiment may output an alarm message based on a decision to output an alarm message. An HMD device (100) according to one embodiment may display an alarm message through a display (e.g., the display (211, 212) of FIG. 2). For example, the alarm message may be output through the display in the form of a pop-up message. For example, if an object related to the alarm message (e.g., the vacuum cleaner (800) of FIG. 8) is displayed through the display, the alarm message may be displayed through a display area corresponding to the object related to the alarm message.

[0108] FIG. 4 is a flowchart illustrating the process of acquiring situational information using a camera and a sensor in an HMD device according to one embodiment. FIG. 5 is a diagram illustrating an example of acquiring situational information using a camera and a sensor in an HMD device (100) according to one embodiment. FIG. 6 is a block diagram for explaining the process of acquiring situational information by inputting an image and a sensing value into a fourth artificial intelligence model (140) in an HMD device (100) according to one embodiment.

[0109] Referring to identification number 410 in FIG. 4, the HMD device (100) can acquire an image using a camera (e.g., the first camera unit (231, 232) in FIG. 2), and the HMD device (100) can acquire at least one sensing value using a sensor (e.g., the sensor module (2376) in FIG. 23). According to one embodiment, the HMD device (100) can acquire at least one sensing value using a sensor (e.g., the sensor module (2376) in FIG. 23). The sensor (e.g., the sensor module (2376) in FIG. 23) can detect the operating state of the electronic device (2301) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, a sensor (e.g., sensor module (2376) of FIG. 23) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor. For example, the sensor may be included in the HMD device (100) or may be included in an electronic device outside the HMD device (100). Referring to FIGS. 5 and 6, for example, the HMD device (100) may acquire an image using a camera (e.g., the first camera unit (231, 232) of FIG. 2), and may acquire the user's gaze direction identified through a gaze detection sensor, the current location identified through a GPS sensor, the temperature of the place where the gaze direction is facing identified through a temperature detection sensor, and Wi-Fi information connected through a communication module.

[0110] Referring to identification number 420, in one embodiment, the HMD device (100) can acquire situational information by inputting acquired images and sensing values ​​into a fourth artificial intelligence model (140). The fourth artificial intelligence model (140) may be an artificial intelligence model trained to receive at least one of an image or a sensing value and output situational information. The situational information may include information collected to provide an alarm to the user. The situational information may include various information including the user's environment, behavior, and the state of the device. For example, referring to FIGS. 5 and 6, the HMD device (100) may input to the fourth artificial intelligence model (140) an image acquired using a camera (e.g., the first camera unit (231, 232) of FIG. 2), the user's gaze direction identified through a gaze detection sensor, the current location identified through a GPS sensor, the temperature of the place where the gaze direction is facing identified through a temperature detection sensor, and Wi-Fi information connected through a communication module. The fourth artificial intelligence model (140) can output situation information based on input images and sensing values, such that the user is currently cooking in the kitchen at home, the user is currently frying carrots, and the internal temperature of the pot is 180°C.

[0111] FIG. 7 is a flowchart illustrating the process of acquiring alarm data based on data received from an external electronic device in an HMD device according to one embodiment. FIG. 8 is a diagram illustrating an example of a situation in which alarm data is acquired based on data received from an external electronic device in an HMD device according to one embodiment. FIG. 9 is a block diagram for explaining the process of acquiring alarm data by inputting data received from an external electronic device into a fifth artificial intelligence model in an HMD device according to one embodiment.

[0112] Referring to identification number 710, an HMD device (100) according to one embodiment may receive data related to the status of an external electronic device from an external electronic device (e.g., a vacuum cleaner (800) of FIG. 8). For example, data related to the status of an external electronic device may include data related to the remaining battery level of the external electronic device, whether it is being charged, the network connection status (whether Wi-Fi, Bluetooth, 5G connection is present and quality), whether a component replacement period has arrived, whether an error has occurred, and environmental data such as temperature, humidity, and noise detected through a sensor embedded in the external electronic device. For example, referring to FIG. 8 and FIG. 9, the HMD device (100) may receive data from the vacuum cleaner (800) that the vacuum cleaner (800) is currently operating, and may receive data from an air purifier that the level of fine dust concentration has increased.

[0113] For example, data related to the status of an external electronic device may include alarm data related to an alarm generated by the external electronic device. For example, the external electronic device may generate an alarm to be provided to the user regarding the status of the external electronic device. In this case, the external electronic device may transmit alarm data related to the generated alarm to the HMD device (100). For example, the HMD device (100) may receive alarm data related to an alarm that the washing cycle of the washing machine has ended from the washing machine.

[0114] Referring to identification number 720, an HMD device (100) according to one embodiment may input data related to the state of a received external electronic device into a fifth artificial intelligence model (150). The fifth artificial intelligence model (150) may be trained to receive data related to the state of an external electronic device, recognize the current context, and output context information. For example, the fifth artificial intelligence model (150) may distinguish and analyze objects, environments, surrounding objects, user behavior, etc. For example, the fifth artificial intelligence model (150) may recognize whether the user is at home, in an office, or in a public place and output context information appropriate thereto. Referring to FIGS. 8 and 9, for example, the HMD device (100) may receive data from a vacuum cleaner (800) that the vacuum cleaner (800) is currently operating and receive data from an air purifier that the fine dust concentration has increased, and apply the received data to the fifth artificial intelligence model (150).

[0115] For example, the HMD device (100) can receive alarm data related to an alarm generated by an external electronic device and input the received alarm data into the fifth artificial intelligence model (150). The fifth artificial intelligence model (150) can receive the received alarm data and analyze at least one of the content of the generated alarm, the state of the external electronic device, or the surrounding environment. For example, a washing machine, which is an external electronic device, can generate an alarm indicating that the laundry needs to be removed from the washing tub because the washing cycle of the washing machine has ended. The HMD device (100) can receive alarm data related to the alarm generated by the washing machine. The HMD device (100) can apply the alarm data received from the washing machine to the fifth artificial intelligence model (150).

[0116] Referring to identification number 730, an HMD device (100) according to one embodiment can acquire alarm data output from a fifth artificial intelligence model. In one embodiment, the fifth artificial intelligence model (150) can receive data related to the state of an external electronic device, recognize the current situation, and output situation information. The HMD device (100) can acquire the situation information output from the fifth artificial intelligence model as alarm data. For example, referring to FIGS. 8 and 9, the HMD device (100) can receive data from a vacuum cleaner (800) that the vacuum cleaner (800) is currently operating and data from an air purifier that the fine dust concentration has increased, and apply the received data to the fifth artificial intelligence model (150). Accordingly, the fifth artificial intelligence model (150) can output situation information that the vacuum cleaner is currently operating in the house and that the air pollution level has increased after cleaning. The fifth artificial intelligence model (150) can output situational information stating that the air pollution level has increased as the vacuum cleaner is operated in the house, and therefore the filter of the vacuum cleaner (800) needs to be replaced. The HMD device (100) can obtain the situational information output from the fifth artificial intelligence model (150) as alarm data.

[0117] For example, the fifth artificial intelligence model (150) can receive alarm data generated from an external electronic device, identify a situation related to the external electronic device, and output situation information suitable for the identified situation. For example, the fifth artificial intelligence model (150) can receive alarm data related to an alarm indicating that laundry needs to be removed from the washing tub because the washing course of the washing machine has ended. In this case, the fifth artificial intelligence model (150) can identify a situation related to whether there is an additional washing course entered into the washing machine or whether a drying course of the dryer is scheduled. The fifth artificial intelligence model (150) can also receive data indicating that there was no additional user input to perform a separate washing course after the washing course is completed, and that the drying course of the dryer is scheduled to start at the time the washing course of the washing machine ends. In this case, the fifth artificial intelligence model (150) can output situation information indicating that laundry needs to be removed from the washing machine. The HMD device (100) can obtain the situation information output from the fifth artificial intelligence model (150) as alarm data.

[0118] Referring to identification number 740, an HMD device (100) according to one embodiment may receive alarm data related to an alarm generated by an external electronic device from an external electronic device. For example, an external electronic device (e.g., a vacuum cleaner (800) of FIG. 8) may generate an alarm to be provided to a user regarding the status of the external electronic device. The HMD device (100) may receive alarm data including an alarm generated by an external electronic device and data related thereto. Referring to FIG. 8, for example, the HMD device (100) may receive alarm data related to an alarm that the time has come to replace the filter of the vacuum cleaner (800) and an alarm related to the filter replacement cycle from the vacuum cleaner (800). Alternatively, the HMD device (100) may receive an alarm related to the charging status of the vacuum cleaner (800) from the vacuum cleaner (800).

[0119] FIG. 10 is a flowchart illustrating the process of labeling additional information related to an alarm to alarm data in an HMD device according to one embodiment. FIG. 11 is a block diagram illustrating the process of labeling additional information related to an alarm to alarm data by inputting alarm data into a first artificial intelligence model in an HMD device according to one embodiment.

[0120] An HMD device (100) according to one embodiment can label additional information related to an alarm onto the alarm data based on situation information and alarm data. By labeling additional information onto the alarm data, the HMD device (100) can obtain alarm information with additional information labeled thereon. For example, the additional information labeled onto the alarm data may include keywords related to the alarm data. For example, the additional information labeled onto the alarm data may include keywords related to an object related to the alarm.

[0121] An HMD device (100) according to one embodiment may apply situation information and alarm data to a first artificial intelligence model (110). The first artificial intelligence model (110) may be an artificial intelligence model trained to receive situation information and alarm data and to label additional information related to the alarm onto the alarm data and output it. The first artificial intelligence model (110), having received situation information and alarm data, may label additional information related to the alarm onto the alarm data. For example, with reference to FIG. 11, the first artificial intelligence model (110) may receive alarm data indicating that there is a need to replace the vacuum cleaner filter. Accordingly, the first artificial intelligence model (110) may label additional information including keywords such as 'home appliance, cleaning, vacuum cleaner, filter, replacement, notification time, valid notification time' onto the alarm information indicating that 'the vacuum cleaner filter needs to be replaced' and output it. In addition, the first artificial intelligence model (110) may also output information related to 'notification time' and 'valid notification time' by labeling them together with the alarm information.

[0122] Hereinafter, with reference to FIGS. 10 and FIGS. 11, I will explain the process of labeling additional information on alarm data by acquiring content previously stored by a user and inputting it into a first artificial intelligence model together with alarm data.

[0123] Referring to identification number 1010 in FIG. 10, in one embodiment, the HMD device (100) may acquire content previously stored by the user in relation to alarm data. In one embodiment, the content previously stored by the user may be pre-stored based on user input to the HMD device. The content previously stored by the user may include data related to the history of the user's use of the electronic device. For example, the content previously stored by the user may include message data transmitted and received between the user of the HMD device and other users. For example, the content previously stored by the user may include voice data recorded during a call between the user using the HMD device and another user. For example, the content previously stored by the user may include behavior data that tracks how the user behaves on a specific platform (application or webpage), such as content input, function usage records, search history, webpage visit history, activity history, exercise history, and viewing history. For example, content previously stored by the user may include images and videos taken by the user, and voice data recorded or stored by the user.

[0124] Referring to identification number 1020 in FIG. 10, an HMD device (100) according to one embodiment can input situation information, alarm data, and content previously stored by a user into a first artificial intelligence model (110). For example, referring to FIG. 11, the first artificial intelligence model (110) can receive alarm data indicating that the vacuum cleaner filter needs to be replaced, along with the content of a message received from 'Mom', which is “The vacuum cleaner smells, please check it.”

[0125] Referring to identification number 1030 in FIG. 10, an HMD device (100) according to one embodiment can label additional information related to an alarm into the alarm data through a first artificial intelligence model (110). For example, the first artificial intelligence model (110), which receives situation information, alarm data, and content previously stored by the user, can label additional information related to the alarm into the alarm data. Referring to FIG. 11, for example, the first artificial intelligence model (110) can receive the content of a message received from 'Mom', “The vacuum cleaner smells, please check it,” along with alarm data stating that the vacuum cleaner filter needs to be replaced. Accordingly, the first artificial intelligence model (110) can label the alarm information that "the vacuum cleaner filter needs to be replaced" with additional information such as "home appliance, cleaning, vacuum cleaner, filter, replacement, notification time, valid notification time" and additional information such as "message, family, mom, vacuum cleaner, smell, check, "The vacuum cleaner smelled, please check it."

[0126] FIG. 12 is a flowchart illustrating the process of determining whether to output an alarm message in an HMD device (100) according to one embodiment. FIG. 13 is a block diagram illustrating the process of determining whether to output an alarm message by inputting situation information and alarm data into a second artificial intelligence model (120) in an HMD device (100) according to one embodiment.

[0127] Referring to identification number 1210 in FIG. 12, an HMD device (100) according to one embodiment can acquire user status information. In one embodiment, the user status information may be included in situation information. The user status information may include information indicating the user's current activity or health status. For example, the user status information may include whether the user is performing specific actions such as cleaning, cooking, exercising, or reading, or hand movements, direction of gaze, body posture (standing, sitting, lying down), etc., and may include information related to the user's current state identified by analyzing such information. In addition, biometric information such as heart rate, body temperature, and step count measured through sensors, the user's stress index and fatigue level, and scheduled events (meetings, appointments, etc.) linked through a calendar or application may also be included in the situation information as user status information. For example, referring to FIG. 13, the HMD device (100) according to one embodiment may detect that the number of times the user sneezes has increased and acquire this as user status information.

[0128] Referring to identification number 1210 in FIG. 12, an HMD device (100) according to one embodiment may input user status information, situation information, and received alarm data into a second artificial intelligence model (120) to determine whether to output an alarm message. The second artificial intelligence model (120) may be trained to analyze the user's needs and determine whether to output a predetermined alarm message based on the analysis results. For example, the second artificial intelligence model (120) may analyze the need for an alarm message that the user is unaware of and the need for an alarm message that the user is aware of. Referring to FIG. 13, for example, the HMD device (100) may obtain user status information that the number of times the user sneezes has increased. Additionally, the HMD device (100) may obtain situation information that the vacuum cleaner is currently operating at home and that the air pollution level has increased since cleaning began. Additionally, the HMD device (100) may obtain alarm data that the usage period of the filter has arrived after replacing the vacuum cleaner filter. The HMD device (100) can apply user status information, situation information, and alarm data to the second artificial intelligence model (120). The second artificial intelligence model (120) can receive user status information, situation information, and alarm data and decide to output an alarm message to replace the vacuum cleaner filter.

[0129] FIG. 14 is a flowchart illustrating the process of generating an alarm message in an HMD device according to one embodiment. FIG. 15 is a block diagram illustrating the process of generating an alarm message in an HMD device according to one embodiment by inputting alarm information labeled with additional information into a third artificial intelligence model.

[0130] Referring to identification number 1410 in FIG. 14, an HMD device (100) according to one embodiment can search for guide information to be provided along with an alarm message in relation to additional information labeled in the alarm information. In one embodiment, the HMD device (100) can search for guide information related to an action to be performed by a user in accordance with the alarm. For example, the HMD device (100) can search for guide information related to an action to be performed by a user in accordance with the alarm from an external server. For example, the guide information may include information for ordering and / or replacing parts of an external electronic device related to the alarm. Referring to FIG. 15, for example, the HMD device (100) can search for guide information related to an action to be performed by a user in accordance with the alarm based on the alarm information labeled with additional information. For example, the HMD device (100) can search for guide information related to ‘how to replace the filter, regular ordering of the filter, lowest price for purchasing the filter’ based on alarm information that ‘the filter of the vacuum cleaner needs to be replaced’ which is labeled with additional information such as ‘home appliance, cleaning, vacuum cleaner, filter, replacement, notification time, valid notification time’.

[0131] Referring to identification number 1420 in FIG. 14, an HMD device (100) according to one embodiment may input guide information obtained from a search result into a third artificial intelligence model (130) along with alarm information labeled with additional information. However, the operation of searching for guide information in identification numbers 1410 and 1420 and inputting it into the third artificial intelligence model (130) is not necessarily required to be performed in the method and HMD device (100) according to the present disclosure. For example, referring to FIG. 15, the HMD device (100) may obtain search results related to 'filter replacement method, filter regular order, filter purchase lowest price' as guide information. The HMD device (100) may input the obtained guide information into the third artificial intelligence model (130) along with alarm information labeled with additional information.

[0132] Referring to identification number 1420 in FIG. 14, in one embodiment, an alarm message may be generated by a third artificial intelligence model (130). The third artificial intelligence model (130) according to one embodiment may generate guide information related to actions to be performed by the user in response to the alarm along with the alarm message. In one embodiment, the third artificial intelligence model (130) may combine the alarm data and guide information labeled through the first artificial intelligence model (110) to generate a notification phrase or content in a form that the user can understand. The third artificial intelligence model (130) may generate only the alarm message, or it may analyze data and generate results to convey useful and meaningful information to the user along with the alarm message. For example, referring to FIG. 15, the third artificial intelligence model (130) may receive alarm information labeled with additional information and generate an alarm message such as, "The vacuum cleaner filter needs to be replaced. Mom mentioned it recently, too." Additionally, for example, the third artificial intelligence model (130) can receive alarm information and guide information labeled with additional information together and generate an alarm message related to guide information related to ‘how to replace the filter, regular filter ordering, lowest price for purchasing the filter’ along with an alarm message saying, ‘You need to replace the vacuum cleaner filter. Mom mentioned it recently.’

[0133] FIG. 16 is a flowchart illustrating the process of obtaining user feedback information regarding an output alarm message in an HMD device (100) according to one embodiment, and adjusting weights within the network of a third artificial intelligence model (130) based on the obtained feedback information.

[0134] Referring to identification number 1610, the HMD device (100) can receive user feedback input regarding the output alarm message. In one embodiment, the user feedback input may include input received from the user regarding the output alarm message. The user feedback input may be received through the user's actions (e.g., hand gestures, touch input, etc.) or through the user's voice.

[0135] Referring to identification number 1620, the HMD device (100) can obtain user feedback information regarding an output alarm message. In one embodiment, the user feedback information may include information regarding whether input was received from the user within a predetermined time from the time the alarm message was received regarding the output alarm message, and if no input from the user was received regarding the output alarm message, it may include information that no input from the user was received regarding the output alarm message. For example, the user feedback information may include information regarding the type of input received from the user regarding the output alarm message.

[0136] Referring to identification number 1630, the HMD device (100) can adjust weights within the network of the second artificial intelligence model (120) based on acquired feedback information. In one embodiment, the HMD device (100) can identify at least one alarm message related to an output alarm message and adjust weights within the network of the second artificial intelligence model (120) for the identified at least one alarm message. For example, when input from a user selecting an output alarm message is received, the HMD device (100) can identify at least one alarm message related to the output alarm message and increase the weights within the network of the second artificial intelligence model (120) for the identified at least one alarm message. Accordingly, when determining whether to output an alarm message that is highly related to a previously output alarm message, the probability of an alarm message being output may increase. For example, if no user input is received regarding an output alarm message, the HMD device (100) may identify at least one alarm message related to the output alarm message and reduce the weight within the network of the second artificial intelligence model (120) for the identified at least one alarm message. Accordingly, when determining whether to output an alarm message highly related to the previously output alarm message, the likelihood of an alarm message being output may be reduced. For example, if user input to delete an output alarm message is received, the HMD device (100) may identify at least one alarm message related to the output alarm message and reduce the weight within the network of the second artificial intelligence model (120) for the identified at least one alarm message. Accordingly, when determining whether to output an alarm message highly related to the previously output alarm message, the likelihood of an alarm message being output may be reduced.

[0137] The present disclosure can continuously improve the appropriateness of the next notification by receiving positive or negative feedback regarding the notification from the user. The present disclosure can provide an HMD device (100) and a notification provision method that enhances user experience satisfaction through this process.

[0138] FIG. 17 is a flowchart illustrating a specific process of generating and outputting an alarm message in an HMD device according to one embodiment.

[0139] Referring to identification number 1701, the HMD device (100) can obtain situational information based on images obtained through the camera of the XR device and sensing values ​​provided by sensors. In identification number 1702, the HMD device (100) may receive alarm data related to an alarm for an external electronic device from an external electronic device (e.g., the vacuum cleaner (800) of FIG. 8). In identification number 1703, the HMD device (100) can obtain alarm data based on the obtained situational information or the received alarm data.

[0140] In identification number 1704, the HMD device (100) may decide whether to output an alarm message based on situation information and alarm data. The HMD device (100) may apply the situation information and alarm data to the second artificial intelligence model (120). The second artificial intelligence model (120) may decide whether to output an alarm message. In identification number 1705, if the HMD device (100) decides to output an alarm message, it may transmit a signal requesting the output of an alarm message.

[0141] In identification number 1706, the HMD device (100) can store alarm data received from an external electronic device. In identification number 1707, the HMD device (100) can label additional information related to the alarm on the alarm data. The HMD device (100) can label additional information related to the alarm on the alarm data by applying the alarm data to the first artificial intelligence model (110). By labeling additional information on the alarm data, alarm information labeled with additional information related to the alarm can be obtained. In identification number 1708, the HMD device (100) can label additional information related to guide information to be searched in relation to the alarm on the alarm data. In identification number 1709, the HMD device (100) can search for guide information to be provided along with the alarm message in relation to the additional information labeled on the alarm information. In identification number 1710, the HMD device (100) can generate an alarm message to be provided to the user based on the alarm information labeled with additional information and the guide information obtained as a search result. The HMD device (100) can obtain an alarm message to be provided to the user by applying at least one of alarm information or guide information to the third artificial intelligence model (130).

[0142] In identification number 1712, if the HMD device (100) decides to output an alarm message, it may store the generated alarm message according to a signal requesting the output of an alarm message. In identification number 1713, the HMD device (100) may obtain the generated alarm message as a result of searching for the alarm message in identification number 1711, and may output the obtained alarm message. For example, the HMD device (100) may output the alarm message through a display. In identification number 1714, the HMD device (100) may obtain user feedback information regarding the output alarm message. In identification number 1715, the HMD device (100) may adjust the weights within the network of the second artificial intelligence model (120) based on the user feedback information. Accordingly, when deciding whether to output an alarm message regarding subsequently generated alarm data, the probability of deciding to output the alarm data may be adjusted according to the adjusted weights.

[0143] The alarm providing method and HMD device according to the present disclosure are not required to perform all operations disclosed in FIG. 17, and may perform only some operations.

[0144] FIG. 18 is an example drawing for explaining a specific embodiment of generating and outputting an alarm message in an HMD device (100) according to one embodiment.

[0145] Referring to identification number 1810, the HMD device (100) can acquire an image through a first camera (e.g., the first camera unit (231, 232) of FIG. 2). The acquired image may include at least one object (1801, 1802, 1803). For example, the acquired image may include a bed (1801), a dressing table (1802), and a chair (1803). The HMD device (100) can identify the direction in which the gaze of a user wearing the HMD device (100) is directed through a second camera (e.g., the second camera unit (233, 234) of FIG. 2). The HMD device (100) can acquire situational information that the user is looking at the bed by applying the image acquired through the first camera and the data regarding the direction of the user's gaze identified through the second camera to the first artificial intelligence model (110). Additionally, the HMD device (100) can obtain data that the user has made a phone call about needing to purchase one more bed by using an external electronic device, such as a smartphone.

[0146] The HMD device (100) may decide to output an alarm message related to the additional purchase of a bed by applying acquired situational information and data acquired from a smartphone to a second artificial intelligence model (120). The smartphone may transmit alarm data related to an alarm to the HMD device (100) to provide information related to the replacement of the bed. The HMD device (100) may receive alarm data from the smartphone and apply the received alarm data to a third artificial intelligence model (130) to label the alarm data with additional information related to the alarm, such as 'bed, additional, purchase, call, mattress'. The HMD device (100) may search for guide information to be provided along with the alarm message in relation to 'bed, additional, purchase, call, mattress' and obtain guide information related to 'bed brand and purchase site'. The HMD device (100) can generate alarm messages based on alarm information and guide information by applying the bed brand 'simons' and the address of the website where the bed can be purchased to the third artificial intelligence model (130) based on the acquired guide information.

[0147] For example, referring to identification number 1820, the HMD device (100) can display an alarm message (1804) generated based on alarm information and guide information, such as ‘The bed brand is Simons. The purchase site is as follows. ...’, in the form of a pop-up message through the display. For example, the alarm message (1804) can be displayed in the form of a pop-up message on a display area corresponding to the bed (1801), which is an object related to the alarm message.

[0148] FIG. 19 is an example drawing for explaining a specific embodiment of generating and outputting an alarm message in an HMD device according to one embodiment.

[0149] The HMD device (100) can acquire images through a first camera (e.g., the first camera unit (231, 232) of FIG. 2). For example, the HMD device (100) can acquire sensing values ​​related to the user's gaze direction, the location of the HMD device (100), and connected Wi-Fi information through a second camera (e.g., the second camera unit (233, 234) of FIG. 2), a GPS sensor, or a communication module. The HMD device (100) can acquire situational information that the user is watching the vacuum cleaner (800) and the cleaning location by applying the acquired images and sensing values ​​to the fourth artificial intelligence model (140).

[0150] The HMD device (100) can receive from the vacuum cleaner (800) information that the vacuum cleaner (800) is currently operating and data that a predetermined time has elapsed since the time the filter of the vacuum cleaner (800) was replaced. Additionally, the HMD device (100) can receive from the vacuum cleaner (800) alarm data indicating that the filter of the vacuum cleaner (800) needs to be replaced. By applying the received alarm data and the acquired situational information to the fifth artificial intelligence model (150), the HMD device (100) can acquire alarm data indicating that the user currently wearing the HMD device (100) is operating the vacuum cleaner (800) at home and that the filter of the vacuum cleaner (800) needs to be replaced. By applying the situational information and alarm data to the second artificial intelligence model (120), the HMD device (100) can decide to output an alarm message at the current time when the vacuum cleaner (800) is operating.

[0151] The HMD device (100) can obtain message data received from 'Mom' by a user wearing the HMD device (100) regarding a vacuum cleaner filter, such as 'The vacuum cleaner smells, please check it.' The HMD device (100) can obtain alarm information labeled with additional information such as 'home appliance, cleaning, vacuum cleaner, filter, filter replacement, notification time, smell, Mom, family, message' by applying the alarm data received from the vacuum cleaner (800), the obtained message data, and the obtained situational information to the first artificial intelligence model (110). The HMD device (100) can generate an alarm message such as 'The vacuum cleaner filter needs to be replaced. I think Mom mentioned it recently, too.' by applying the alarm information labeled with additional information to the third artificial intelligence model (130).

[0152] The HMD device (100) can display the generated alarm message through a display (e.g., the display (211, 212) of FIG. 2). For example, the HMD device (100) can display the generated alarm message in the form of a pop-up message (1901) on a display area adjacent to a display area corresponding to the vacuum cleaner (800).

[0153] FIG. 20 is an example drawing for explaining a specific embodiment of generating and outputting an alarm message in an HMD device (100) according to one embodiment.

[0154] The HMD device (100) can acquire images through a first camera (e.g., the first camera unit (231, 232) of FIG. 2). For example, the HMD device (100) can acquire sensing values ​​related to the user's gaze direction, movement, the location of the HMD device (100), and connected Wi-Fi information through a second camera (e.g., the second camera unit (233, 234) of FIG. 2), an accelerometer, a gyroscope, a GPS sensor, or a communication module. The HMD device (100) can acquire situational information that the user is lying on a bed inside the house by applying the acquired images and sensing values ​​to a fourth artificial intelligence model (140).

[0155] The HMD device (100) can obtain alarm data indicating that the user currently wearing the HMD device (100) intends to sleep by applying the acquired situational information to the fifth artificial intelligence model (150). The HMD device (100) can decide to output an alarm message providing information related to the appropriate sleep time to the user by applying the situational information and alarm data to the second artificial intelligence model (120).

[0156] The HMD device (100) can obtain alarm information labeled with additional information such as 'sleep, nap, bed, noon' by applying alarm data and acquired situational information to the first artificial intelligence model (110). The HMD device (100) can generate an alarm message such as 'The appropriate nap time is 30 minutes' by applying the alarm information labeled with additional information to the third artificial intelligence model (130). The HMD device (100) can display the generated alarm message through a display (e.g., the display (211, 212) of FIG. 2).

[0157] For example, the HMD device (100) may display a generated alarm message through a display (e.g., the display (211, 212) of FIG. 2) and output a UI or message that allows the user to select whether to output another alarm related to the generated alarm message. For example, the HMD device (100) may display a generated alarm message, "The appropriate nap time is 30 minutes," and provide a UI through a display (e.g., the display (211, 212) of FIG. 2) that allows the user to select whether to provide an alarm after 30 minutes. For example, the HMD device (100) may display an alarm message, "The appropriate nap time is 30 minutes," and together with this, display a UI that includes a message, "Would you like to set the alarm bell to ring after 30 minutes? Y / N." In this case, the HMD device (100) can be set to output an alarm bell after 30 minutes as it receives input from the user to ring an alarm bell after 30 minutes.

[0158] FIG. 21 is an example drawing for explaining a specific embodiment of generating and outputting an alarm message in an HMD device according to one embodiment.

[0159] The HMD device (100) can acquire images through a first camera (e.g., the first camera unit (231, 232) of FIG. 2). For example, the HMD device (100) can acquire sensing values ​​related to the user's gaze direction, movement, the location of the HMD device (100), and connected Wi-Fi information through a second camera (e.g., the second camera unit (233, 234) of FIG. 2), an accelerometer, a gyroscope, a GPS sensor, or a communication module. The HMD device (100) can acquire situational information that the user is sitting on a sofa in the living room and looking around by applying the acquired images and sensing values ​​to a fourth artificial intelligence model (140). Additionally, the HMD device (100) can acquire data that the user is searching for furniture to place inside the living room by using an external electronic device, such as a smartphone.

[0160] The HMD device (100) can obtain alarm data indicating that the user currently wearing the HMD device (100) intends to purchase furniture other than a sofa to be placed in the living room by applying the acquired situation information and data to the fifth artificial intelligence model (150). The HMD device (100) can decide to output an alarm message recommending furniture that can be placed in each location within the living room by applying the situation information and alarm data to the second artificial intelligence model (120).

[0161] The HMD device (100) can obtain alarm information labeled with additional information such as 'furniture, flowerpot, chair, chest of drawers' by applying alarm data and acquired situational information to the first artificial intelligence model (110). The HMD device (100) can generate an alarm message related to recommended furniture that can be placed at each location as an alarm message by applying the alarm information labeled with additional information to the third artificial intelligence model (130). The HMD device (100) can display the generated alarm message through a display (e.g., the display (211, 212) of FIG. 2). For example, the HMD device (100) can display images of recommended furniture that can be placed at each location inside the living room as an alarm message through the display (e.g., the display (211, 212) of FIG. 2).

[0162] FIG. 22 is a block diagram showing artificial intelligence models of an HMD device (100), an external electronic device (1240) (e.g., a vacuum cleaner (800) of FIG. 8) and a server (1250) that can be used to generate and output an alarm message in one embodiment of the present disclosure.

[0163] According to one embodiment, the first artificial intelligence model (110), the second artificial intelligence model (120), the third artificial intelligence model (130), the fourth artificial intelligence model (140), and the fifth artificial intelligence model (150) are all stored in the HMD device (100) and can be directly accessed by the HMD device (100), but are not limited thereto. For example, the first artificial intelligence model (110) may be stored in the HMD device (100) and the second artificial intelligence model (120) may be stored in a server, the first artificial intelligence model (110) may be directly accessed by the HMD device (100), and the HMD device (100) may receive information output by the second artificial intelligence model (120) from the server by querying the server containing the second artificial intelligence model (120). For example, the first artificial intelligence model (110), the second artificial intelligence model (120), the third artificial intelligence model (130), the fourth artificial intelligence model (140), and the fifth artificial intelligence model (150) are all stored in a server, and the HMD device (100) can receive information output by the first artificial intelligence model (110), the second artificial intelligence model (120), the third artificial intelligence model (130), the fourth artificial intelligence model (140), and the fifth artificial intelligence model (150) from the server by querying the server that includes the first artificial intelligence model (110), the second artificial intelligence model (120), the third artificial intelligence model (130), the fourth artificial intelligence model (140), and the fifth artificial intelligence model (150). The query transmitted from the HMD device (100) to the server may include information input to the first artificial intelligence model (110), the second artificial intelligence model (120), the third artificial intelligence model (130), the fourth artificial intelligence model (140), and the fifth artificial intelligence model (150).

[0164] In the present disclosure, artificial intelligence models can each be trained to perform a predetermined role.

[0165] In one embodiment, the fourth artificial intelligence model (140) can track the current state and motion information of the user based on data such as the user's movement, behavior, and gaze. For example, by analyzing the user's movement trajectory, head and hand movements, and the direction of the gaze, it can determine in real time what actions the user is performing. Based on this data, it can provide information on the interaction between the user and the surrounding environment.

[0166] In one embodiment, the fifth artificial intelligence model (150) can recognize the current context based on information provided by the fourth artificial intelligence model (140) and can output context information. The fifth artificial intelligence model (150) can distinguish and analyze objects, environments, surrounding objects, and user behavior to define the space where the user is located and the state of interaction. For example, the fifth artificial intelligence model (150) can recognize whether the user is at home, in an office, or in a public place and provide information appropriate thereto.

[0167] In one embodiment, the second artificial intelligence model (120) can analyze the user's needs and determine whether to output a predetermined alarm message based thereon. The second artificial intelligence model (120) can output potential needs that the user is unaware of and immediate needs that the user clearly recognizes. For example, it can analyze and output potential needs, such as the time to replace an air purifier filter, and immediate needs, such as providing detailed information about the media currently being viewed.

[0168] In one embodiment, the first artificial intelligence model (110) receives context information and alarm data related to the alarm and can label additional information on the alarm data. The first artificial intelligence model (110) receives context information and alarm data and can cluster the alarm data based on the labeled additional information. The labeling AI can label additional information related to notification data and user data (e.g., photos, videos, messages, contacts). The labeled additional information can be used as basic data for analyzing the context related to the user and generating an appropriate alarm.

[0169] In one embodiment, the third artificial intelligence model (130) can combine data labeled through the first artificial intelligence model (110) with an external data source to generate a notification phrase or content in a form that the user can understand. The third artificial intelligence model (130) may generate only an alarm message, or it may analyze data and generate results to deliver useful and meaningful information to the user along with the alarm message.

[0170] For example, an electronic device (1240) outside the HMD device (100) can store a seventh artificial intelligence model (1241). The seventh artificial intelligence model (1241) can support operations of at least one of the first artificial intelligence model (110), the second artificial intelligence model (120), the third artificial intelligence model (130), the fourth artificial intelligence model (140), and the fifth artificial intelligence model (150). For example, the seventh artificial intelligence model (1241) can exchange data with a cloud server and assist in the operation of at least one of the first artificial intelligence model (110), the second artificial intelligence model (120), the third artificial intelligence model (130), the fourth artificial intelligence model (140), and the fifth artificial intelligence model (150).

[0171] For example, the server (1250) can store the eighth artificial intelligence model (1251). The eighth artificial intelligence model (1251) can perform training and updates on at least one of the first artificial intelligence model (110), the second artificial intelligence model (120), the third artificial intelligence model (130), the fourth artificial intelligence model (140), and the fifth artificial intelligence model (150). For example, the eighth artificial intelligence model (1251) can train at least one of the first artificial intelligence model (110), the second artificial intelligence model (120), the third artificial intelligence model (130), the fourth artificial intelligence model (140), and the fifth artificial intelligence model (150) with new data, or deploy an improved artificial intelligence model.

[0172] FIG. 23 is a block diagram of an electronic device (2301) in a network environment (2300) according to various embodiments.

[0173] Referring to FIG. 23, in a network environment (2300), an electronic device (2301) may communicate with an electronic device (2302) through a first network (2398) (e.g., a short-range wireless communication network) or with at least one of an electronic device (2304) or a server (2308) through a second network (2399) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (2301) may communicate with the electronic device (2304) through a server (2308). According to one embodiment, the electronic device (2301) may include a processor (2320), memory (2330), input module (2350), sound output module (2355), display module (2360), audio module (2370), sensor module (2376), interface (2377), connection terminal (2378), haptic module (2379), camera module (2380), power management module (2388), battery (2389), communication module (2390), subscriber identification module (2396), or antenna module (2397). In some embodiments, at least one of these components (e.g., connection terminal (2378)) may be omitted from the electronic device (2301), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (2376), camera module (2380), or antenna module (2397)) may be integrated into a single component (e.g., display module (2360)).

[0174] The processor (2320) can, for example, execute software (e.g., program (2340)) to control at least one other component (e.g., hardware or software component) of the electronic device (2301) connected to the processor (2320) and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (2320) can store commands or data received from other components (e.g., sensor module (2376) or communication module (2390)) in volatile memory (2332), process the commands or data stored in volatile memory (2332), and store the resulting data in non-volatile memory (2334). According to one embodiment, the processor (2320) may include a main processor (2321) (e.g., a central processing unit or an application processor) or an auxiliary processor (2323) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (2301) includes a main processor (2321) and an auxiliary processor (2323), the auxiliary processor (2323) may be configured to use less power than the main processor (2321) or to be specialized for a specified function. The auxiliary processor (2323) may be implemented separately from the main processor (2321) or as part thereof.

[0175] The auxiliary processor (2323) may control at least some of the functions or states associated with at least one component of the electronic device (2301) (e.g., display module (2360), sensor module (2376), or communication module (2390)) on behalf of the main processor (2321) while the main processor (2321) is in an inactive (e.g., sleep) state, or together with the main processor (2321) while the main processor (2321) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (2323) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (2380) or communication module (2390)). According to one embodiment, the auxiliary processor (2323) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (2301) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (2308)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0176] The memory (2330) can store various data used by at least one component of the electronic device (2301) (e.g., processor (2320) or sensor module (2376)). The data may include, for example, software (e.g., program (2340)) and input or output data for related commands. The memory (2330) may include volatile memory (2332) or non-volatile memory (2334).

[0177] The program (2340) may be stored as software in memory (2330) and may include, for example, an operating system (2342), middleware (2344), or an application (2346).

[0178] The input module (2350) can receive commands or data to be used for a component of the electronic device (2301) (e.g., processor (2320)) from outside the electronic device (2301) (e.g., user). The input module (2350) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0179] The sound output module (2355) can output a sound signal to the outside of the electronic device (2301). The sound output module (2355) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0180] The display module (2360) can visually provide information to an external (e.g., user) of the electronic device (2301). The display module (2360) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (2360) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0181] The audio module (2370) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (2370) can acquire sound through the input module (2350) or output sound through the sound output module (2355) or an external electronic device (e.g., electronic device (2302)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (2301).

[0182] The sensor module (2376) can detect the operating state of the electronic device (2301) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (2376) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0183] The interface (2377) may support one or more specified protocols that can be used for the electronic device (2301) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (2302)). According to one embodiment, the interface (2377) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0184] The connection terminal (2378) may include a connector through which the electronic device (2301) can be physically connected to an external electronic device (e.g., electronic device (2302)). According to one embodiment, the connection terminal (2378) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0185] The haptic module (2379) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (2379) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0186] The camera module (2380) can capture still images and video. According to one embodiment, the camera module (2380) may include one or more lenses, image sensors, image signal processors, or flashes.

[0187] The power management module (2388) can manage the power supplied to the electronic device (2301). According to one embodiment, the power management module (2388) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0188] The battery (2389) can supply power to at least one component of the electronic device (2301). According to one embodiment, the battery (2389) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0189] The communication module (2390) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (2301) and an external electronic device (e.g., electronic device (2302), electronic device (2304), or server (2308)), and the performance of communication through the established communication channel. The communication module (2390) may include one or more communication processors that operate independently of the processor (2320) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (2390) may include a wireless communication module (2392) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (2394) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (2304) via a first network (2398) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (2399) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (2392) can identify or authenticate the electronic device (2301) within a communication network such as the first network (2398) or the second network (2399) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (2396).

[0190] The wireless communication module (2392) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (2392) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (2392) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (2392) can support various requirements specified in the electronic device (2301), external electronic device (e.g., electronic device (2304)), or network system (e.g., second network (2399)). According to one embodiment, the wireless communication module (2392) can support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0191] An antenna module (2397) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (2397) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (2397) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (2398) or a second network (2399), may be selected from the plurality of antennas, for example, by a communication module (2390). A signal or power may be transmitted or received between the communication module (2390) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (2397).

[0192] According to various embodiments, the antenna module (2397) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0193] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0194] According to one embodiment, commands or data may be transmitted or received between the electronic device (2301) and an external electronic device (2304) through a server (2308) connected to a second network (2399). Each of the external electronic devices (2302, or 2304) may be the same or a different type of device as the electronic device (2301). According to one embodiment, all or part of the operations performed on the electronic device (2301) may be performed on one or more of the external electronic devices (2302, 2304, or 2308). For example, if the electronic device (2301) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (2301) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (2301). The electronic device (2301) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (2301) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (2304) may include an Internet of Things (IoT) device. The server (2308) may be an intelligent server using machine learning and / or neural networks.According to one embodiment, an external electronic device (2304) or server (2308) may be included within the second network (2399). The electronic device (2301) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0195] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0196] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0197] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0198] Various embodiments of the present document may be implemented as software (e.g., program (2340)) comprising one or more instructions stored in a storage medium (e.g., internal memory (2336) or external memory (2338)) readable by a machine (e.g., electronic device (2301)). For example, a processor (e.g., processor (2320)) of the machine (e.g., electronic device (2301)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0199] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0200] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0201] FIG. 24 is a diagram showing a system including a generative artificial intelligence model according to one embodiment.

[0202] In the present disclosure, at least one of the first artificial intelligence model (110), the second artificial intelligence model (120), the third artificial intelligence model (130), the fourth artificial intelligence model (140), the fifth artificial intelligence model (150), the seventh artificial intelligence model (1241), or the eighth artificial intelligence model (1251) may include the generative artificial intelligence model illustrated in FIG. 24.

[0203] Referring to FIG. 24, the User Query / Response Interface (2410) can receive user input. The user input may be in the form of natural language, images, and / or videos. Additionally, context information may be transmitted along with the user input. Context information may include various additional information at the time of user input. For example, information about the application currently being used by the user or the user's location information. Furthermore, user input may be in a mixed form of the aforementioned natural language, images, sounds, and context information. Additionally, user input may be in a non-natural language form, such as selecting a menu. The User Query / Response Interface (2410) can output results from a generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of actions requested by the user.

[0204] The AI ​​framework (2420) can receive user input and coordinate and control each component necessary to perform the user's intent based on the user's query.

[0205] User input received from the User Query / Response Interface (2410) can be sent to the Prompt design component (2421). The Prompt design component (2421) can be used to generate a prompt suitable for inputting user input into a Large Language Model (LLM) or Large Multimodal Models (LMM). The Prompt design component (2421) may be an AI component that uses machine learning algorithms or neural networks to develop better prompts over time. Based on user input, the Prompt design component (2421) can generate a prompt by accessing a knowledge component (e.g., knowledge repositories (2440)) containing user preference data, a prompt library, and prompt examples, and can pass the generated prompt to the LLM or LMM.

[0206] The API / Plug-in management component (2423) can perform the role of communicating with external information when there is a request for additional information when passing user input as input to a generative model. The API / Plug-in management component (2423) establishes a channel to communicate with the outside of the AI ​​Interface via the API, and through the established channel, it can enable access to various data sources (e.g., knowledge repositories (2440)). Additionally, if the application or service needs to perform an action that executes the user input as a final step rather than an intermediate result, the API / Plug-in management component (2423) can request that action from the application / service component (2430) via the API. The information obtained from the outside can be used to generate a prompt in the Prompt design component (2421) along with the user input, or it can be passed as input to the generative model.

[0207] The Refiner component (e.g., output modification component (2425)) allows for detailed tuning of the output from a generative model. For instance, the Refiner component can verify whether the content generated by LLM and / or LMM is irrelevant, contains biased content, or includes harmful content. Additionally, the Refiner component can determine the extent to which the output matches the user's desired outcome and, if necessary, proceed with additional processing. Furthermore, the Refiner component can configure and provide hints to the user to help avoid unwanted outputs.

[0208] A Generative AI Model (2450) generally refers to an artificial intelligence neural network that generates new forms of data based on user input information. A Generative AI Model (2450) may include models that generate images and / or models that generate language. Models that generate images include, but are not limited to, GANs (generative adversarial networks) and VAEs (variational autoencoders), and examples include Diffusion-based generative models that use VAEs and Transformer structures. Models that generate language are models trained to output the most statistically appropriate output value based on input values, and examples include models such as CHAT-GPT 3 and CHAT-GPT 4. There are also LMMs that can recognize various forms of data input, such as text, images, and voice, and generate new data corresponding to them.

[0209] According to one embodiment, the HMD device (100) of FIGS. 1 to 23 may be configured to include at least some of the User Query / Response Interface (2410) AI framework (2420), application / service component (2430), knowledge repositories (2440), or Generative AI Model (2450) of FIG. 24. According to one embodiment, at least some of the User Query / Response Interface (2410) AI framework (2420), application / service component (2430), knowledge repositories (2440), or Generative AI Model (2450) of FIG. 24 may be included in another electronic device (e.g., an external electronic device and / or server).

[0210] The following describes a method for providing alarm messages and an HMD device that can be implemented based on the above content.

[0211] A method for a head-mounted display (HMD) device to provide an alarm message according to the present disclosure may include an operation of acquiring an image using a first camera of the HMD device. The image may include at least one object including a first electronic device. The method may include an operation of acquiring situation information related to the HMD device using at least one sensor of the HMD device. The method may include an operation of acquiring alarm data regarding an alarm generated by the first electronic device, and may include an operation of acquiring alarm information labeled with additional information related to the alarm by applying the situation information and the alarm data to a first artificial intelligence model, and may include an operation of determining whether to output an alarm message related to the alarm information by applying the situation information and the alarm data to a second artificial intelligence model trained to determine whether to output an alarm message, and may include an operation of generating an alarm message related to the alarm information by applying the alarm information labeled with additional information to a third artificial intelligence model trained to generate an alarm message, and may include an operation of outputting the alarm message based on the decision to output the alarm message.

[0212] The above method may further include an operation of acquiring content pre-stored in the HMD device in relation to the alarm generated by the first electronic device, and the pre-stored content may be pre-stored based on user input to the HMD device. The operation of acquiring the alarm information may include an operation of acquiring the alarm information labeled with additional information related to the alarm by applying the situation information, the acquired content, and the alarm data to the first artificial intelligence model.

[0213] The operation of acquiring the above content may further include the operation of acquiring message data transmitted and received by the user of the HMD device with other users in relation to the alarm generated by the first electronic device.

[0214] The operation of acquiring the above alarm information may include the operation of acquiring the alarm information labeled with additional information related to the alarm by applying the above situation information, the above message data, and the above alarm data to the first artificial intelligence model.

[0215] The above at least one sensor may include a second camera that photographs the eyes of a user wearing the HMD device.

[0216] The above situation information may include gaze information related to the user's gaze obtained based on an image captured by the second camera.

[0217] The above situation information may include behavioral information related to the actions of a user wearing the HMD device.

[0218] The at least one sensor may include a sensor that identifies the location of the HMD device. The situation information may include location information related to the location of the HMD device detected using the sensor that detects the location of the HMD device.

[0219] The above method may include an operation of searching for guide information for ordering or replacing a component of the first electronic device related to the alarm from an external server, and may include an operation of applying the guide information together with the alarm information labeled with the additional information to the third artificial intelligence model.

[0220] The above method may include an operation of obtaining feedback information from a user regarding the output alarm message, and may further include an operation of adjusting weights within the network of the second artificial intelligence model based on the feedback information.

[0221] The operation of acquiring the above situation information may include the operation of acquiring the situation information by applying at least one sensing value acquired using the acquired image and at least one sensor to the fourth artificial intelligence model.

[0222] The above-mentioned fourth artificial intelligence model may be trained to output situational information of the HMD device by analyzing an image obtained using the camera and at least one sensing value obtained using the at least one sensor.

[0223] The above method may include an operation of acquiring the alarm data by applying data related to the state of at least one external electronic device received from at least one external electronic device including the first electronic device to a fifth artificial intelligence model, and the fifth artificial intelligence model may be trained to acquire the alarm data by analyzing the situation information and the data related to the state of the at least one external electronic device.

[0224] The operation of outputting the above alarm message may include the operation of displaying the above alarm message through a display area corresponding to the first object.

[0225] A head-mounted display (HMD) device according to the present disclosure may include a first camera, at least one sensor, at least one processor, and a memory for storing instructions.

[0226] The above instructions may be executed individually or collectively by the at least one processor to enable the HMD device to acquire an image using the first camera. The image may include at least one object including a first electronic device.

[0227] The above instructions are executed individually or collectively by the at least one processor, so that the HMD device can acquire situation information related to the HMD device using the at least one sensor and acquire alarm data regarding an alarm generated for the first electronic device, and by applying the situation information and the alarm data to a first artificial intelligence model, the alarm information with additional information related to the alarm labeled can be acquired, and by applying the situation information and the alarm data to a second artificial intelligence model trained to determine whether to provide an alarm message, the alarm message related to the alarm information can be output.

[0228] The above commands are executed individually or collectively by the at least one processor, so that the HMD device can generate an alarm message related to the alarm information by applying the alarm information labeled with the additional information to a third artificial intelligence model trained for generating an alarm message, and can output the alarm message based on a decision to output the alarm message.

[0229] The above instructions may be executed individually or collectively by the at least one processor to enable the HMD device to acquire content pre-stored in the HMD device in relation to the alarm generated by the first electronic device. The pre-stored content may be pre-stored based on user input to the HMD device.

[0230] The above commands are executed individually or collectively by the at least one processor, so that the HMD device can obtain the alarm information labeled with additional information related to the alarm by applying the situation information, the acquired content, and the alarm data to the first artificial intelligence model.

[0231] The above instructions are executed individually or collectively by the at least one processor, so that the HMD device can acquire message data transmitted and received by the user of the HMD device with other users in relation to the alarm generated by the first electronic device, and by applying the situation information, the message data, and the alarm data to the first artificial intelligence model, the alarm information labeled with additional information related to the alarm can be acquired.

[0232] The above at least one sensor may include a second camera that photographs the eyes of a user wearing the HMD device.

[0233] The above situation information may include gaze information related to the user's gaze obtained based on an image captured by the second camera.

[0234] The above situation information may include behavioral information related to the actions of a user wearing the HMD device.

[0235] The above at least one sensor may include a sensor that identifies the position of the HMD device.

[0236] The above situation information may include location information related to the location of the HMD device, detected using the sensor that detects the location of the HMD device.

[0237] The above commands may be executed individually or collectively by the at least one processor to enable the HMD device to obtain user feedback information regarding the output alarm message and to adjust weights within the network of the second artificial intelligence model based on the feedback information.

[0238] The above commands may be executed individually or collectively by the at least one processor, so that the HMD device may acquire the situation information by applying the acquired image and at least one sensing value acquired using the at least one sensor to the fourth artificial intelligence model.

[0239] The above-mentioned fourth artificial intelligence model may be trained to output situational information of the HMD device by analyzing an image obtained using the camera and at least one sensing value obtained using the at least one sensor.

[0240] The above instructions may be executed individually or collectively by the at least one processor to enable the HMD device to acquire the alarm data by applying data related to the state of the at least one external electronic device received from the at least one external electronic device including the first electronic device to the fifth artificial intelligence model.

[0241] The above-mentioned fifth artificial intelligence model may be trained to acquire the alarm data by analyzing the situation information and data related to the state of the at least one external electronic device.

[0242] Methods according to the claims or embodiments described in the specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0243] When implemented in software, a computer-readable storage medium may be provided for storing one or more programs (software modules). One or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. One or more programs include instructions that cause the electronic device to execute methods according to the claims or embodiments described in the specification of this disclosure.

[0244] In the present disclosure, the function or operation performed by an electronic device may be performed by one or more processors executing one or more instructions stored in memory. The function or operation of the electronic device mentioned in the present disclosure may be performed by a single processor executing one or more instructions, or by a combination of multiple processors executing one or more instructions. A processor mentioned in the present disclosure is understood to include a circuit for performing operations or controlling other components of the electronic device. For example, the one or more processors may include at least one of a central processing unit (CPU), a micro-processor unit (MPU), an application processor (AP), a communication processor (CP), a neural processing unit (NPU), a system on chip (SoC), an application-specific integrated circuit (ASIC), or an integrated circuit (IC) configured to execute one or more instructions. The one or more processors may be configured to perform the operation of the electronic device described above.

[0245] In the present disclosure, a program (software module, software) may be stored in a random access memory, a non-volatile memory including flash memory, a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic disc storage device, a compact disc-ROM (CD-ROM), digital versatile discs (DVDs), or other forms of optical storage devices, or a magnetic cassette. Alternatively, it may be stored in a memory composed of some or all of these. The memory may be composed of a single storage medium or a combination of multiple storage media. The one or more instructions may be stored in a single storage medium or distributed across multiple storage media.

[0246] Additionally, the above program may be stored on an attachable storage device that can be accessed via a communication network such as the Internet, Intranet, LAN (local area network), WLAN (wide LAN), or SAN (storage area network), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure through an external port. Additionally, a separate storage device on a communication network may be connected to a device performing an embodiment of the present disclosure.

[0247] In the specific embodiments of the present disclosure described above, the components included in the disclosure are expressed in a singular or plural form according to the specific embodiments presented. However, the singular or plural expression is selected to suit the situation presented for convenience of explanation, and the present disclosure is not limited to singular or plural components; even if a component is expressed in the plural form, it may be composed of a singular form, and even if a component is expressed in the singular form, it may be composed of a plural form.

[0248] Additionally, in the present disclosure, terms such as “part,” “module,” etc. may be a hardware component, such as a processor or circuit, and / or a software component executed by a hardware component, such as a processor.

[0249] "Parts" and "modules" may be implemented by a program that is stored on an addressable storage medium and can be executed by a processor. For example, "parts" and "modules" may be implemented by components such as software components, object-oriented software components, class components, and task components, as well as by processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0250] The specific embodiments described in this disclosure are merely examples and do not limit the scope of this disclosure in any way. For the sake of brevity, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted.

[0251] Additionally, in the present disclosure, “comprising at least one of a, b, or c” may mean “comprising only a, comprising only b, comprising only c, or comprising a combination of two or more (comprising a and b, comprising b and c, comprising a and c, or comprising all of a, b, and c).”

[0252] Meanwhile, although specific embodiments have been described in the detailed description of the present disclosure, it is understood that various modifications are possible within the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof.

Claims

1. A method for a head-mounted display (HMD) device to provide an alarm message, The operation of acquiring an image using a first camera of the HMD device, wherein the image includes at least one object including a first electronic device; An operation of acquiring situational information related to the HMD device using at least one sensor of the HMD device; An operation to acquire alarm data regarding an alarm generated by the first electronic device; An operation of obtaining alarm information labeled with additional information related to the alarm by applying the above situation information and the above alarm data to a first artificial intelligence model; An operation to determine whether to output an alarm message related to the alarm information by applying the above situation information and the above alarm data to a second artificial intelligence model trained to determine whether to output an alarm message; The operation of generating an alarm message related to the alarm information by applying the alarm information labeled with the above additional information to a third artificial intelligence model trained for generating an alarm message; and Based on the decision to output the above alarm message, the operation of outputting the above alarm message is included. method.

2. In Claim 1, The method further includes an operation of acquiring content pre-stored in the HMD device in relation to the alarm generated by the first electronic device, wherein the pre-stored content is pre-stored based on user input to the HMD device. The operation of acquiring the above alarm information is, A method comprising the operation of obtaining alarm information labeled with additional information related to the alarm by applying the above situation information, the above acquired content, and the above alarm data to the first artificial intelligence model, method.

3. In Claim 2, The operation of acquiring the above content is, In relation to the alarm generated by the first electronic device, the operation of acquiring message data transmitted and received by the user of the HMD device with other users is further included. The operation of acquiring the above alarm information is, The operation of obtaining the alarm information labeled with additional information related to the alarm by applying the above situation information, the above message data, and the above alarm data to the first artificial intelligence model, method.

4. In Claim 1, The above at least one sensor includes a second camera that photographs the eyes of a user wearing the HMD device, and The above situation information includes gaze information related to the user's gaze obtained based on an image captured by the second camera, method.

5. In Claim 1, The above situation information includes behavioral information related to the behavior of a user wearing the HMD device, method.

6. In Claim 1, The above at least one sensor includes a sensor that identifies the position of the HMD device, and The above situation information includes location information related to the location of the HMD device, detected using the sensor that detects the location of the HMD device. method.

7. In Claim 1, The operation of searching an external server for guide information for ordering or replacing a component of the first electronic device related to the above alarm; The operation of applying the above guide information to the above third artificial intelligence model together with the above alarm information labeled with the above additional information, method.

8. In Claim 1, Regarding the above-mentioned output alarm message, an operation to obtain feedback information from the user; and Further including an operation to adjust weights within the network of the second artificial intelligence model based on the above feedback information, method.

9. In Claim 1, The operation of acquiring the above situation information is, The method includes the operation of acquiring the situation information by applying the acquired image and at least one sensing value acquired using the at least one sensor to the fourth artificial intelligence model. The above-mentioned fourth artificial intelligence model is trained to output situational information of the HMD device by analyzing an image obtained using the camera and at least one sensing value obtained using the at least one sensor. method.

10. In Claim 1, The operation of acquiring alarm data by applying data related to the state of at least one external electronic device received from at least one external electronic device including the first electronic device to a fifth artificial intelligence model, wherein the fifth artificial intelligence model is trained to acquire alarm data by analyzing the situation information and data related to the state of the at least one external electronic device. method.

11. In Claim 1, The operation of outputting the above alarm message is, The operation of displaying the above alarm message through a display area corresponding to the first electronic device, method.

12. In a head-mounted display (HMD) device, First camera; At least one sensor; At least one processor; and It includes memory for storing instructions, The above instructions are executed individually or collectively by the at least one processor, so that the HMD device: An image is acquired using the first camera, and the image includes at least one object including a first electronic device; Using the above at least one sensor, situation information related to the HMD device is obtained, and Acquire alarm data regarding an alarm generated for the first electronic device, and By applying the above situation information and the above alarm data to the first artificial intelligence model, alarm information labeled with additional information related to the alarm is obtained, and By applying the above situation information and the above alarm data to a second artificial intelligence model trained to determine whether to provide an alarm message, it is determined whether to output an alarm message related to the above alarm information, and By applying the alarm information labeled with the above additional information to a third artificial intelligence model trained for alarm message generation, an alarm message related to the above alarm information is generated, and Based on the decision to output the above alarm message, the alarm message is to be output. HMD device.

13. In Claim 12, The above instructions are executed individually or collectively by the at least one processor, so that the HMD device: In relation to the alarm generated by the first electronic device, content previously stored in the HMD device is obtained, and the content previously stored is pre-stored based on user input to the HMD device. By applying the above situation information, the above acquired content, and the above alarm data to the above first artificial intelligence model, the alarm information labeled with additional information related to the alarm is obtained. HMD device.

14. In Claim 13, The above instructions are executed individually or collectively by the at least one processor, so that the HMD device: In relation to the alarm generated by the first electronic device, the user of the HMD device obtains message data transmitted and received with other users, and By applying the above situation information, the above message data, and the above alarm data to the above first artificial intelligence model, the alarm information labeled with additional information related to the alarm is obtained. HMD device.

15. In Claim 12, The above at least one sensor includes a second camera that photographs the eyes of a user wearing the HMD device, and The above situation information includes gaze information related to the user's gaze obtained based on an image captured by the second camera, HMD device.

Images