Electronic device, method, and non-transitory computer-readable storage medium for summarizing text by using sensor data

Abstract

An electronic device may comprise a memory storing instructions, one or more sensors, and at least one processor. The instructions may cause the electronic device to: identify an event for outputting a notification for first text of a first length; determine the length of text to be used for outputting the notification, by using sensor data acquired via the one or more sensors; acquire second text of a second length from the first text, on the basis of determining, as the second length, the length of the text to be used for outputting the notification; and output the acquired second text.

Description

Electronic device, method, and non-transient computer-readable storage medium for summarizing text using sensor data

[0001] The present disclosure relates to an electronic device, a method, and a non-transient computer-readable storage medium for summarizing text using sensor data.

[0002] The electronic device may include a plurality of sensors, including a camera, a microphone, a light sensor, and an accelerometer. The electronic device may acquire sensor data regarding the surrounding environment of the electronic device through the plurality of sensors. For example, the electronic device may acquire an image of the surrounding environment through the camera. For example, the electronic device may acquire an audio signal regarding the surrounding environment through the microphone.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure.

[0004] No claim or determination is made as to whether any of the foregoing can be applied as prior art related to the present disclosure.

[0005] An electronic device is described. The electronic device may include a memory comprising one or more storage media for storing instructions. The electronic device may include one or more sensors. The electronic device may include at least one processor comprising a processing circuit. The instructions may cause the electronic device to identify an event for outputting a notification for a first text of a first length when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to determine the length of the text to be used to output the notification by using sensor data obtained through the one or more sensors when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to obtain a second text of a second length from the first text based on determining the length of the text to be used to output the notification as a second length when executed individually or collectively by the at least one processor. The above instructions may cause the electronic device to output the acquired second text when executed individually or collectively by the at least one processor.

[0006] A method is provided. The method may be executed within an electronic device having one or more sensors. The method may include an operation of identifying an event for outputting a notification for a first text of a first length. The method may include an operation of determining the length of text to be used to output the notification using sensor data obtained through the one or more sensors. The method may include an operation of obtaining a second text of a second length from the first text based on determining the length of the text to be used to output the notification as a second length. The method may include an operation of outputting the obtained second text.

[0007] A non-transient computer-readable storage medium is provided. The non-transient computer-readable storage medium may store one or more programs. The one or more programs may include instructions that cause the electronic device to identify an event for outputting a notification for a first text of a first length when executed by an electronic device having one or more sensors. The one or more programs may include instructions that cause the electronic device to determine the length of the text to be used to output the notification by using sensor data obtained through the one or more sensors when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to obtain a second text of a second length from the first text based on determining the length of the text to be used to output the notification as a second length when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to output the obtained second text when executed by the electronic device.

[0008] Figure 1 illustrates an example of an electronic device that provides notifications.

[0009] Figure 2 is a simplified block diagram of an exemplary electronic device.

[0010] Figure 3 is a flowchart illustrating the operation of an electronic device that outputs text.

[0011] FIGS. 4a and 4b illustrate an exemplary operation of an electronic device that obtains a second text from a first text using a trained model.

[0012] Figure 5 is a flowchart illustrating the operation of an electronic device that determines the length of text using a prompt indicating the user's status.

[0013] Figure 6 is a flowchart illustrating the operation of an electronic device that activates a camera based on an audio signal.

[0014] FIG. 7 illustrates examples of operations performed within an electronic device and an external electronic device.

[0015] FIG. 8 is a block diagram of an electronic device in a network environment according to various embodiments.

[0016] FIG. 9a illustrates an example of a perspective view of a wearable device.

[0017] FIG. 9b illustrates an example of one or more pieces of hardware placed within a wearable device.

[0018] FIGS. 10a and 10b illustrate an example of the appearance of a wearable device.

[0019] FIG. 11 illustrates an example of a block diagram of a wearable device.

[0020] Figure 12 shows an example of a block diagram of an electronic device for displaying an image in virtual space.

[0021] Figure 13 is a schematic diagram of an exemplary AI system.

[0022] Figure 1 illustrates an example of an electronic device that provides notifications.

[0023] Referring to FIG. 1, a user (120) may use an electronic device (100) and an external electronic device (110). For example, the electronic device (100) may be referred to as a wearable device or a head-worn electronic device. For example, the electronic device (100) may be used to provide augmented reality (AR) or mixed reality (MR). For example, the electronic device (100) may include AR glasses. For example, the electronic device (100) may include a video see-through (VST) device.

[0024] For example, the electronic device (100) can identify an event to provide a notification. For example, the electronic device (100) can identify text (140) by said event. For example, the electronic device (100) can display or provide the text (140) through the display of the electronic device (100).

[0025] According to one embodiment, an electronic device (100) may be connected to or paired with an external electronic device (110). For example, the electronic device (100) may be connected to the external electronic device (110) via WiFi communication techniques, Bluetooth communication techniques, or UWB (ultra-wideband) communication techniques. For example, the external electronic device (110) may identify an event for providing text (140). For example, the external electronic device (110) may transmit various types of data (e.g., text, images, videos, and / or audio) to the electronic device (100). For example, the external electronic device (110) may transmit data representing text (140) to the electronic device (100). For example, the external electronic device (110) may transmit a signal instructing the electronic device (100) to display text (140). For example, the electronic device (100) can receive the data from an external electronic device (110). For example, the electronic device (100) can receive the signal from an external electronic device (110). For example, the electronic device (100) can use the data to display text (140) through the display of the electronic device (100).

[0026] For example, a user (120) of an electronic device (100) may be in a driving state. For example, a user (120) of an electronic device (100) may be in a state of focusing on the external environment. For example, when the user (120) is in the driving state or when the user (120) is in a state of focusing on the external environment, it may be difficult for the user (120) to perceive notifications or text provided by the electronic device (100). For example, the notification (or text) provided or output by the electronic device (100) may interfere with the user (120)'s driving state or state of focusing on the external environment. For example, the electronic device (100) may provide notifications in a manner that is difficult for the user (120) to perceive. For example, when the user of the electronic device (100) is in a driving state, visual notifications may be difficult for the user (120) to perceive. For example, when the user (120) is in a conversational state, auditory notifications may be difficult to perceive by the user (120). For example, the electronic device (100) may be required to provide a notification in a suitable manner based on the user's (120) current state. For example, the electronic device (100) may be required to provide a notification in a manner that is easy for the user (120) to perceive.

[0027] For example, the electronic device (100) may be required to summarize the text (140) identified by the notification. For example, the electronic device (100) may be required to convert the image (or video) identified by the notification into text, summarize, or express it. For example, the electronic device (100) may be required to display the text converted from the image identified by the notification. For example, the electronic device (100) may be required to reduce the length of the text. For example, the electronic device (100) may be required to output or provide the result of the summary of the text (140). For example, the electronic device (100) may acquire sensor data regarding the surrounding environment of the electronic device (100) through one or more sensors included in the electronic device (100) (e.g., one or more sensors (211) of FIG. 2). For example, the electronic device (100) may acquire information indicating the state of the user (120) using the sensor data. For example, the electronic device (100) can obtain information indicating the state of the user (120) by providing the sensor data to a trained artificial intelligence model. For example, the electronic device (100) can use the information to convert a text (140) of a first length into a result text of a second length shorter than the first length. For example, the electronic device (100) can use the information to obtain the result text of the second length from the text (140). For example, the electronic device (100) can output the result text through output components (e.g., output components (212) of FIG. 2).

[0028] For example, the electronic device (100) can minimize the user (120)'s distraction by outputting the result text of the second length without outputting the text of the first length (140). For example, the electronic device (100) can help the user (120) focus on the external environment by outputting the result text.

[0029] For example, the electronic device (100) may include hardware components used to perform or execute the above operations. The hardware components are described and illustrated with reference to FIG. 2.

[0030] Figure 2 is a simplified block diagram of an exemplary electronic device.

[0031] Referring to FIG. 2, the electronic device (100) may include at least one processor (207), memory (206), one or more sensors (211) and output components (212).

[0032] According to one embodiment, at least one processor (207) may include a hardware component for processing data using instructions stored in memory (206). The hardware component for processing data may include a central processing unit (e.g., including a processing circuit). The hardware component for processing data may include a graphic processing unit (e.g., including a processing circuit). The hardware component for processing data may include a display processing unit (e.g., including a processing circuit). The hardware component for processing data may include a neural processing unit (e.g., including a processing circuit).

[0033] At least one processor (207) may include one or more cores. For example, at least one processor (207) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0034] Memory (206) may include a hardware component for storing data and / or instructions that are input to and / or output from at least one processor (207). Memory (206) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, and embedded multimedia card (EMMC).

[0035] One or more sensors (211) may be used to acquire data about the surrounding environment of the electronic device (100). For example, one or more sensors (211) may include a camera (203) and a microphone (204). However, they are not limited thereto.

[0036] The camera (203) may include one or more light sensors (e.g., a CCD (charged coupled device) sensor, a CMOS (complementary metal oxide semiconductor) sensor) that generate an electrical signal indicating the color and / or brightness of light. For example, the camera (203) may be described as an image sensor. For example, the camera (203) may be available to acquire an image of the environment surrounding the electronic device (100).

[0037] The microphone (204) may include a hardware component for supporting the reception of audio (e.g., the voice of the user (120)) or a signal of audio. The microphone (204) may be used to acquire audio data or an audio signal by receiving audio, voice, speech, or utterance.

[0038] Output components (212) may be used to output text. For example, output components (212) may include a display (208), a haptic actuator (209), and a speaker (210). However, they are not limited thereto.

[0039] The display (208) can output visualized information. For example, the display (208) can output visualized information to a user under the control of at least one processor (207). The display (208) may include hardware components of an electronic device (100) used to display a screen. For example, the display (208) may include light-emitting elements and circuits (e.g., transistors) that control the light-emitting elements to emit light. For example, each of the light-emitting elements may include an organic light-emitting diode (OLED) or a micro LED. However, it is not limited thereto. For example, the display (208) may include a liquid crystal display (LCD).

[0040] A haptic actuator (209) can be used to provide vibration notifications. For example, an electronic device (100) can control the haptic actuator (209) along with outputting text.

[0041] The speaker (210) can be used to output an audio signal. For example, the speaker (210) can be used to output an audio signal obtained as TTS is performed from text.

[0042] At least one processor (207) can identify an event for outputting a notification for a first text of a first length (e.g., the first text (420) of FIG. 4a). For example, at least one processor (207) can determine the length of the text to be used to output the notification by using sensor data obtained through one or more sensors (211). For example, one or more sensors (211) can be used to obtain sensor data about the environment around the electronic device (100). For example, at least one processor (207) can obtain a second text of a second length (e.g., the second text (430) of FIG. 4a) from the first text (e.g., the first text (420) of FIG. 4a) based on determining the length of the text to be used to output the notification as a second length less than the first length. For example, at least one processor (207) can output the second text. For example, at least one of the output components (212) can be used to output the second text.

[0043] According to one embodiment, the first text or the second text may include binary data, electrical signals, or digital data for the text to be processed by output components (212).

[0044] FIG. 3 is a flowchart illustrating the operation of an electronic device that outputs text. This method may be executed by the electronic device (100) illustrated in FIG. 2 or by at least one processor (207) of the electronic device (100).

[0045] Referring to FIG. 3, in operation 310, the electronic device (100) can identify an event for outputting a notification for a first text of a first length (e.g., the first text (420) of FIG. 4a). For example, the event may be triggered by a plurality of software applications (e.g., a software application for messages, a software application for mail, and a software application for schedules). For example, when the electronic device (100) identifies the event, it may obtain the first text of the first length. For example, the electronic device (100) may receive data representing the first text from an external electronic device through a communication circuit (not shown). For example, the electronic device (100) may obtain text data representing the first text through the plurality of software applications. For example, an operation for identifying an event for outputting a notification for the first text is described, but the embodiment is not limited thereto. For example, the electronic device (100) can identify an event for outputting a notification for a first content (e.g., the first content (440) of FIG. 4a) (e.g., text, image, and audio).

[0046] In operation 320, the electronic device (100) can determine the length of text to be used to output a notification by using sensor data obtained through one or more sensors (211). For example, the operation of determining the length of the text may include the operation of determining the immersive level of the user's surrounding environment. For example, the size (or length) of the content may increase or decrease depending on the immersive level of the surrounding environment. According to one embodiment, the content to be output by the output components (212) (e.g., the second content (450) of FIG. 4a) may change according to the immersive level. For example, the electronic device (100) may obtain sensor data regarding the surrounding environment of the electronic device (100) based on identifying the event. For example, the electronic device (100) may determine the length of text to be used to output a notification based on obtaining the sensor data. For example, the electronic device (100) can obtain information indicating the state of the user (120) using the sensor data. For example, the electronic device (100) can determine the length of text to be used to output a notification based on identifying the state of the user (120). For example, the electronic device (100) can determine a second length shorter than a first length as the length of the text to be output based on identifying that the user (120) is in a state where he / she cannot focus on the notification. For example, the electronic device (100) can determine a third length longer than the second length and shorter than the first length as the length of the text to be output based on identifying that the user (120) is in a state where he / she can focus on the notification. For example, an operation to determine the length of text to be used to output a notification is described, but the embodiment is not limited thereto.For example, the electronic device (100) can use the sensor data to determine the length (or amount, number) of the content (e.g., text, image, and audio) to be used to output a notification.

[0047] In operation 330, a second text of a second length (e.g., the second text of FIG. 4a (430)) can be obtained from a first text (e.g., the first text of FIG. 4a (420)). For example, the electronic device (100) can obtain or generate the second text of the second length using the first text based on the above determination. For example, the electronic device (100) can generate or obtain the second text by summarizing the first text. For example, the electronic device (100) can generate the second text using some of the characters included in the first text. For example, the electronic device (100) can generate the second text using some of the words included in the first text. For example, an operation of obtaining the second text from the first text is described, but the embodiments are not limited thereto. For example, the electronic device (100) can obtain a second content of the second length (e.g., the second content (450) of FIG. 4a) from the first content by determining the length (or amount, number) of the content (e.g., text, image, and audio) to be used to output a notification as a second length (or amount, number) less than the first length (or amount, number).

[0048] According to one embodiment, the operation of determining the length of the text may include the operation of determining the level of concentration on the user (120)'s surrounding environment. For example, the size (or length) of the second content (e.g., the second content (450) of FIG. 4a) may increase or decrease depending on the level of concentration on the user (120)'s surrounding environment. For example, the size of the second content to be output by the output components (212) may be changed according to the level of concentration.

[0049] In operation 340, the electronic device (100) can output a second text (e.g., the second text (430) of FIG. 4a). For example, the electronic device (100) can output the second text through at least one of the output components (212). For example, the electronic device (100) can determine the output component to which the second text is output by using sensor data obtained through one or more sensors (211). For example, the electronic device (100) can determine the output component to which the second text is output by using information indicating the state of the user (120) obtained through the sensor data. For example, to output the second text, the electronic device (100) can control at least one output component among the output components that is indicated by a response obtained by providing the first text to a second model (e.g., the second model (425) of FIG. 4a). For example, the operation of outputting the second text is described, but the embodiment is not limited thereto. For example, the electronic device (100) can output the acquired second content (e.g., the second content (450) of FIG. 4a).

[0050] For example, the operation of providing or outputting the second text may include the operation of providing content (e.g., the second content (450) of FIG. 4a) generated using at least one of content provided by a software application (e.g., the first content (440) of FIG. 4a), a category for the software application (e.g., the type of the software application), and information about the user's environment (e.g., concentration level and sensor data (410)).

[0051] The electronic device (100) can maintain the user's (120) immersion by outputting a second text (430) of a second length. For example, the electronic device (100) can display a second text (430) of a second length shorter than a first length, in a size smaller than the content, through the display (208) to avoid disturbing the user (120) who is concentrating on the content being displayed through the display (208). For example, the electronic device (100) can display the second text (430) of the second length through the display (208) to avoid disturbing the user (120) who is talking to another person. For example, the electronic device (100) can output an audio signal corresponding to the second text (430) through the speaker (210) to avoid disturbing the user (120) who is in a driving state. For example, the electronic device (100) may avoid interfering with the user's (120) activities by outputting a second text (430) of a second length shorter than the first length. For example, the user (120) may maintain focus on the activities being performed by receiving the second text (430) of the second length.

[0052] For example, the electronic device (100) can obtain the information indicating the state of the user (120) using a trained model. For example, the electronic device (100) can obtain the second text using another trained model. The acquisition of the information using the trained model and the acquisition of the second text using the other trained model are described and illustrated in more detail with reference to FIG. 4a.

[0053] FIGS. 4a and 4b illustrate an exemplary operation of an electronic device that obtains a second text from a first text using a trained model.

[0054] Referring to FIG. 4a, the electronic device (100) may include a first model (415) and a second model (425). The electronic device (100) may acquire sensor data (410) regarding the environment surrounding the electronic device (100) through one or more sensors (211). For example, one or more sensors (211) may include a camera (203) or a microphone (204). However, it is not limited thereto. For example, one or more sensors (211) may include a global positioning system (GPS), a light sensor, an accelerometer, an infrared sensor, and a flicker sensor. For example, the electronic device (100) may identify events to provide notifications. For example, the electronic device (100) may acquire sensor data (410) through one or more sensors (211). For example, the electronic device (100) may acquire sensor data (410) through one or more sensors (211) based on identifying the event. However, it is not limited thereto. For example, the electronic device (100) may acquire other sensor data through one or more sensors (211) before identifying the event. For example, the electronic device (100) may acquire information indicating the state of the user (120) by using the other sensor data and the sensor data (410) acquired after identifying the event. For example, the electronic device (100) may acquire information indicating the state of the user (120) by comparing the other sensor data and the sensor data (410).

[0055] For example, the electronic device (100) can obtain information indicating the state of the user (120) by providing sensor data (410) to the first model (415). For example, the information may indicate the state of the user (120) (e.g., resting state, driving state, conversational state, or state of concentrating on media content). For example, the first model (415) may be a model trained based on machine learning. For example, the first model (415) may be a model trained based on deep learning. For example, the first model (415) may be a large multimodal model (LMM), a large language model (LLM), or a large visual model (LVM). For example, the first model (415) may be described as a model trained to output information indicating the state of the user (120) in the form of text by receiving at least one of an image, an audio signal, and text. The first model (415) may be referred to as the first artificial intelligence model, the first generative artificial intelligence model, or the first generative model. The second model (425) may be referred to as the second artificial intelligence model, the second generative artificial intelligence model, or the second generative model.

[0056] According to one embodiment, the first model (415) can output information indicating the location where the user (120) is located by using sensor data (410) representing an image. For example, the first model (415) can output information indicating the location where the user (120) is located by performing object detection from the image. For example, the first model (415) can output information indicating the location where the user (120) is located by performing location recognition from the image.

[0057] According to one embodiment, the first model (415) can output information indicating the location where the user (120) is located and / or whether the user (120) is conversing, by using sensor data (410) representing an audio signal. For example, the first model (415) can output information indicating the location where the audio signal was generated by identifying the audio signal. For example, the first model (415) can output information indicating the number of people who caused the audio signal to be generated by identifying the audio signal.

[0058] According to one embodiment, the first model (415) can output information indicating the location where the user (120) is located by using sensor data (410) indicating the location obtained via GPS. For example, the first model (415) can output other information determining the manner of notification to be provided to the user (120) by using the sensor data (410) indicating the location. However, it is not limited thereto. For example, the information may include the other information.

[0059] According to one embodiment, an electronic device (100) can obtain information indicating the environment of a user (120) by using sensor data (410) obtained through at least one of a flicker sensor, a light sensor, an accelerometer, and an infrared sensor. For example, the electronic device (100) can obtain information about the environment in which the electronic device (100) is located by using sensor data (410) obtained through a flicker sensor. For example, the electronic device (100) can obtain information about the illuminance of the location where the electronic device (100) is located by using sensor data (410) obtained through a light sensor. For example, the electronic device (100) can identify the location where the electronic device (100) is located as a quiet indoor space (e.g., a library) based on identifying illuminance data below a threshold illuminance. For example, the electronic device (100) can output a second text (430) in a manner suitable for the location based on the identification. For example, the electronic device (100) can control the volume of the audio signal corresponding to the second text (430) based on the identification. For example, the electronic device (100) can lower the volume of the audio signal corresponding to the second text (430) based on the identification. For example, the electronic device (100) can output an audio signal of a second volume lower than the first volume of the audio signal output from another space based on identifying the location where the electronic device (100) is located as a quiet indoor space. For example, the electronic device (100) can output an audio signal of a second length less than the first length of the audio signal output from another space based on identifying the location where the electronic device (100) is located as a quiet indoor space. For example, the electronic device (100) can output a second text (430) of a second length shorter than the first length of the first text (420) based on identifying the location where the electronic device (100) is located as a quiet indoor space.

[0060] For example, the electronic device (100) can identify whether the user (120) is in a moving state (e.g., driving state) by using sensor data (410) obtained through an acceleration sensor. For example, the electronic device (100) can identify the movement of an external object around the electronic device (100) by using sensor data (410) obtained through an infrared sensor.

[0061] For example, the electronic device (100) can obtain information indicating the state of the user (120) by providing sensor data (410) to the first model (415). For example, the information may indicate the state of the user (120) (e.g., driving state, conversation state, or state of focusing on media content) and the level at which the user (120) can focus on notifications. For example, the information may be a text-based prompt. For example, the information may be a prompt containing a natural language sentence. For example, the information may be provided to the second model (425). For example, the electronic device (100) may use the first model (415) to obtain a prompt containing a natural language sentence indicating the length of text to be used to output a notification from the sensor data (410). For example, the electronic device (100) may obtain a prompt containing a natural language sentence indicating the length of text to be used to output a notification by providing sensor data (410) to the first model (415). For example, the prompt may include a natural language sentence indicating the length of text to be used to output a notification. For example, the prompt may include at least one of a character, word, natural language, or number to indicate the length of the text or to summarize the text. For example, the prompt may include another natural language sentence indicating the state of the user (120). For example, the electronic device (100) may obtain a second text (430) by providing the first text (420) and the prompt to the second model (425).

[0062] According to one embodiment, the electronic device (100) may obtain a second text (430) by providing a first text (420) to a second model (425). However, it is not limited thereto. For example, the electronic device (100) may obtain second data of a second type (e.g., text, image, video, and audio) by providing first data of a first type (e.g., text, image, video, and audio) to the second model (425). For example, the first type may be the same as the second type. However, it is not limited thereto. For example, the first type may be different from the second type. For example, the second data of the second type may be data abbreviated from the first data. For example, the electronic device (100) may obtain text representing the image by providing an image to the second model (425). For example, the electronic device (100) may display the text through a display (208). For example, the electronic device (100) can output an audio signal representing the text through a speaker (210). For example, the electronic device (100) can obtain text representing the audio signal by providing the audio signal to a second model (425).

[0063] The second content (450) obtained by the second model (425) may consist of data of multiple types. For example, the electronic device (100) may obtain the second content (450) of at least one type of image, video, text, or audio by providing the second model (425) with the first content (440) of at least one type of image, video, text, or audio. For example, the electronic device (100) may obtain the second content (450) of text and image by providing the second model (425) with the first content (440) of text and image.

[0064] The second model (425) may be an artificial intelligence model trained through machine learning. For example, the second model (425) may be a model trained through deep learning. For example, the second model (425) may be a model trained for natural language processing. For example, the second model (425) may be an LMM, LLM, or LVM. For example, the second model (425) may be described as a model trained to output result content (e.g., second text (430)) using a prompt and source content (e.g., first text (420)) indicating the state of the user (120). For example, the second model (425) may be a different model from the first model (415). However, it is not limited thereto. For example, the first model (415) and the second model (425) may be integrated.

[0065] The electronic device (100) may identify a first text (420) of a first length based on identifying an event for providing a notification. For example, the event may include the provision of a notification by a software application (e.g., a software application for messages, a software application for mail, or a software application for schedules). For example, the first text (420) may be provided by the software application, but is not limited thereto. For example, the electronic device (100) may obtain an image or audio signal from the software application based on identifying the event.

[0066] The electronic device (100) can obtain or generate a second text (430) of a second length shorter than the first length by providing information indicating the state of the user (120), output from the first model (415), and a first text (420) of a first length to the second model (425). For example, the electronic device (100) can obtain a second text (430) containing some of the characters included in the first text (420) by using information indicating the state of the user (120). For example, the electronic device (100) can determine the number of characters to be included in the second text (430) among the characters included in the first text (420) by using information indicating the state of the user (120). For example, the electronic device (100) can obtain a second text (430) containing some of the words included in the first text (420) by using information indicating the state of the user (120). For example, the electronic device (100) can determine the number of words to be included in the second text (430) among the words included in the first text (420) by using information indicating the state of the user (120). For example, the electronic device (100) can obtain a first text (420) of a first length and a second text (430) of a second length shorter than the first length based on a determination that the user (120)’s concentration level obtained from the first model (415) is a first level. For example, the electronic device (100) can obtain a second text (430) of a third length shorter than the second length based on a determination that the user (120)’s concentration level is a second level higher than the first level. For example, if the first text (420) is "I have an appointment with Kim Su-min at a Japanese restaurant near Gangnam Station at 4 PM" and the user (120)'s concentration level is the first level, the second text (430) of the second length may appear as "4 PM, Gangnam Station, Japanese restaurant, appointment".For example, if the user (120)'s concentration level is at the second level, the second text (430) of the third length may appear as "4 PM, appointment". Although it is described that the concentration level and the length of the second text (430) are inversely proportional, this is merely illustrative.

[0067] According to one embodiment, the electronic device (100) can obtain a second text (430) longer than the first text (420) using a second model (425). For example, the electronic device (100) can supplement incomplete parts (e.g., ambiguous expressions, missing expressions) of the first text (420) by providing the first text (420) to the second model (425). For example, the electronic device (100) can obtain a second text (430) in which the incomplete parts of the first text (420) are supplemented. For example, the length of the second text (430) in which the incomplete parts are supplemented may be longer than the first text (420). For example, the electronic device (100) can obtain the supplemented second text (430) from the first text (420) using sensor data (410) that indicates information about the user's state (e.g., the utterance of the speaker). For example, the electronic device (100) can obtain an audio signal corresponding to a second text (430) supplemented from a first text (420).

[0068] According to one embodiment, the electronic device (100) can obtain a second content (450) summarized from the first content (440) by providing the first content (440) (e.g., text, image, and audio) to the second model (425). For example, the electronic device (100) can obtain a second content (450) (e.g., text, image, and audio) representing the first content (440) representing the image by providing the first content (440) representing the image to the second model. For example, the electronic device (100) can obtain a second content (450) (e.g., text, image, and audio) representing the first content (440) representing the audio. For example, the length (or amount, number, size) of the second content (450) may vary depending on the level of concentration of the user (120). For example, when the concentration level of the user (120) is at a first level, the electronic device (100) can acquire a second content (450) of a second size smaller than the first size of the first content (440). For example, when the concentration level of the user (120) is at a second level, the electronic device (100) can acquire a second content (450) of a third size smaller than the first size and the second size.

[0069] According to one embodiment, the electronic device (100) can determine the length of text to be used to output a notification by using information about the type of software application providing the notification (e.g., message, schedule, banking, navigation, or call). For example, the electronic device (100) can summarize a first text (420) based on the type. For example, the electronic device (100) can determine keywords from the first text (420) according to the type of software application. For example, the electronic device (100) can obtain a second text (430) containing keywords of the first text (420) according to the type of software application linked to the first text (420) by providing the first text (420) to a second model (425). For example, the electronic device (100) can obtain a second text (430) of a second length from a first text (420) of a first length depending on the type of software application. For example, the electronic device (100) can obtain a second text (430) that is adaptive to the type of software application by providing a first text (420) to a first model (415).

[0070] For example, the electronic device (100) can determine the length of text to be used to output a notification by using sensor data (410) and information (e.g., type) about a software application that causes a notification. For example, the electronic device (100) can obtain a second text (430) by providing a second model (425) with a prompt indicating the state of the user (120) obtained using the sensor data (410) and the information. For example, the electronic device (100) can determine the length of the text obtained by the second model (425) differently based on the type of the software application. For example, the electronic device (100) can obtain a second text (430) of a second length less than the first length through the second model (425) based on the determination that the notification is caused by a first software application (e.g., a software application for a bank). For example, the electronic device (100) can obtain a third text of a third length less than the second length through a second model (425) based on the determination that the notification is caused by a second software application (e.g., a software application for messages).

[0071] The electronic device (100) can determine an output component for outputting a second text (430) by using information indicating the state of the user (120). For example, the electronic device (100) can obtain a second text (430) generated from a first text (420) by providing the information to a second model (425). For example, the second model (425) can modify the first text (420) into a form suitable for providing a notification to the user (120) based on the information. For example, the electronic device (100) can obtain a second text (430) by providing the second model (425) with information indicating the state of the user (120) output from the first text (420) and the first model (415).

[0072] According to one embodiment, the electronic device (100) can output a second text (430) through output components (212). For example, the electronic device (100) can output the second text (430) through at least one of the output components (212). For example, the output components (212) may include a display (208), a speaker (210), and a haptic actuator (209). For example, the electronic device (100) can determine the output component to output the second text (430) by using information indicating the state of the user (120) obtained from the first model (415). For example, the electronic device (100) can determine the output component to output the second text (430) by using the second model (425). For example, the electronic device (100) can output a second text (430) through an output component that appears by result content output from the second model (425).

[0073] For example, the electronic device (100) may control at least one output component indicated by a response obtained by providing the first text (420) to the second model (425) among the output components (212) in order to output the second text (430). For example, the response may include the second text (430). For example, the response may include data indicating an output component for outputting the second text (430). For example, the response may be referenced as result content. For example, the electronic device (100) may obtain the second text (430) and data indicating at least one output component for outputting the second text (430) by providing the first text (420) and information (or prompt) about the environment of the user (120) to the second model (425).

[0074] For example, the result content may include a second text (430). For example, the result content may include an audio signal representing the second text (430). For example, the result content may include an image associated with the second text (430). For example, the second model (425) may be trained to generate the result content for outputting the result content from the first text (420) through an output component based on the environment, using information representing the environment of the user (120) and the first text (420).

[0075] For example, the electronic device (100) may display the second text (430) through the display (208) among the output components (212). For example, the electronic device (100) may output an audio signal obtained by performing TTS on the second text (430) through the speaker (210) among the output components (212). For example, the electronic device (100) may control a haptic actuator (209) to provide a vibration notification along with displaying the second text (430) or outputting the audio signal. However, it is not limited thereto. For example, the electronic device (100) may output the audio signal through the speaker (210) at the same time as displaying the second text (430) through the display (208).

[0076] According to one embodiment, the electronic device (100) can perform adaptive noise cancellation (ANC) when outputting the audio signal. For example, the ANC can be described as a technique that provides an optimal noise cancellation function by monitoring the ambient noise of the electronic device (100) in real time. For example, the electronic device (100) can perform ANC based on identifying that audio data acquired through the microphone (204) is included in the sensor data (410). For example, the electronic device (100) can perform ANC when outputting an audio signal corresponding to the second text (430) through the speaker (210) based on receiving audio data about the environment around the electronic device (100) through the microphone (204).

[0077] Referring to FIG. 4b, in operation 460, the electronic device (100) can determine the environment of the user (120). For example, the electronic device (100) can obtain information about the environment of the user (120) by providing sensor data (410) to the first model (415).

[0078] In operation 465, the electronic device (100) can identify an event of a notification. For example, the electronic device (100) can identify an event for outputting a notification. For example, operation 465 of FIG. 4b can correspond to operation 310 of FIG. 3.

[0079] In operation 470, the electronic device (100) can determine the manner of notification. For example, the electronic device (100) can determine the manner of notification (e.g., at least one of image, text, video, or audio) using sensor data (410).

[0080] In operation 475, the electronic device (100) can generate a mixed notification corresponding to the environment of the user (120). For example, the electronic device (100) can generate a mixed notification using information about the environment of the user (120) obtained using sensor data (410). For example, the electronic device (100) can obtain a second content (450) from a first content (440) using the information.

[0081] In operation 480, the electronic device (100) can output the mixed notification. For example, the electronic device (100) can provide the mixed notification by using at least some of the output components (212).

[0082] For example, the electronic device (100) can obtain information indicating the state of the user (120) by providing sensor data (410) to the first model (415). For example, the information may be in the form of a prompt that can be provided to the second model (425). For example, the information may be a prompt indicating the state of the user (120). The information or the prompt is described and illustrated in more detail with reference to FIG. 5.

[0083] FIG. 5 is a flowchart illustrating the operation of an electronic device that determines the length of text using a prompt indicating the user's state. This method may be executed by the electronic device (100) illustrated in FIG. 2 or by at least one processor (207) of the electronic device (100).

[0084] Referring to FIG. 5, in operation 510, the electronic device (100) can acquire sensor data (410) through one or more sensors (211) based on identifying an event to provide a notification.

[0085] In operation 520, the electronic device (100) can obtain a prompt indicating the state of the user (120) of the electronic device (100) using sensor data (410). For example, the electronic device (100) can obtain a prompt indicating the state of the user (120) by providing sensor data (410) to the first model (415). For example, the prompt may indicate the state of the user (120) (e.g., resting state, driving state, concentration state, or conversational state). For example, the prompt may be based on natural language or text. For example, the prompt may indicate the level of concentration of the user (120). For example, the prompt may be described as a prompt provided to the second model (425). For example, the prompt may include a prompt instructing the first text (420) to be summarized into a single sentence. For example, the prompt may include a prompt instructing to summarize the first text (420) into two sentences. For example, the prompt may include a prompt instructing to summarize the first text (420) so that the user (120) can recognize it even when conversing with another person.

[0086] In operation 530, the electronic device (100) can determine the length of text to be used to output a notification based on a prompt indicating the state of the user (120). For example, the electronic device (100) can determine the length of the text by providing the prompt to the second model (425). For example, the electronic device (100) can obtain the text of the determined length by providing the prompt to the second model (425).

[0087] The electronic device (100) can determine the length of the text using a prompt indicating the state of the user (120). However, it is not limited thereto. For example, the electronic device (100) can obtain information about the environment of the user (120) by providing sensor data (410) to the first model (415). For example, the electronic device (100) can determine whether to summarize the first text (420) (or the first content (440)) using the information about the environment of the user (120) obtained by the first model (415).

[0088] For example, the electronic device (100) may determine, based on the prompt, that the length of the text to be used to output a notification is a second length shorter than the first length. For example, the electronic device (100) may obtain a second text (430) from a first text (420) by providing a second model (425) with a prompt obtained from a first model (415). For example, the electronic device (100) may obtain a second text (430) by providing the first text (420) and the prompt to the second model (425). For example, the electronic device (100) may output a second text (430) summarizing the first text (420) by providing the second model (425) with a prompt indicating that the user (120) is in a conversational state. For example, the electronic device (100) can display the second text (430) through the display (208) by providing the second model (425) with a prompt indicating that the user (120) is in a conversational state.

[0089] According to one embodiment, the electronic device (100) can acquire sensor data (410) representing images of the surrounding environment of the electronic device (100) through a camera (203) included in one or more sensors (211). For example, the electronic device (100) can use the images to identify external objects included in the surrounding environment. For example, the electronic device (100) can use the images to identify the type of the external object (e.g., media content, person, or animal). For example, the electronic device (100) can acquire a response output by a second model (425) based on the type of the external object. For example, the electronic device (100) can acquire a text, image, or audio signal output by the second model (425) according to the type of the external object. For example, the electronic device (100) may display a second text (430) through a display (208) according to the type of the external object. For example, the electronic device (100) may output an audio signal corresponding to the second text (430) through a speaker (210) according to the type of the external object.

[0090] According to one embodiment, the electronic device (100) can acquire sensor data (410) containing image data of the surrounding environment of the electronic device (100) through a camera (203) included in one or more sensors (211). For example, the electronic device (100) can provide various information to a user (120) using the sensor data (410). For example, the electronic device (100) can identify a specific location (e.g., a bus stop) based on the image. For example, the electronic device (100) can provide information linked to the specific location (e.g., bus dispatch intervals) based on identifying that the electronic device (100) is located at the specific location. For example, the electronic device (100) can output the information through a speaker (210) or display it through a display (208). For example, the electronic device (100) can generate logging video content using the sensor data (410). For example, the logging video content can be described as video content containing images of the user's (120) daily life.

[0091] For example, when the electronic device (100) identifies an event to provide a notification, it may activate at least one of one or more sensors (211). For example, the electronic device (100) may activate the remaining sensors based on sensor data obtained through the at least one. For example, the activation of the at least one and the activation of the remaining sensors are described and illustrated in more detail with reference to FIG. 6.

[0092] FIG. 6 is a flowchart illustrating the operation of an electronic device that activates a camera based on an audio signal. This method may be executed by the electronic device (100) illustrated in FIG. 2 or by at least one processor (207) of the electronic device (100).

[0093] Referring to FIG. 6, in operation 610, when the electronic device (100) identifies an event to provide a notification, it can identify the volume of an audio signal obtained through a microphone (204) included in one or more sensors (211). For example, the electronic device (100) can identify the volume of the audio signal using the root-mean-square (RMS) of the audio signal obtained through the microphone (204). For example, the electronic device (100) can determine the level at which ANC is to be performed using the RMS. For example, the electronic device (100) can obtain an audio signal of the surrounding environment of the electronic device (100) by activating the microphone (204) based on identifying the event. For example, the electronic device (100) can determine whether to activate a camera (203) included in one or more sensors (211) by analyzing the audio signal.

[0094] In operation 620, the electronic device (100) may switch the state of the camera (203) from an inactive state to an active state during a time interval based on identifying the volume of an audio signal that exceeds a threshold volume. For example, the electronic device (100) may activate the camera (203) during a time interval based on the determination that the volume of the audio signal exceeds a threshold volume. For example, the electronic device (100) may activate the camera (203) during a time interval based on identifying the volume of an audio signal that exceeds the threshold volume.

[0095] In operation 630, the electronic device (100) can acquire at least one image through a camera (203) that is active during a time interval. For example, the at least one image may be an image of the surrounding environment of the electronic device (100). For example, the electronic device (100) can acquire images of the surrounding environment of the electronic device (100) through a camera (203) that is active during a time interval. For example, the electronic device (100) can acquire sensor data (410) representing the images through the camera (203).

[0096] In operation 640, the electronic device (100) can determine the length of text to be used to output a notification by using the audio signal and the at least one image. For example, the electronic device (100) can obtain a prompt indicating the state of the user (120) by providing the audio signal and the at least one image represented by the sensor data (410) to the first model (415). For example, the electronic device (100) can determine the length of text to be used to output a notification based on the prompt. For example, the electronic device (100) can obtain a second text (430) of a second length less than the first length by providing the prompt and the first text (420) to the second model (425).

[0097] For example, the electronic device (100) can determine the length of text to be used to output a notification by using audio signals and the images. For example, the electronic device (100) can provide sensor data (410) representing the images acquired through the camera (203) during the time interval to the first model (415). For example, the electronic device (100) can obtain information representing the state of the user (120) by providing the sensor data (410) representing the images to the first model (415). For example, the electronic device (100) can determine the length of text to be used to output a notification based on the information. The operation 640 may correspond to operation 320 of FIG. 3.

[0098] According to one embodiment, the electronic device (100) can activate the camera (203) during a time interval using an acceleration sensor or a light sensor. For example, the electronic device (100) can identify the acceleration of the electronic device (100) through the acceleration sensor. For example, the electronic device (100) can switch the state of the camera (203) from an inactive state to an active state during a time interval based on identifying acceleration exceeding a threshold acceleration. For example, the electronic device (100) can acquire images of the surrounding environment through the activated camera (203). For example, the electronic device (100) can acquire information indicating the state of the user (120) by providing the images to the first model (415).

[0099] For example, the electronic device (100) can identify the illuminance of the surrounding environment of the electronic device (100) through a light sensor. For example, the electronic device (100) can switch the state of the camera (203) from an inactive state to an active state during a time interval based on identifying illuminance exceeding a threshold illuminance. For example, the electronic device (100) can acquire images of the surrounding environment through the activated camera (203). For example, the electronic device (100) can acquire information indicating the state of the user (120) by providing the images to the first model (415).

[0100] According to one embodiment, an operation to activate the camera (203) is described based on the volume of an audio signal obtained through a microphone (204), acceleration obtained through an accelerometer, and illuminance obtained through a light sensor, but the embodiment is not limited thereto. For example, the electronic device (100) may activate the camera (203) based on a change in characteristics regarding the frequency and wavelength of the audio signal. For example, the electronic device (100) may activate the camera (203) based on a change in echo characteristics identified through a speaker (210) and a microphone (204). For example, the electronic device (100) may determine whether to activate the camera (203) using an inertial measurement unit (IMU) sensor. For example, the electronic device (100) may determine whether to activate the camera (203) using at least one of the color temperature, brightness, blinking information, wavelength, frequency, and incident angle of external light obtained through a light sensor.

[0101] For example, the electronic device (100) can reduce the amount of power consumed by the camera (203) by activating the camera (203) based on some of one or more sensors (211). For example, the amount of power consumed by the camera (203) that is always in an active state is greater than the amount of power consumed by the camera (203) that switches to an active state according to specified conditions. For example, the electronic device (100) can reduce the amount of power consumed by the camera (203) by activating the camera (203) based on an audio signal acquired by the microphone (204).

[0102] According to one embodiment, an electronic device (100) may acquire sensor data (410) based on receiving content to be delivered to a user (120). For example, the electronic device (100) may acquire sensor data (410) based on identifying an event for outputting a notification. The event may include an event in which a notification is provided from a software application. For example, the event may include an event in which the content is received. For example, the electronic device (100) may acquire sensor data (410) through one or more sensors (211) based on the identification of the event.

[0103] According to one embodiment, an operation in which a second text (430) obtained by an electronic device (100) is output by the electronic device (100) is described, but the embodiment is not necessarily limited thereto. For example, the electronic device (100) may output a second text (430) obtained by an external electronic device (110). The acquisition of the second text (430) by the external electronic device (110) is described and illustrated in more detail with reference to FIG. 7.

[0104] FIG. 7 illustrates examples of operations performed within an electronic device and an external electronic device.

[0105] Referring to FIG. 7, in operation 710, an external electronic device (110) may transmit a first signal instructing an electronic device (100) to acquire sensor data (410). For example, the external electronic device (110) may transmit the first signal instructing the electronic device (100) to acquire sensor data (410) to provide a notification based on identifying an event to provide a notification. For example, the electronic device (100) may receive the first signal through a communication circuit (not shown).

[0106] According to one embodiment, the external electronic device (110) may be located around the electronic device (100). However, it is not limited thereto. For example, the external electronic device (110) may be a server.

[0107] In operation 720, the electronic device (100) can acquire sensor data (410) through one or more sensors (211). For example, the electronic device (100) can acquire sensor data (410) about the environment around the electronic device (100) through one or more sensors (211) included in the electronic device (100) based on receiving the first signal.

[0108] In operation 730, the electronic device (100) can transmit sensor data (410) to an external electronic device (110) through the communication circuit. For example, the external electronic device (110) can receive sensor data (410) from the electronic device (100).

[0109] According to one embodiment, the electronic device (100) may transmit at least one of information related to a user (120) stored in the memory (206) of the electronic device (100), information about media content stored in the memory (206), or information about the schedule of the user (120) stored in the memory (206), together with sensor data (410), to an external electronic device (110) through the communication circuit. For example, the external electronic device (110) may receive at least one of information related to the user (120), information about media content, and information about the schedule of the user (120) from the electronic device (100).

[0110] In operation 740, the external electronic device (110) can obtain a prompt indicating the state of the user (120). For example, the external electronic device (110) can obtain a prompt indicating the state of the user (120) by providing sensor data (410) to the first model (415) included in the external electronic device (110). For example, the external electronic device (110) can obtain a prompt indicating the state of the user (120) by further using at least one of information related to the user (120), information about media content, and information about the schedule of the user (120).

[0111] In operation 750, the external electronic device (110) can obtain the second text (430) by providing the prompt to the second model (425) included in the external electronic device (110). For example, the external electronic device (110) can obtain the second text (430) by using the prompt, the first text (420), and information about the software application that causes the notification.

[0112] In operation 760, the external electronic device (110) may transmit a second signal to the electronic device (100) instructing it to output the second text (430) through at least some of the output components (212). For example, the electronic device (100) may receive the second signal from the external electronic device (110) through the communication circuit.

[0113] In operation 770, the electronic device (100) can output a second text (430). For example, the electronic device (100) can output the second text (430) through at least some of the output components (212). For example, the electronic device (100) can output the second text (430) through an output component indicated by the second model (425). For example, the electronic device (100) can output the second text (430) through an output component indicated by the result content output by the second model (425). For example, the electronic device (100) can display the second text (430) through a display (208). For example, the electronic device (100) can output an audio signal corresponding to the second text (430) and having TTS performed on the second text (430) through a speaker (210). For example, the electronic device (100) can control a haptic actuator (209) to provide a vibration notification along with the display and / or output.

[0114] FIG. 8 is a block diagram of an electronic device in a network environment according to various embodiments.

[0115] FIG. 8 is a block diagram of an electronic device (801) (e.g., the electronic device (100) of FIG. 1) in a network environment (800) according to various embodiments. Referring to FIG. 8, in the network environment (800), the electronic device (801) may communicate with an electronic device (802) through a first network (898) (e.g., a short-range wireless communication network) or with at least one of an electronic device (804) or a server (808) through a second network (899) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (801) may communicate with the electronic device (804) through a server (808). According to one embodiment, the electronic device (801) may include a processor (820), memory (830), input module (850), sound output module (855), display module (860), audio module (870), sensor module (876), interface (877), connection terminal (878), haptic module (879), camera module (880), power management module (888), battery (889), communication module (890), subscriber identification module (896), or antenna module (897). In some embodiments, at least one of these components (e.g., connection terminal (878)) may be omitted from the electronic device (801), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (876), camera module (880), or antenna module (897)) may be integrated into a single component (e.g., display module (860)).

[0116] The processor (820) can control at least one other component (e.g., a hardware or software component) of the electronic device (801) connected to the processor (820) by executing software (e.g., a program (840)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (820) can store commands or data received from other components (e.g., a sensor module (876) or a communication module (890)) in volatile memory (832), process the commands or data stored in volatile memory (832), and store the resulting data in non-volatile memory (834). According to one embodiment, the processor (820) may include a main processor (821) (e.g., a central processing unit or an application processor) or an auxiliary processor (823) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (801) includes a main processor (821) and an auxiliary processor (823), the auxiliary processor (823) may be configured to use lower power than the main processor (821) or to be specialized for a designated function. The auxiliary processor (823) may be implemented separately from the main processor (821) or as part thereof.

[0117] The auxiliary processor (823) may control at least some of the functions or states associated with at least one component of the electronic device (801) (e.g., display module (860), sensor module (876), or communication module (890)) on behalf of the main processor (821) while the main processor (821) is in an inactive (e.g., sleep) state, or together with the main processor (821) while the main processor (821) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (823) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (880) or communication module (890)). According to one embodiment, the auxiliary processor (823) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (801) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (808)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0118] The memory (830) can store various data used by at least one component of the electronic device (801) (e.g., processor (820) or sensor module (876)). The data may include, for example, software (e.g., program (840)) and input or output data for related commands. The memory (830) may include volatile memory (832) or non-volatile memory (834).

[0119] The program (840) may be stored as software in memory (830) and may include, for example, an operating system (842), middleware (844), or an application (846).

[0120] The input module (850) can receive commands or data to be used for a component of the electronic device (801) (e.g., processor (820)) from outside the electronic device (801) (e.g., user). The input module (850) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0121] The sound output module (855) can output a sound signal to the outside of the electronic device (801). The sound output module (855) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0122] The display module (860) can visually provide information to an external (e.g., user) of the electronic device (801). The display module (860) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (860) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0123] The audio module (870) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (870) can acquire sound through the input module (850) or output sound through the sound output module (855) or an external electronic device (e.g., electronic device (802)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (801).

[0124] The sensor module (876) can detect the operating state of the electronic device (801) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (876) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0125] The interface (877) may support one or more specified protocols that can be used for the electronic device (801) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (802)). According to one embodiment, the interface (877) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0126] The connection terminal (878) may include a connector through which the electronic device (801) can be physically connected to an external electronic device (e.g., electronic device (802)). According to one embodiment, the connection terminal (878) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0127] The haptic module (879) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (879) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0128] The camera module (880) can capture still images and video. According to one embodiment, the camera module (880) may include one or more lenses, image sensors, image signal processors, or flashes.

[0129] The power management module (888) can manage the power supplied to the electronic device (801). According to one embodiment, the power management module (888) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0130] The battery (889) can supply power to at least one component of the electronic device (801). According to one embodiment, the battery (889) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0131] The communication module (890) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (801) and an external electronic device (e.g., electronic device (802), electronic device (804), or server (808)), and the performance of communication through the established communication channel. The communication module (890) may include one or more communication processors that operate independently of the processor (820) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (890) may include a wireless communication module (892) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (894) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (804) through a first network (898) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (899) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (892) can identify or authenticate the electronic device (801) within a communication network such as the first network (898) or the second network (899) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (896).

[0132] The wireless communication module (892) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (892) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (892) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (892) can support various requirements specified in the electronic device (801), external electronic device (e.g., electronic device (804)), or network system (e.g., second network (899)). According to one embodiment, the wireless communication module (892) can support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0133] An antenna module (897) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (897) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (897) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (898) or a second network (899), may be selected from the plurality of antennas, for example, by a communication module (890). A signal or power may be transmitted or received between the communication module (890) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (897).

[0134] According to various embodiments, the antenna module (897) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0135] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0136] According to one embodiment, commands or data may be transmitted or received between the electronic device (801) and an external electronic device (804) through a server (808) connected to a second network (899). Each of the external electronic devices (802, or 804) may be the same or a different type of device as the electronic device (801). According to one embodiment, all or part of the operations performed on the electronic device (801) may be performed on one or more of the external electronic devices (802, 804, or 808). For example, if the electronic device (801) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (801) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (801). The electronic device (801) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (801) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (804) may include an Internet of Things (IoT) device. The server (808) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (804) or the server (808) may be included within the second network (899).The electronic device (801) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0137] FIG. 9a illustrates an example of a perspective view of a wearable device. FIG. 9b illustrates an example of one or more hardware components disposed within the wearable device. According to one embodiment, the wearable device (999) may have the form of glasses that are wearable on a part of a user's body (e.g., head). The wearable device (999) of FIG. 9a and FIG. 9b may be an example of the electronic device (801) and / or electronic device (100) of FIG. 8. The wearable device (999) may include a head-mounted display (HMD). For example, the housing of the wearable device (999) may include a flexible material such as rubber and / or silicone that has a shape that adheres to a part of the user's head (e.g., a part of the face covering both eyes). For example, the housing of the wearable device (999) may include one or more straps that can be twined around the user's head, and / or one or more temples that can be attached to the ears of the head.

[0138] Referring to FIG. 9a, a wearable device (999) according to one embodiment may include at least one display (950) and a frame (900) supporting at least one display (950).

[0139] According to one embodiment, a wearable device (999) may be worn on a part of a user's body. The wearable device (999) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (999). For example, the wearable device (999) may display a virtual reality image provided by at least one optical device (982, 984) of FIG. 9b on at least one display (950) in response to a specified gesture of the user obtained through the motion recognition camera (960-2, 960-3) of FIG. 9b.

[0140] According to one embodiment, at least one display (950) can provide visual information to a user. For example, at least one display (950) may include a transparent or translucent lens. At least one display (950) may include a first display (950-1) and / or a second display (950-2) spaced apart from the first display (950-1). For example, the first display (950-1) and the second display (950-2) may be positioned at locations corresponding to the user's left eye and right eye, respectively.

[0141] Referring to FIG. 9b, at least one display (950) may provide visual information transmitted from external light to a user through a lens included in at least one display (950) and other visual information distinct from said visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (950) may include a first surface (931) and a second surface (932) opposite to the first surface (931). A display area may be formed on the second surface (932) of at least one display (950). When a user wears the wearable device (999), external light may be transmitted to the user by being incident on the first surface (931) and transmitted through the second surface (932). As another example, at least one display (950) can display an augmented reality image combined with a virtual reality image provided by at least one optical device (982, 984) on a real image transmitted through external light in a display area formed on a second surface (932).

[0142] In one embodiment, at least one display (950) may include at least one waveguide (933, 934) that diffracts light emitted from at least one optical device (982, 984) and transmits it to a user. At least one waveguide (933, 934) may be formed based on at least one of glass, plastic, or polymer. A nano pattern may be formed on the exterior or at least a portion of the interior of at least one waveguide (933, 934). The nano pattern may be formed based on a polygonal and / or curved grating structure. Light incident on one end of at least one waveguide (933, 934) may be propagated to the other end of at least one waveguide (933, 934) by the nano pattern. At least one waveguide (933, 934) may include at least one diffractive element (e.g., DOE (diffractive optical element), HOE (holographic optical element)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (933, 934) may be placed within a wearable device (999) to guide a screen displayed by at least one display (950) to the user's eye. For example, the screen may be transmitted to the user's eye based on total internal reflection (TIR) ​​occurring within at least one waveguide (933, 934).

[0143] A wearable device (999) can analyze an object included in a real-world image collected through a camera (960-4), combine a virtual object corresponding to an object among the analyzed objects that is the target of augmented reality provision, and display it on at least one display (950). The virtual object may include at least one of text and an image regarding various information related to the object included in the real-world image. The wearable device (999) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (999) can perform spatial recognition (e.g., SLAM (simultaneous localization and mapping)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (999) can view the image displayed on at least one display (950).

[0144] According to one embodiment, the frame (900) may be formed as a physical structure that allows the wearable device (999) to be worn on the user's body. According to one embodiment, the frame (900) may be configured so that when the user wears the wearable device (999), the first display (950-1) and the second display (950-2) can be positioned corresponding to the user's left and right eyes. The frame (900) may support at least one display (950). For example, the frame (900) may support the first display (950-1) and the second display (950-2) so that they are positioned corresponding to the user's left and right eyes.

[0145] Referring to FIG. 9a, the frame (900) may include a region (920) in which at least a portion of the frame contacts a part of the user's body when the user wears the wearable device (999). For example, the region (920) of the frame (900) in contact with a part of the user's body may include a region in contact with a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the wearable device (999) contacts. According to one embodiment, the frame (900) may include a nose pad (910) that contacts a part of the user's body. When the wearable device (999) is worn by the user, the nose pad (910) may contact a part of the user's nose. The frame (900) may include a first temple (904) and a second temple (905) that contact a different part of the user's body distinct from the part of the user's body.

[0146] For example, the frame (900) may include a first rim (901) covering at least a portion of a first display (950-1), a second rim (902) covering at least a portion of a second display (950-2), a bridge (903) positioned between the first rim (901) and the second rim (902), a first pad (911) positioned along a portion of the edge of the first rim (901) from one end of the bridge (903), a second pad (912) positioned along a portion of the edge of the second rim (902) from the other end of the bridge (903), a first temple (904) extending from the first rim (901) and fixed to a portion of the wearer's ear, and a second temple (905) extending from the second rim (902) and fixed to a portion of the ear opposite to the ear. The first pad (911) and the second pad (912) may come into contact with a part of the user's nose, and the first temple (904) and the second temple (905) may come into contact with a part of the user's face and a part of the ear. The temples (904, 905) may be rotatably connected to the rim through the hinge units (906, 907) of FIG. 9B. The first temple (904) may be rotatably connected to the first rim (901) through a first hinge unit (906) positioned between the first rim (901) and the first temple (904). The second temple (905) may be rotatably connected to the second rim (902) through a second hinge unit (907) positioned between the second rim (902) and the second temple (905). According to one embodiment, a wearable device (999) can identify an external object (e.g., a user's fingertip) touching the frame (900) and / or a gesture performed by said external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of the surface of the frame (900).

[0147] According to one embodiment, the wearable device (999) may include hardware that performs various functions (e.g., hardware to be described later based on the block diagram of FIG. 2). For example, the hardware may include a battery module (970), an antenna module (975), at least one optical device (982, 984), speakers (e.g., speakers (955-1, 955-2)), microphones (e.g., microphones (965-1, 965-2, 965-3)), a light-emitting module (not shown), and / or a PCB (printed circuit board) (990) (e.g., a printed circuit board). The various hardware may be placed within a frame (900).

[0148] According to one embodiment, a microphone (e.g., microphones (965-1, 965-2, 965-3)) of a wearable device (999) is positioned on at least a portion of a frame (900) to acquire a sound signal. A first microphone (965-1) positioned on a bridge (903), a second microphone (965-2) positioned on a second rim (902), and a third microphone (965-3) positioned on a first rim (901) are shown in FIG. 9b, but the number and position of the microphones (965) are not limited to the embodiment of FIG. 9b. If there are two or more microphones (965) included in the wearable device (999), the wearable device (999) can identify the direction of the sound signal by using a plurality of microphones positioned on different portions of the frame (900).

[0149] According to one embodiment, at least one optical device (982, 984) may project a virtual object onto at least one display (950) to provide various image information to a user. For example, at least one optical device (982, 984) may be a projector. At least one optical device (982, 984) may be disposed adjacent to at least one display (950) or included within at least one display (950) as part of at least one display (950). According to one embodiment, a wearable device (999) may include a first optical device (982) corresponding to a first display (950-1) and a second optical device (984) corresponding to a second display (950-2). For example, at least one optical device (982, 984) may include a first optical device (982) positioned at the edge of a first display (950-1) and a second optical device (984) positioned at the edge of a second display (950-2). The first optical device (982) may transmit light to a first waveguide (933) positioned on the first display (950-1), and the second optical device (984) may transmit light to a second waveguide (934) positioned on the second display (950-2).

[0150] In one embodiment, the camera (960) may include a shooting camera (960-4), an eye tracking camera (ET CAM) (960-1), and / or a motion recognition camera (960-2, 960-3). The shooting camera (960-4), the eye tracking camera (960-1), and the motion recognition camera (960-2, 960-3) may be positioned at different locations on the frame (900) and may perform different functions. The eye tracking camera (960-1) may output data indicating the position of the eyes or the gaze of a user wearing the wearable device (999). For example, the wearable device (999) may detect the gaze from an image containing the user's pupils obtained through the eye tracking camera (960-1). A wearable device (999) can identify an object focused by a user (e.g., a real object, and / or a virtual object) by using the user's gaze obtained through an eye-tracking camera (960-1). Upon identifying the focused object, the wearable device (999) can perform a function for interaction between the user and the focused object (e.g., gaze interaction). The wearable device (999) can represent a portion corresponding to the eyes of an avatar representing the user in a virtual space by using the user's gaze obtained through the eye-tracking camera (960-1). The wearable device (999) can render an image (or screen) displayed on at least one display (950) based on the position of the user's eyes. For example, the visual quality of a first region associated with the gaze within the image and the visual quality of a second region distinct from the first region (e.g., resolution, brightness, saturation, grayscale, PPI) may differ from each other. The wearable device (999) can obtain an image having a visual quality of a first region that matches the user's gaze and a visual quality of a second region by using foveated rendering.For example, if the wearable device (999) supports an iris recognition function, user authentication can be performed based on iris information obtained using an eye-tracking camera (960-1). An example in which the eye-tracking camera (960-1) is positioned toward the user's right eye is illustrated in FIG. 9b, but the embodiment is not limited thereto, and the eye-tracking camera (960-1) may be positioned solely toward the user's left eye or toward both eyes.

[0151] In one embodiment, the camera (960-4) can capture a real image or background to be matched with a virtual image in order to implement augmented reality or mixed reality content. The camera (960-4) can be used to acquire high-resolution images based on HR (high resolution) or PV (photo video). The camera (960-4) can capture an image of a specific object located at the position viewed by the user and provide the image to at least one display (950). The at least one display (950) can display a single image in which information regarding a real image or background including the image of the specific object acquired using the camera (960-4) and a virtual image provided through at least one optical device (982, 984) are superimposed. The wearable device (999) can compensate for depth information (e.g., the distance between the wearable device (999) and an external object acquired through a depth sensor) using the image acquired through the camera (960-4). The wearable device (999) can perform object recognition through an image acquired using a shooting camera (960-4). The wearable device (999) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the shooting camera (960-4). The wearable device (999) can perform a pass-through function to superimpose an image acquired through the shooting camera (960-4) onto at least a portion of a screen representing a virtual space while displaying the screen representing a virtual space on at least one display (950). In one embodiment, the shooting camera (960-4) may be placed on a bridge (903) positioned between a first rim (901) and a second rim (902).

[0152] The eye tracking camera (960-1) can achieve more realistic augmented reality by tracking the gaze of a user wearing a wearable device (999), thereby matching the user's gaze with visual information provided to at least one display (950). For example, when the user looks straight ahead, the wearable device (999) can naturally display environmental information related to the user's front on at least one display (950) at the location where the user is situated. The eye tracking camera (960-1) may be configured to capture an image of the user's pupil to determine the user's gaze. For example, the eye tracking camera (960-1) may receive a gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the eye tracking camera (960-1) may be positioned at locations corresponding to the user's left and right eyes. For example, the eye-tracking camera (960-1) may be positioned within the first rim (901) and / or the second rim (902) to face the direction in which the user wearing the wearable device (999) is located.

[0153] The motion recognition camera (960-2, 960-3) can provide a specific event to a screen provided on at least one display (950) by recognizing the movement of the user's entire body or part thereof, such as the user's torso, hands, or face. The motion recognition camera (960-2, 960-3) can recognize the user's gesture, acquire a signal corresponding to the gesture, and provide a display corresponding to the signal to at least one display (950). The processor can identify the signal corresponding to the gesture and, based on the identification, perform a designated function. The motion recognition camera (960-2, 960-3) can be used to perform spatial recognition functions using SLAM and / or depth maps for a 6-degrees-of-freedom pose (6 dof pose). The processor can use the motion recognition camera (960-2, 960-3) to perform gesture recognition functions and / or object tracking functions. In one embodiment, a motion recognition camera (960-2, 960-3) may be placed on the first rim (901) and / or the second rim (902).

[0154] The camera (960) included in the wearable device (999) is not limited to the eye-tracking camera (960-1) and motion recognition camera (960-2, 960-3) described above. For example, the wearable device (999) can identify external objects included within the FoV by using a camera positioned toward the user's FoV. The identification of external objects by the wearable device (999) can be performed based on a sensor for identifying the distance between the wearable device (999) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (960) positioned toward the FoV may support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (999) may include a camera (960) (e.g., a face tracking camera) positioned toward the face to acquire an image including the face of a user wearing the wearable device (999).

[0155] Although not illustrated, according to one embodiment, the wearable device (999) may further include a light source (e.g., LED) that emits light toward a subject (e.g., user's eye, face, and / or an object outside the FoV) being photographed using a camera (960). The light source may include an LED of infrared wavelength. The light source may be placed in at least one of the frame (900) and hinge units (906, 907).

[0156] According to one embodiment, the battery module (970) can supply power to the electronic components of the wearable device (999). In one embodiment, the battery module (970) may be placed within the first temple (904) and / or the second temple (905). For example, the battery module (970) may be a plurality of battery modules (970). The plurality of battery modules (970) may each be placed in the first temple (904) and the second temple (905). In one embodiment, the battery module (970) may be placed at the end of the first temple (904) and / or the second temple (905).

[0157] The antenna module (975) can transmit a signal or power to the outside of the wearable device (999) or receive a signal or power from the outside. In one embodiment, the antenna module (975) may be placed within the first temple (904) and / or the second temple (905). For example, the antenna module (975) may be placed close to one side of the first temple (904) and / or the second temple (905).

[0158] The speaker (955) can output an acoustic signal to the outside of the wearable device (999). The acoustic output module may be referred to as the speaker. In one embodiment, the speaker (955) may be placed within a first temple (904) and / or a second temple (905) to be positioned adjacent to the ear of a user wearing the wearable device (999). For example, the speaker (955) may include a second speaker (955-2) positioned adjacent to the user's left ear by being placed within the first temple (904), and a first speaker (955-1) positioned adjacent to the user's right ear by being placed within the second temple (905).

[0159] A light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (999) to the user. For example, if the wearable device (999) requires charging, it may emit red light at a constant frequency. In one embodiment, the light-emitting module may be placed on the first rim (901) and / or the second rim (902).

[0160] According to one embodiment, a wearable device (999) including a display (950) may be an example of the electronic device (100) of FIG. 2. However, the embodiment is not limited thereto. For example, a head-wearable electronic device including a camera (e.g., camera (960)), a microphone (e.g., microphone (965)), a speaker (e.g., speaker (955)), and a communication circuit without including a display (e.g., display (950)) may be an example of the electronic device (100) of FIG. 2. For example, the head-wearable electronic device may perform operations 310 to 340 of FIG. 3.

[0161] Referring to FIG. 9b, according to one embodiment, a wearable device (999) may include a printed circuit board (PCB) (990). The PCB (990) may be included in at least one of a first temple (904) or a second temple (905). The PCB (990) may include an interposer disposed between at least two sub-PCBs. On the PCB (990), one or more hardware components included in the wearable device (999) (e.g., hardware components illustrated by different blocks in FIG. 2) may be disposed. The wearable device (999) may include a flexible PCB (FPCB) for interconnecting the hardware components.

[0162] According to one embodiment, a wearable device (999) may include at least one of a gyroscope sensor, a gravity sensor, and / or an acceleration sensor for detecting the posture of the wearable device (999) and / or the posture of a body part (e.g., head) of a user wearing the wearable device (999). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis) that are perpendicular to each other. The gyroscope sensor may measure the angular velocity of each of the designated three-dimensional axes (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyroscope sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (999) can identify a user's motion and / or gesture performed to execute or stop a specific function of the wearable device (999) based on an IMU.

[0163] FIGS. 10a and 10b illustrate an example of the appearance of a wearable device (e.g., a wearable device (999)). The wearable device (999) of FIGS. 10a and 10b may be an example of the wearable device (999) of FIG. 8. According to one embodiment, an example of the appearance of a first surface (1010) of the housing of the wearable device (999) may be illustrated in FIG. 10a, and an example of the appearance of a second surface (1020) opposite to the first surface (1010) may be illustrated in FIG. 10b.

[0164] Referring to FIG. 10a, according to one embodiment, a first surface (1010) of a wearable device (999) may have a shape that is attachable to a part of a user's body (e.g., the user's face). Although not illustrated, the wearable device (999) may further include a strap for securing to a part of a user's body and / or one or more temples (e.g., a first temple (904) and / or a second temple (905) of FIG. 9a to 9b). A first display (950-1) for outputting an image to the left eye among the user's two eyes and a second display (950-2) for outputting an image to the right eye among the two eyes may be disposed on the first surface (1010). The wearable device (999) may further include rubber or silicone packing formed on the first surface (1010) to prevent interference by light different from light emitted from the first display (950-1) and the second display (950-2) (e.g., ambient light).

[0165] According to one embodiment, the wearable device (999) may include cameras (960-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (950-1) and the second display (950-2). The cameras (960-1) may be referenced to the eye-tracking camera (960-1) of FIG. 9B. According to one embodiment, the wearable device (999) may include cameras (960-5, 960-6) for photographing and / or recognizing the user's face. The cameras (960-5, 960-6) may be referenced to FT cameras. The wearable device (999) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (960-5, 960-6). For example, the wearable device (999) can change the texture and / or shape of a part of an avatar (e.g., a part of an avatar representing a human face) by using information obtained by cameras (960-5, 960-6) (e.g., FT cameras) and representing the facial expression of a user wearing the wearable device (999).

[0166] Referring to FIG. 10b, on a second surface (1020) opposite to the first surface (1010) of FIG. 10a, a camera (e.g., cameras (960-7, 960-8, 960-9, 960-10, 960-11, 960-12)), and / or a sensor (e.g., a depth sensor (1030)) may be disposed to obtain information related to the external environment of the wearable device (999). For example, cameras (960-7, 960-8, 960-9, 960-10) may be disposed on the second surface (1020) to recognize external objects. The cameras (960-7, 960-8, 960-9, 960-10) can be referenced to the motion recognition cameras (960-2, 960-3) of FIG. 9b.

[0167] For example, using cameras (960-11, 960-12), the wearable device (999) can acquire images and / or videos to be transmitted to each of the user's two eyes. Camera (960-11) may be placed on the second surface (1020) of the wearable device (999) to acquire an image to be displayed through a second display (950-2) corresponding to the right eye among the two eyes. Camera (960-12) may be placed on the second surface (1020) of the wearable device (999) to acquire an image to be displayed through a first display (950-1) corresponding to the left eye among the two eyes. Cameras (960-11, 960-12) may be referenced to the shooting camera (960-4) of FIG. 9b.

[0168] According to one embodiment, the wearable device (999) may include a depth sensor (1030) disposed on a second surface (1020) to identify the distance between the wearable device (999) and an external object. Using the depth sensor (1030), the wearable device (999) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (999). Although not illustrated, a microphone may be disposed on the second surface (1020) of the wearable device (999) to obtain sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0169] Hereinafter, with reference to FIG. 11, the hardware or software configuration of the wearable device (999) will be described.

[0170] FIG. 11 illustrates an example of a block diagram of a wearable device (e.g., a wearable device (999)). The wearable device (999) of FIG. 11 may be an example of the electronic device (801) of FIG. 8 and the wearable device (999) of FIG. 9a through FIG. 10b.

[0171] Referring to FIG. 11, a wearable device (999) according to one embodiment may include a processor (1110), a memory (1115), a display (950) (e.g., a first display (950-1) and / or a second display (950-2) of FIG. 9a, FIG. 9b, FIG. 10a, and FIG. 10b), and / or a sensor (1120). The processor (1110), memory (1115), display (950) and / or sensor (1120) may be electrically and / or operationally connected to each other by an electronic component such as a communication bus (1102). In the present disclosure, the operational connection of the electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the wearable device (999) are not limited to those shown in FIG. 11. For example, the wearable device (999) may include only some of the electronic components shown in FIG. 11.

[0172] A processor (1110) of a wearable device (999) according to one embodiment may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (999) may include one or more processors. The processor (1110) may have a structure of a multi-core processor such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor structure of the processor (1110) may include a structure based on a plurality of core circuits (e.g., a big-little structure), distinguished by power consumption, clock, and / or computational power per unit time. In one embodiment comprising a processor (1110) having a multi-core processor structure, the operations and / or functions of the present disclosure may be performed individually or collectively by one or more cores included in the processor (1110).

[0173] A memory (1115) of a wearable device (999) according to one embodiment may include electronic components for storing data and / or instructions that are input to or output from a processor (1110). The memory (1115) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, and embedded multi-media card (eMMC). In one embodiment, the memory (1115) may be referred to as storage.

[0174] In one embodiment, a display (950) of a wearable device (999) can output visualized information to a user of the wearable device (999). A display (950) arranged in front of the eyes of a user wearing the wearable device (999) may be placed in at least a part of the housing of the wearable device (999) (e.g., a first display (950-1) and / or a second display (950-2) of FIG. 9a, FIG. 9b, FIG. 10a, and FIG. 10b). For example, the display (950) may be controlled by a processor (1110) including circuits such as a CPU, a GPU (graphic processing unit), and / or a DPU (display processing unit) to output visualized information to the user. The display (950) may include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (950) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). Embodiments are not limited thereto, for example, if the wearable device (999) includes a lens for transmitting external light (or ambient light), the display (950) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (950) may be referred to as a display panel and / or a display module. Pixels included in the display (950) may be positioned toward either of the user's two eyes when worn by the user of the wearable device (999).For example, the display (950) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0175] In one embodiment, a sensor (1120) of a wearable device (999) may generate electrical information that can be processed by a processor (1110) and / or memory (1115) from non-electronic information associated with the wearable device (999). For example, the sensor (1120) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable device (999). In addition to the GPS method, the sensor (1120) may generate information indicating the geographic location of the wearable device (999) based on a global navigation satellite system (GNSS), such as Galileo or Beidou (compass), for example. The above information may be stored in memory (1115), processed by a processor (1110), and / or transmitted to another electronic device distinct from the wearable device (999) through a communication circuit.

[0176] According to one embodiment, within the memory (1115) of the wearable device (999), one or more instructions (or commands) representing data to be processed by the processor (1110) of the wearable device (999), calculations to be performed, and / or operations may be stored. A set of one or more instructions may be referred to as a program, firmware, operating system, process, routine, sub-routine, and / or software application (hereinafter, application). For example, the wearable device (999) and / or processor (1110) may perform at least one of the operations of FIG. 3, FIG. 4a, FIG. 5, FIG. 6, and FIG. 7 when a set of a plurality of instructions distributed in the form of an operating system, firmware, driver, program, and / or software application is executed. In the following, the statement that a software application is installed in the wearable device (999) may mean that one or more instructions provided in the form of a software application (or package) are stored in memory (1115), and that said one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the wearable device (999)) by the processor (1110). For example, the application may include a program and / or library related to a service provided to the user.

[0177] Referring to FIG. 11, programs installed on a wearable device (999) may be included in any one of different layers, including an application layer (1140), a framework layer (1150), and / or a hardware abstraction layer (HAL) (1180), based on the target. For example, within the hardware abstraction layer (1180), programs (e.g., modules, or drivers) designed to target the hardware of the wearable device (999) (e.g., a display (950), and / or a sensor (1120)) may be included. The framework layer (1150) may be referred to as an XR framework layer in that it includes one or more programs for providing XR (extended reality) services. For example, the layers illustrated in FIG. 11 are logically (or for convenience of explanation) separated, and may not imply that the address space of memory (1115) is separated by said layers.

[0178] For example, within the framework layer (1150), programs designed to target at least one of the hardware abstraction layer (1180) and / or the application layer (1140) (e.g., a location tracker (1171), a spatial recognizer (1172), a gesture tracker (1173), an eye tracker (1174), and / or a face tracker (1175)) may be included. The programs included in the framework layer (1150) may provide an application programming interface (API) that is executable (or invokeable) based on other programs.

[0179] For example, within the application layer (1140), a program designed to target a user of the wearable device (999) may be included. Examples of programs included in the application layer (1140) include an XR (extended reality) system UI (user interface) (1141) and / or an XR application (1142), but embodiments are not limited thereto. For example, programs included in the application layer (1140) (e.g., software applications) may call an API to cause the execution of a function supported by programs included in the framework layer (1150).

[0180] For example, the wearable device (999) may display one or more visual objects on the display (950) to perform interaction with the user based on the execution of the XR system UI (1141). A visual object may mean an object that can be placed on the screen for the transmission of information and / or interaction, such as text, images, icons, videos, buttons, checkboxes, radio buttons, text boxes, sliders, and / or tables. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (999) may provide the user with functions available in a virtual space based on the execution of the XR system UI (1141).

[0181] Referring to FIG. 11, a lightweight renderer (1143) and / or an XR plugin (1144) are depicted within the XR system UI (1141), but are not limited thereto. For example, based on the XR system UI (1141), the processor (1110) may execute the lightweight renderer (1143) and / or an XR plugin (1144) within the framework layer (1150).

[0182] For example, a wearable device (999) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute a rendering pipeline, which is permitted to be partially modified, based on the execution of a lightweight renderer (1143). The lightweight renderer (1143) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline, which is permitted to be partially modified. The lightweight renderer (1143) may include a renderer built prior to the execution of a software application (e.g., a pre-built renderer). For example, the wearable device (999) may acquire resources (e.g., APIs, system processes and / or libraries) used to define, create, and / or execute the entire rendering pipeline based on the execution of an XR plugin (1144). The XR plugin (1144) can be referred to as an open XR native client in terms of defining (or setting) the entire rendering pipeline.

[0183] For example, the wearable device (999) may display a screen representing at least a portion of a virtual space on the display (950) based on the execution of the XR application (1142). The XR plugin (1144-1) included in the XR application (1142) may include instructions that support functions similar to the XR plugin (1144) of the XR system UI (1141). Descriptions of the XR plugin (1144-1) that overlap with descriptions of the XR plugin (1144) may be omitted. The wearable device (999) may trigger the execution of the virtual space manager (1151) based on the execution of the XR application (1142).

[0184] For example, the wearable device (999) may display an image on the display (950) in virtual space based on the execution of the application (1145). The application (1145) may be configured to output image information for displaying a two-dimensional image. The wearable device (999) may trigger the execution of a virtual space manager (1151) based on the execution of the application (1145). The wearable device (999) may generate dual image information to display the two-dimensional image in three-dimensional virtual space based on the execution of the application (1145). Here, the dual image information may include a first image information for the left eye and a second image information for the right eye, taking into account binocular parallax. To display the two-dimensional image in three-dimensional virtual space, the wearable device (999) may generate the dual image information based on the image information for displaying the two-dimensional image.

[0185] According to one embodiment, the wearable device (999) can provide a virtual space service based on the execution of a virtual space manager (1151). For example, the virtual space manager (1151) may include a platform for supporting the virtual space service. Based on the execution of the virtual space manager (1151), the wearable device (999) can identify a virtual space formed based on the user's location indicated by data acquired through a sensor (1130) and can display at least a portion of the virtual space on a display (950). The virtual space manager (1151) may be referred to as a composition presentation manager (CPM).

[0186] For example, the virtual space manager (1151) may include a runtime service (1152). For example, the runtime service (1152) may be referred to as an OpenXR runtime module (or OpenXR runtime program). The wearable device (999) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (1152). For example, the wearable device (999) may perform rendering for a virtual space service for the user based on the execution of the runtime service (1152). For example, a virtual space-related function, executable by the application layer (1140), may be supported based on the execution of the runtime service (1152).

[0187] For example, the virtual space manager (1151) may include a pass-through manager (1153). The wearable device (999) may display an image and / or video representing the real space acquired through an external camera superimposed on at least a portion of the screen while displaying a screen representing the virtual space on the display (950) based on the execution of the pass-through manager (1153).

[0188] For example, the virtual space manager (1151) may include an input manager (1154). The wearable device (999) may identify acquired data (e.g., sensor data) by executing one or more programs included within the recognition service layer (1170) based on the execution of the input manager (1154). The wearable device (999) may identify user inputs associated with the wearable device (999) using the acquired data. The user inputs may be associated with user motions (e.g., hand gestures), gaze, and / or speech identified by a sensor (1120) (e.g., an image sensor (1130) such as an external camera). The user inputs may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0189] For example, the perception abstract layer (1160) can be used for data exchange between the virtual space manager (1151) and the perception service layer (1170). In terms of being used for data exchange between the virtual space manager (1151) and the perception service layer (1170), the perception abstract layer (1160) can be referred to as an interface. As an example, the perception abstract layer (1160) can be referred to as OpenPX. The perception abstract layer (1160) can be used for a perception client and a perception service.

[0190] According to one embodiment, the recognition service layer (1170) may include one or more programs for processing data obtained from the sensor (1120). The one or more programs may include at least one of a location tracker (1171), a spatial recognizer (1172), a gesture tracker (1173), and / or an eye tracker (1174). The type and / or number of the one or more programs included in the recognition service layer (1170) are not limited to those shown in FIG. 11.

[0191] For example, the wearable device (999) can identify the posture of the wearable device (999) using the sensor (1130) based on the operation of the position tracker (1171). The wearable device (999) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (999) using data acquired using an external camera (e.g., image sensor (1121)) and / or an IMU (e.g., motion sensor (1122) including a gyroscope, accelerometer, and / or geomagnetic sensor) based on the operation of the position tracker (1171). The position tracker (1171) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0192] For example, the wearable device (999) may acquire information to provide a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (999) (or the user of the wearable device (999)) based on the execution of the space recognizer (1172). The wearable device (999) may reproduce the surrounding environment of the wearable device (999) in three dimensions using data acquired using an external camera (e.g., image sensor (1121)) based on the execution of the space recognizer (1172). The wearable device (999) may identify at least one of a plane, an incline, and a staircase based on the surrounding environment of the wearable device (999) reproduced in three dimensions based on the execution of the space recognizer (1172). The space recognizer (1172) may be referred to as a scene understanding (SU) module (or scene understanding program).

[0193] For example, the wearable device (999) can identify (or recognize) the pose and / or gesture of the user's hand of the wearable device (999) based on the execution of the gesture tracker (1173). For example, the wearable device (999) can identify the pose and / or gesture of the user's hand using data acquired from an external camera (e.g., image sensor (1121)) based on the execution of the gesture tracker (1173). For example, the wearable device (999) can identify the pose and / or gesture of the user's hand based on data (or images) acquired using an external camera based on the execution of the gesture tracker (1173). The gesture tracker (1173) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0194] For example, the wearable device (999) can identify (or track) the movement of the user's eyes of the wearable device (999) based on the execution of the eye tracker (1174). For example, the wearable device (999) can identify the movement of the user's eyes using data obtained from an eye-tracking camera (e.g., image sensor (1121)) based on the execution of the eye tracker (1174). The eye tracker (1174) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0195] For example, the recognition service layer (1170) of the wearable device (999) may further include a face tracker (1175) for tracking the user's face. For example, the wearable device (999) may identify (or track) the movement of the user's face and / or the user's facial expression based on the execution of the face tracker (1175). The wearable device (999) may estimate the user's facial expression based on the movement of the user's face based on the execution of the face tracker (1175). For example, the wearable device (999) may identify the movement of the user's face and / or the user's facial expression based on data (e.g., images and / or videos) obtained using a camera (e.g., a camera facing at least a part of the user's face) based on the execution of the face tracker (1175).

[0196] Referring to FIG. 11, the renderer (1190) may include instructions for rendering images in a three-dimensional virtual space. The processor (1110) executing the renderer (1190) may obtain at least one image to be displayed at least partially in a display area of ​​the display (950) in a software application. For example, the processor (1110) executing the renderer (1190) may determine the location of the area where an application (e.g., XR application (1142), application (1145)) will be rendered. The processor (1110) executing the renderer (1190) may generate an image of said application to be displayed on the display (950). The renderer (1190) may composite images to generate a composite image to be displayed on the display (950).

[0197] For example, a processor (1110) that executes a renderer (1190) can divide the display area of ​​a display (950) into a foveated portion (or may be referred to as a foveated area) and a peripheral portion (or may be referred to as a residual area) using a gaze position calculated using a position tracker (1171) and / or a gaze tracker (1174). For example, a processor (1110) that detects coordinate values ​​of the gaze position can determine the portion of the display area containing said coordinate values ​​as the foveated area. A DPU that executes a renderer (1190) can acquire at least one image corresponding to each of said foveated area and said residual area, having a size smaller than the size of the entire display area of ​​the display (950) or having a resolution less than the resolution of the display area.

[0198] A processor (1110) that executes a renderer (1190) can obtain or generate a composite image to be displayed on a display (950) by synthesizing an image corresponding to a foveated area and an image corresponding to a surrounding area. For example, the processor (1110) can perform upscaling to enlarge the image corresponding to the surrounding area to the size of the entire display area of ​​the display (950). On the enlarged image, the processor (1110) can combine the image corresponding to the foveated area to generate a composite image to be displayed on the display (950). Along the boundary line of the image corresponding to the foveated area, the processor (1110) can mix the enlarged image and the image corresponding to the foveated area by applying a visual effect such as blur.

[0199] FIG. 12 illustrates an example of a block diagram of an electronic device (e.g., electronic device (801), wearable device (999)) for displaying an image in a virtual space. FIG. 12 describes an example in which multiple programs / instructions for displaying an image in a virtual space are executed. The multiple programs / instructions may all be executed on a single processor (e.g., AP) or may be executed by multiple processors (e.g., AP, GPU (graphic processing unit), NPU (neural processing unit)). The meaning of being able to be executed by multiple processors is that some programs / instructions may be executed by a first processor and other programs / instructions may be executed by a second processor different from the first processor.

[0200] Referring to FIG. 12, the electronic device (801) may execute a virtual space manager (1250) (e.g., the virtual space manager (1151) of FIG. 11, CPM) to render an image in a virtual space. For the virtual space manager (1250), at least some of the descriptions of the virtual space manager (1151) of FIG. 11 may be referenced. The virtual space manager (1250) may include a platform for supporting virtual space services. The virtual space manager (1250) may include a runtime service (1251) (e.g., OpenXR Runtime), a panel renderer (1252) (e.g., 2D Panel Render), and an XR composite unit (1253) (XR Compositor). Based on the execution of the runtime service (1251), the electronic device (801) may execute at least one of a user pose prediction function, a frame timing function, and / or a spatial input function. For the runtime service (1251), at least some of the descriptions of the runtime service (1152) of FIG. 11 may be referenced. The electronic device (801) may display at least one image (video) on a panel (e.g., a 2D panel) to enable the implementation of a virtual space through a display, based on the execution of panel rendering (1252). For example, the electronic device (801) may display a rendering image corresponding to RGB information (1266) for a panel from the spatialization manager (1240) described later through a display (e.g., a display (950)). The electronic device (801) may composite an image of a real area (hereinafter, a pass-through image) captured through a camera in virtual space with an image of a virtual area based on the execution of an XR composite unit (1253) (XR Compositor). For example, the electronic device (801) can generate a composite image by merging the pass-through image and the virtual region image based on the execution of the XR synthesis unit (1253).The electronic device (801) can transmit the generated composite image to a display buffer so that the composite image is displayed. The electronic device (801) can identify a virtual space through a virtual space manager (1250) and display at least a portion of the virtual space on a display (950). The virtual space manager (1250) may be referred to as CPM. The electronic device (801) can execute the virtual space manager (1250) to render an image corresponding to at least a portion of the virtual space.

[0201] According to one embodiment, an electronic device (801) may execute a spatialization manager (1240). The spatialization manager (1240) may perform processing for displaying an image in a three-dimensional virtual space. The electronic device (801) may perform preprocessing based on the execution of the spatialization manager (1240) so that an image can be rendered in a three-dimensional virtual space through a virtual space manager (1250). For example, the electronic device (801) may perform at least some of the functions of the renderer (1190) of FIG. 11 based on the execution of the spatialization manager (1240). The electronic device (801) may process image information provided by an application (e.g., an XR application (1210), an application providing a general 2D screen that is not XR (1220), an application providing a system UI (1230)) based on the execution of the spatialization manager (1240). A spatialization manager (1240) (e.g., Space Flinger) may include a system screen manager (1241) (e.g., System scene), an input manager (1242) (e.g., Input Routing), and a lightweight rendering engine (1243) (e.g., Impress Engine). The system screen manager (1241) may be executed to display a system UI (1230). System UI-related information (1264) may be transmitted to the system screen manager (1241) from a program (e.g., API) that provides the system UI (1230). System UI-related information (1264) may be obtained through a spatializer API and / or a Same-process private API. The spatialization manager (1240) may determine the layout (e.g., position, display order) of the system UI (1230) screen in three-dimensional space through pre-allocated resources.The system screen manager (1241) may transmit image information (1267) for rendering a screen of the system UI (1230) according to the layout to the virtual space manager (1250). The input manager (1242) may be configured to process user input (e.g., user input on a system screen or app screen). The lightweight rendering engine (1243) may be a renderer for image generation (e.g., lightweight renderer (1143)). For example, the lightweight rendering engine (1243) may be used to display the system UI (1230). According to one embodiment, the spatialization manager (1240) may include a lightweight rendering engine (1243) for rendering the system UI. According to one embodiment, if the lightweight rendering engine (1243) does not have sufficient resources to render an avatar used in an HMD, at least one external rendering engine may be used. At this time, to resolve compatibility issues with external rendering (e.g., 3rd party engine), an external rendering engine support module may be added inside the spatialization manager (1240).

[0202] According to one embodiment, the electronic device may execute an application. For example, in response to the execution of an XR application (1210) (e.g., an XR application (1142), a 3D game, an XR map, or other immersive application), the virtual space manager (1250) may be executed. The electronic device (801) may provide dual image information (1261) provided from the XR application (1210) to the virtual space manager (1250). To display images in three-dimensional space, the dual image information (1261) may include two image information that account for binocular parallax. For example, the dual image information (1261) may include a first image information for the user's left eye and a second image information for the user's right eye to render in three-dimensional virtual space. Hereinafter, the term dual image information is used in the present disclosure to refer to image information for displaying images for both eyes in three-dimensional space. In addition to the dual image information, the above dual image information may utilize binocular image information, dual image information, dual image data, dual image, binocular image data, stereoscopic image information, 3D image information, spatial image information, spatial image data, 2D-3D conversion data, dimension conversion image data, binocular parallax image data, and / or equivalent technical terms. The electronic device (801) can generate a composite image by merging image layers through a virtual space manager (1250). The electronic device (801) can transmit the generated composite image to a display buffer. The composite image can be displayed on the display (950) of the electronic device (801).

[0203] According to one embodiment, the electronic device may execute at least one application among an XR application (1210) and other applications (1220) (e.g., a first application (1220-1), a second application (1220-2), ..., a Nth application (1220-N)). According to one embodiment, the application (1220) may be configured to output image information for displaying a two-dimensional image. In other words, the application (1220) may provide a two-dimensional image. For example, the application (1220) may be a video application, a schedule application, or an internet browser application. If, in response to the execution of the application (1220), image information (1262) provided from the application (1220) is provided to the virtual space manager (1250), the image information (1262) has only x-coordinates and y-coordinates within a two-dimensional plane, so it may be difficult to consider the sequential relationship between other applications centered on the user (i.e., distance from the user). The electronic device (801) may execute a spatialization manager (1240) to provide dual image information to a virtual space manager (1250), even when displaying an application (1220) that provides a general 2D screen. For example, based on the execution of the spatialization manager (1240), the electronic device (801) may receive application-related information (1263) from the first application (1220-1). For example, the application-related information (1263) may include image information representing a 2D image of the first application (1220-1) (e.g., information including RGB per pixel) and / or content information in the first application (1220-1) (e.g., characteristics of the content executed in the first application, type of content). The application-related information (1263) may be obtained through a spatializer API.Based on the execution of the spatialization manager (1240), the electronic device (801) can identify information regarding the location of the area to be rendered and the size of the area to be rendered (hereinafter, location information). Based on the execution of the spatialization manager (1240), the electronic device (801) can generate dual image information (1265, e.g., RGBx2) that takes into account the user's binocular parallax through the image information and the location information. Based on the execution of the spatialization manager (1240), the electronic device (801) can provide the dual image information (1265) to the virtual space manager (1250). By converting a simple two-dimensional image into dual image information (1265), the problem caused by the image information (1262) being directly transmitted to the virtual space manager (1250) can be resolved. Additionally, as at least some of the functions for displaying images in virtual space are performed by the spatialization manager (1240) instead of the virtual space manager (1250), the burden on the virtual space manager (1250) may be reduced.

[0204] Some of the operations described above may be executed (or performed) through an artificial intelligence (AI) system described with reference to FIG. 13.

[0205] Figure 13 is a schematic diagram of an exemplary AI system.

[0206] Referring to FIG. 13, the AI ​​system (1300) may include an input / output interface (1310), an AI framework (1320), a generative AI model (1330), and / or a knowledge repository (1390).

[0207] The input / output interface (1310) can receive input. The input may include user input and / or data obtained or generated by an electronic device (e.g., the electronic device (100) or electronic device (801) described above). The data may include images, videos, and / or sensor data generated by at least one processor of the electronic device (e.g., at least one processor (207) or processor (820)), such as: illuminance data around the electronic device obtained from a sensor or sensor hub (e.g., auxiliary processor (823); attitude data (or orientation data) of the electronic device; temperature inside the electronic device (e.g., display (208)); or temperature of at least one processor (207); size information of the display area of ​​the display (208); and / or images obtained through an image sensor of the electronic device (e.g., included in a camera module (880)). The user input may include natural language, touch data obtained through a touch circuit included in the display panel (e.g., used to identify input from a finger and / or stylus), an image displayed (and / or to be displayed) on the display panel, and / or video. By example, without limitation, the user input may be received by an input / output interface (1310) along with context information. The context information may be described as additional information obtained in relation to the user input. The context information may be related to the state at the time the user input is received (e.g., the state of the electronic device and / or the state of the surroundings of the electronic device (e.g., user state)). For example, the context information may include information about one or more software applications executed within the electronic device at the time the user input is received.For example, the above situation information may include information about the location of the electronic device (or the location of the user of the electronic device) at the time the user input is received. For example, the user input may be integrated with the situation information. For example, the user input with the situation information integrated as input may be received by the input / output interface (1310).

[0208] The input / output interface (1310) may transmit (or provide) an output. The output may include a result (or result information) generated or obtained by the AI ​​system (1300) based on at least part of the input. The format of the output may vary. For example, the output may include natural language. For example, the output may include content (e.g., media content and / or multimedia content). For example, the output may include actions related to the user of the electronic device. For example, the output may have a format according to the user settings of the electronic device.

[0209] The input / output interface (1310) can be described as a user question / response interface (1310).

[0210] The AI ​​framework (1320) can be used to obtain information (or data) about the input from the input / output interface (1310) and to control one or more components related to the AI ​​system (1300) using the obtained information.

[0211] For example, a prompt design component (1321) within an AI framework (1320) can generate or obtain prompts for a generative AI model (1330) (e.g., including a large language model (LLM) or a large multimodal model (LMM)) using the acquired information. For example, the prompt design component (1321) may be described as an AI component that uses a learning algorithm and / or a neural network to provide prompts that are enhanced over time. For example, the prompt design component (1321) can generate or obtain prompts by accessing a knowledge component (e.g., a knowledge repository (1390)) containing user preference data, a prompt library, and / or prompt examples using the acquired information. The generated prompts may be provided to the generative AI model (1330) (e.g., including an LLM or LMM).

[0212] For example, an API / plugin management component (1322) within the AI ​​framework (1320) may be used to support communication for additional information requested (or induced) in relation to the prompt provided (or to be provided) to the generative AI model (1330). For example, the API / plugin management component (1322) may be used to create or establish a channel for communication with various data sources (e.g., knowledge repository (1390)). For example, the API / plugin management component (1322) may support access to at least some of the data sources. For example, the API / plugin management component (1322) may be used to request another component (e.g., application / service component (1380)) that performs feedback (or response) according to the prompt. As a non-limiting example, information obtained (or generated) through the API / plugin management component (1322) may be provided to the prompt design component (1321) for generating a prompt. As a non-limiting example, information obtained (or generated) through the API / plugin management component (1322) may be provided to the generative AI model (1330).

[0213] For example, an improvement component (1323) within the AI ​​framework (1320) can at least partially tune (or adjust) (or change) the result (e.g., content) obtained (or output) from the generative AI model (1330). For example, the improvement component (1323) can determine or verify whether the content obtained from the generative AI model (1330) is related to the input. For example, the improvement component (1323) can determine or verify whether the content obtained from the generative AI model (1330) contains biased content. For example, the improvement component (1323) can determine or verify whether the content obtained from the generative AI model (1330) contains harmful content. For example, the improvement component (1323) can support or assist in performing additional processing to improve the content obtained from the generative AI model (1330). For example, the improvement component (1323) may support providing a hint to the user to improve the content.

[0214] A generative AI model (1330) can be described as an artificial intelligence neural network that generates feedback in response to a prompt. For example, the feedback may include additional data and / or information relative to the prompt, but relative to the prompt. For example, the feedback may include new content relative to the prompt. For example, the generative AI model (1330) may include a model that generates images and / or a model that generates language. For example, the model that generates images may include a generative adversarial network (GAN) and / or a variational autoencoder (VAE). For example, the model that generates images may include a diffusion-based generative model (e.g., a transformer VAE). For example, the model that generates language may include CHAT-GPT 3 and / or CHAT-GPT 4. For example, the generative AI model (1330) may include an LMM that generates the feedback by recognizing text, images, and / or voice.

[0215] As an example without limitation, the AI ​​framework (1320) and / or generative AI model (1330) may be included within an AI module (e.g., including a processing circuit) within the electronic device. For example, the AI ​​module may be operatively coupled with at least one processor of the electronic device (e.g., at least one processor (207) or processor (820)). For example, the AI ​​module may be operatively coupled with a display driving circuit of the electronic device (e.g., a display driving circuit or DDI). For example, the AI ​​module may be operatively coupled with a sensor hub of the electronic device for one or more sensors within the electronic device.

[0216] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs.

[0217] An electronic device as described above (e.g., electronic device (100)) may include a memory (e.g., memory (206)) for storing instructions. The electronic device may include one or more sensors (e.g., one or more sensors (211)). The electronic device may include at least one processor (e.g., at least one processor (207)). The instructions may cause the electronic device to identify an event for outputting a notification for a first text of a first length (e.g., first text (420)) when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to determine the length of the text to be used for outputting the notification by using sensor data (e.g., sensor data (410)) obtained through the one or more sensors when executed individually or collectively by the at least one processor. The above instructions may cause the electronic device to obtain a second text of the second length (e.g., a second text (430)) from the first text, based on determining the length of the text to be used to output the notification as a second length when executed individually or collectively by the at least one processor. The above instructions may cause the electronic device to output the obtained second text when executed individually or collectively by the at least one processor.

[0218] According to one embodiment, the second length may be less than the first length.

[0219] According to one embodiment, the instructions may cause the electronic device to obtain the second text from the first text using a trained model for natural language processing (e.g., a second model (425)) when executed individually or collectively by the at least one processor.

[0220] According to one embodiment, the trained model may be a first trained model. The instructions may cause the electronic device to obtain a prompt containing a natural language sentence indicating the length from the sensor data by using a second trained model (e.g., the first model (415)) when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to obtain the second text by providing the first trained model with the first text and the prompt when executed individually or collectively by the at least one processor.

[0221] According to one embodiment, the electronic device may further include output components including a display, a speaker, and a haptic actuator. The instructions may cause the electronic device to control at least one output component among the output components, which is indicated by a response obtained by providing the first text to a trained model, in order to output the second text when executed individually or collectively by the at least one processor.

[0222] According to one embodiment, the instructions may cause the electronic device to obtain a prompt indicating the state of the user of the electronic device using the sensor data when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to determine the length of text to be used to output the notification based on the prompt when executed individually or collectively by the at least one processor.

[0223] According to one embodiment, the one or more sensors may include a microphone and a camera. The instructions may cause the electronic device to identify the volume of an audio signal acquired through the microphone based on identifying the event when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to switch the state of the camera from an inactive state to an active state during a time interval based on identifying the volume exceeding a threshold volume when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to acquire at least one image through the camera in the active state during the time interval when executed individually or collectively by the at least one processor. The instructions may cause the electronic device to determine the length of text to be used to output the notification using the audio signal and the at least one image when executed individually or collectively by the at least one processor.

[0224] According to one embodiment, the instructions may cause the electronic device to determine the length of text to be used to output the notification by using the sensor data and information about the software application that causes the notification, when executed individually or collectively by the at least one processor.

[0225] According to one embodiment, the one or more sensors may include at least one of a microphone, a camera, a GPS (global positioning system), an accelerometer, and a light sensor.

[0226] A method performed by an electronic device (e.g., electronic device (100)) having one or more sensors (e.g., one or more sensors (211)) as described above may include an operation of identifying an event for outputting a notification for a first text of a first length (e.g., first text (420)). The method may include an operation of determining the length of text to be used to output the notification using sensor data (e.g., sensor data (410)) obtained through the one or more sensors. The method may include an operation of obtaining a second text of a second length (e.g., second text (430)) from the first text based on determining the length of the text to be used to output the notification as a second length. The method may include an operation of outputting the obtained second text.

[0227] According to one embodiment, the second length may be less than the first length.

[0228] According to one embodiment, the method may include the operation of obtaining the second text from the first text using a trained model for natural language processing (e.g., a second model (425)).

[0229] According to one embodiment, the trained model may be a first trained model. The method may include the operation of obtaining a prompt containing a natural language sentence indicating the length from the sensor data using a second trained model (e.g., a first model (415)). The method may include the operation of obtaining the second text by providing the first text and the prompt to the first trained model.

[0230] According to one embodiment, the electronic device may further include output components including a display, a speaker, and a haptic actuator. The method may include an operation of controlling at least one output component among the output components, which is indicated by a response obtained by providing the first text to a trained model, in order to output the second text.

[0231] According to one embodiment, the method may include an operation of obtaining a prompt indicating the state of a user of the electronic device using the sensor data. The method may include an operation of determining the length of text to be used to output the notification based on the prompt.

[0232] According to one embodiment, the one or more sensors may include a microphone and a camera. The method may include an operation of identifying the volume of an audio signal acquired through the microphone based on identifying the event. The method may include an operation of switching the state of the camera from an inactive state to an active state during a time interval based on identifying the volume exceeding a threshold volume. The method may include an operation of acquiring at least one image through the camera in the active state during the time interval. The method may include an operation of determining the length of text to be used to output the notification using the audio signal and the at least one image.

[0233] According to one embodiment, the method may include an operation of determining the length of text to be used to output the notification by using the sensor data and information about the software application that causes the notification.

[0234] According to one embodiment, the one or more sensors are,

[0235] It may include at least one of a microphone, a camera, a GPS (global positioning system), an accelerometer, and a light sensor.

[0236] In a computer-readable storage medium in which one or more programs are stored as described above, the one or more programs may include instructions that cause the electronic device (e.g., electronic device (100)) having one or more sensors (e.g., one or more sensors (211)) to identify an event for outputting a notification for a first text of a first length (e.g., first text (420)) when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to determine the length of the text to be used to output the notification by using sensor data (e.g., sensor data (410)) obtained through the one or more sensors when executed by the electronic device. The above one or more programs may include instructions that cause the electronic device to obtain a second text of the second length (e.g., second text (430)) from the first text, based on determining the length of the text to be used to output the notification when executed by the electronic device as a second length. The above one or more programs may include instructions that cause the electronic device to output the obtained second text when executed by the electronic device.

[0237] According to one embodiment, the second length may be less than the first length.

[0238] According to one embodiment, the one or more programs may include instructions that cause the electronic device to obtain the second text from the first text by using a trained model for natural language processing (e.g., a second model (425)) when executed by the electronic device.

[0239] According to one embodiment, the trained model may be a first trained model. The one or more programs may include instructions that cause the electronic device to obtain a prompt containing a natural language sentence indicating the length from the sensor data by using a second trained model (e.g., a first model (415)) when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to obtain the second text by providing the first text and the prompt to the first trained model when executed by the electronic device.

[0240] According to one embodiment, the electronic device may further include output components including a display, a speaker, and a haptic actuator. The one or more programs may include instructions that cause the electronic device to control at least one output component among the output components, which is indicated by a response obtained by providing the first text to a trained model, in order to output the second text when executed by the electronic device.

[0241] According to one embodiment, the one or more programs may include instructions that cause the electronic device to obtain a prompt indicating the state of the user of the electronic device using the sensor data when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to determine the length of text to be used to output the notification based on the prompt when executed by the electronic device.

[0242] According to one embodiment, the one or more sensors may include a microphone and a camera. The one or more programs may include instructions that cause the electronic device to identify the volume of an audio signal acquired through the microphone based on identifying the event when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to switch the state of the camera from an inactive state to an active state during a time interval based on identifying the volume exceeding a threshold volume when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to acquire at least one image through the camera in the active state during the time interval when executed by the electronic device. The one or more programs may include instructions that cause the electronic device to determine the length of text to be used to output the notification using the audio signal and the at least one image when executed by the electronic device.

[0243] According to one embodiment, the one or more programs may include instructions that cause the electronic device to determine the length of text to be used to output the notification by using the sensor data and information about the software application that causes the notification when executed by the electronic device.

[0244] According to one embodiment, the one or more sensors may include at least one of a microphone, a camera, a GPS (global positioning system), an accelerometer, and a light sensor.

[0245] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs.

[0246] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0247] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0248] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may continuously store a computer-executable program, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or several combined hardware, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0249] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0250] Therefore, other implementations, other embodiments, and equivalents to the claims set forth below are also within the scope of the claims. According to one embodiment, the method according to the various embodiments disclosed herein may be provided as a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0251] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, One or more sensors; Memory comprising one or more storage media for storing instructions; and It includes at least one processor comprising processing circuitry, and When the above instructions are executed individually or collectively by the at least one processor, Identify an event for outputting a notification for a first text of a first length, and Using sensor data obtained through the one or more of the above sensors, the length of the text to be used to output the notification is determined, and Based on determining the length of the text to be used to output the above notification as a second length, obtaining a second text of the second length from the first text, and To output the second text obtained above, causing the above electronic device, Electronic device.

2. In claim 1, the second length is, Less than the above first length, Electronic device.

3. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor, Using a trained model for natural language processing to obtain the second text from the first text, causing the above electronic device, Electronic device.

4. In claim 3, the trained model is, It is the first trained model, and When the above instructions are executed individually or collectively by the at least one processor, Using a second trained model, obtain a prompt including a natural language sentence indicating the length from the sensor data, and By providing the first text and the prompt to the first trained model, the second text is obtained. causing the above electronic device, Electronic device.

5. In claim 1, the electronic device is, It further includes output components including a display, a speaker, and a haptic actuator, and When the above instructions are executed individually or collectively by the at least one processor, In order to output the second text above, among the output components, to control at least one output component indicated by the response obtained by providing the first text to the trained model, causing the above electronic device, Electronic device.

6. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor, Using the above sensor data, a prompt indicating the state of the user of the electronic device is obtained, and Based on the above prompt, to determine the length of the text to be used to output the above notification, causing the above electronic device, Electronic device.

7. In claim 1, the one or more sensors are, Includes microphone and camera, When the above instructions are executed individually or collectively by the at least one processor, Based on identifying the above event, the volume of the audio signal obtained through the microphone is identified, and Based on identifying the volume exceeding the threshold volume, the state of the camera is switched from an inactive state to an active state during a time interval, and At least one image is acquired through the camera in the active state during the time interval, and Using the above audio signal and the above at least one image, to determine the length of the text to be used to output the notification, causing the above electronic device, Electronic device.

8. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor, Using the sensor data and information about the software application that causes the notification, to determine the length of the text to be used to output the notification, causing the above electronic device, Electronic device.

9. In claim 1, the one or more sensors are, A device comprising at least one of a microphone, a camera, a GPS (global positioning system), an accelerometer, and a light sensor. Electronic device.

10. In a non-transient computer-readable storage medium storing one or more programs, said one or more programs are, When executed by an electronic device having one or more sensors, Identify an event for outputting a notification for a first text of a first length, and Using sensor data obtained through the one or more of the above sensors, the length of the text to be used to output the notification is determined, and Based on determining the length of the text to be used to output the above notification as a second length, obtaining a second text of the second length from the first text, and To output the second text obtained above, Including instructions that cause the above electronic device, Non-transient computer-readable storage media.

11. In claim 10, the second length is, Less than the above first length, Non-transient computer-readable storage media.

12. In Claim 10, When the above one or more programs are executed by the electronic device, Using a trained model for natural language processing to obtain the second text from the first text, Including instructions that cause the above electronic device, Non-transient computer-readable storage media.

13. In claim 12, the trained model is, It is the first trained model, and When the above one or more programs are executed by the electronic device, Using a second trained model, obtain a prompt including a natural language sentence indicating the length from the sensor data, and By providing the first text and the prompt to the first trained model, the second text is obtained. Including instructions that cause the above electronic device, Non-transient computer-readable storage media.

14. In claim 10, the electronic device is, It further includes output components including a display, a speaker, and a haptic actuator, and When the above one or more programs are executed by the electronic device, In order to output the second text above, among the output components, to control at least one output component indicated by the response obtained by providing the first text to the trained model, Including instructions that cause the above electronic device, Non-transient computer-readable storage media.

15. A method executed within an electronic device comprising one or more sensors, An action for identifying an event to output a notification for a first text of a first length, and An operation to determine the length of text to be used to output the notification using sensor data obtained through the one or more sensors mentioned above, and Based on determining the length of the text to be used to output the above notification as a second length, the operation of obtaining a second text of the second length from the first text, and including the operation of outputting the second text obtained above, method.

Images