Electronic device, method, and non-transitory storage medium for providing response to user query

Abstract

The present document relates to an electronic device, method, and non-transitory storage medium for providing a response to a user query. According to an embodiment, the electronic device may comprise: an input device including at least one of a sensor, a microphone, or a camera; an output device including at least one of a speaker or a display; at least one processor including a processing circuit; and a memory for storing instructions. The instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to: receive a user query through the input device; identify a task related to the user query on the basis of receiving the user query; identify information about the task and a plurality of steps for performing the task; acquire, through the input device in real time, data related to a situation of a user who is to perform the task; identify, on the basis of the data, a specific step corresponding to the situation of the user among the plurality of steps; and output, through the output device, the information about the task and information about the specific step, in response to the user query. Various other embodiments are also possible.

Description

Electronic device, method, and non-transient storage medium for providing a response to a user query

[0001] The present disclosure relates to an electronic device, a method, and a non-transient storage medium for providing a response to a user query.

[0002] Various services and additional functions provided through electronic devices, such as portable electronic devices like smartphones, are gradually increasing. To enhance the utility value of these devices and satisfy the needs of diverse users, telecommunications service providers or electronic device manufacturers are competitively developing devices to offer various functions and differentiate themselves from competitors. Consequently, the various functions provided through electronic devices are also becoming increasingly sophisticated.

[0003] Electronic devices can provide services (e.g., programs or applications) in response to user queries. Recently, in order to provide a variety of services, electronic devices are utilizing artificial intelligence (AI) models to offer diverse services for responding to user queries.

[0004] Meanwhile, the electronic device may be implemented with at least some of the various AI models for various services as AI assistant models and / or generative AI models that provide conversational services. Depending on the implementation, the AI ​​models may operate in a form where multiple AI models are connected.

[0005] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0006] According to one embodiment of the present disclosure, an electronic device may include an input device including at least one of a sensor, a microphone, or a camera, an output device including at least one of a speaker or a display, at least one processor including a processing circuit, and a memory for storing instructions.

[0007] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to receive a user query through the input device.

[0008] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify an operation related to the user query based on receiving the user query.

[0009] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify information about the task and a plurality of steps for performing the task.

[0010] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the user to acquire data related to the situation of the user performing the task in real time through the input device.

[0011] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify a specific step among the plurality of steps corresponding to the user's situation based on the data.

[0012] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to output information about the operation and information about the specific step as a response to the user query through the output device.

[0013] According to one embodiment, a method of operation in an electronic device may include receiving a user query through an input device of the electronic device.

[0014] According to one embodiment, the method may include an operation of identifying an operation related to the user query based on receiving the user query.

[0015] According to one embodiment, the method may include an operation of confirming information about the operation and a plurality of steps for performing the operation.

[0016] According to one embodiment, the method may include an operation of acquiring data related to the situation of a user performing the task through the input device in real time.

[0017] According to one embodiment, the method may include an operation of identifying a specific step corresponding to the user's situation among the plurality of steps based on the data.

[0018] According to one embodiment, the method may include an operation of outputting information about the operation and information about the specific step as a response to the user query through an output device of the electronic device.

[0019] According to one embodiment, in a non-transient computer-readable storage medium storing one or more programs, the one or more programs may include an executable instruction that causes the electronic device to perform an operation of receiving a user query through an input device of the electronic device when executed by at least one processor of the electronic device.

[0020] According to one embodiment, the one or more programs may include an executable instruction that, when executed by at least one processor of an electronic device, causes the electronic device to execute an operation that identifies a task related to the user query based on receiving the user query.

[0021] According to one embodiment, the one or more programs may include an executable instruction that causes the electronic device to perform an operation of verifying information about the task and a plurality of steps for performing the task when executed by at least one processor of the electronic device.

[0022] According to one embodiment, the one or more programs may include an executable command that, when executed by at least one processor of an electronic device, causes the electronic device to execute an operation of acquiring data related to the situation of a user performing the task in real time through the input device.

[0023] According to one embodiment, the one or more programs may include an executable instruction that, when executed by at least one processor of an electronic device, causes the electronic device to execute an operation of identifying a specific step among the plurality of steps corresponding to the user's situation based on the data.

[0024] According to one embodiment, the one or more programs may include an executable command that, when executed by at least one processor of an electronic device, causes the electronic device to perform an operation of outputting information about the operation and information about the specific step as a response to the user query through an output device of the electronic device.

[0025] According to one embodiment, the response to the task includes information related to the performance of the identified task and information regarding a plurality of subtasks corresponding to the plurality of steps, and the information regarding the specific step includes information related to the performance of the specific step among the plurality of subtasks corresponding to the plurality of steps, and the response to the user query can be obtained through an artificial intelligence model.

[0026] According to one embodiment, the one or more programs may include executable commands that, when executed by at least one processor of an electronic device, cause the electronic device to execute an operation of obtaining additional information including a detailed description of a sub-task of a specific step through an artificial intelligence model based on data related to the user's situation, and an operation of outputting said additional information through the output device.

[0027] According to one embodiment, the one or more programs may include executable instructions that, when executed by at least one processor of an electronic device, cause the electronic device to execute the operation of obtaining a first message for guiding that the current situation does not correspond to information regarding the specific step, based on identifying that the current situation of a user at the specific step does not correspond to information regarding the specific step based on the data, and the operation of outputting the first message through the output device.

[0028] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0029] FIG. 2 is a block diagram showing an example of the configuration of an electronic device according to one embodiment.

[0030] FIG. 3 is a block diagram showing an example of an artificial intelligence model in an electronic device according to one embodiment.

[0031] FIG. 4 is a diagram illustrating an example of providing a response to a user query in an electronic device according to one embodiment.

[0032] FIGS. 5A, FIGS. 5B, FIGS. 5C, and FIGS. 5D are drawings illustrating examples of providing a response to a user query in an electronic device.

[0033] FIG. 6 is a diagram showing an example of a method of operation in an electronic device according to one embodiment.

[0034] FIG. 7 is a diagram illustrating an example of a method of operation in an electronic device according to one embodiment.

[0035] FIG. 8 is a diagram illustrating an example of providing a response to a user query in an electronic device according to one embodiment.

[0036] FIGS. 9a and 9b are drawings illustrating an example of providing a response to a user query in an electronic device according to one embodiment.

[0037] FIG. 10 is a diagram illustrating an example of providing a response to a user query in an electronic device according to one embodiment.

[0038] FIG. 11 is a diagram illustrating an example of providing a response to a user query in an electronic device according to one embodiment.

[0039] FIG. 12 is a perspective view showing the structure of an electronic device according to one embodiment.

[0040] FIG. 13a is a perspective view showing the structure of an electronic device according to one embodiment.

[0041] FIGS. 13b and FIGS. 13c are perspective views showing the structure of an electronic device according to one embodiment.

[0042] FIG. 14 is a diagram illustrating a generative artificial intelligence system according to one embodiment.

[0043] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0044] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity. The term "user" as used in the embodiments of the present disclosure may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0045] Furthermore, the term "user query" as used in this specification may be used to mean a user making a request to an electronic device by voice, but it may also be understood as or substituted with the meanings of "user request" or "user command." That is, it is obvious to those skilled in the art that, in order for a user to obtain a response or guidance, etc. from an electronic device, it is possible not only to use sentences in the form of a query or inquiry, but also to use sentences in the form of a command or request.

[0046] Furthermore, before the user first provides a query or command, the electronic device may determine whether a specific task is in progress or scheduled to be performed based on an image captured through a video input device (e.g., a camera), and based on the determination result, the electronic device may first query the user and receive the user's query, request, or command in the form of an answer to provide guidance for performing the task.

[0047] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0048] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0049] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0050] The number of processors (120) may be one or more. For example, the processor (120) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0051] The processor (120) can control the operations of the electronic device (101) by executing instructions stored in memory (130). For example, the processor (120) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0052] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0053] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0054] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0055] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0056] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0057] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0058] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0059] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0060] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0061] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0062] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0063] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0064] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0065] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0066] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0067] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0068] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0069] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0070] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0071] FIG. 2 is a block diagram showing an example of the configuration of an electronic device according to one embodiment. FIG. 3 is a block diagram showing an example of an artificial intelligence model in an electronic device according to one embodiment.

[0072] Referring to FIG. 2, an electronic device (201) according to one embodiment (e.g., the electronic device (101) of FIG. 1) may include at least one processor (210), memory (220), display (230), camera circuit (240), communication circuit (250), microphone circuit (260) including two or more microphones, and speaker (270). Not limited thereto, the electronic device (201) may be implemented identically or similarly to the electronic device (101) of FIG. 1 and may further include other components of the electronic device (101) of FIG. 1. In addition, the electronic device (201) may be configured to include other components necessary for the method of operation of the present disclosure.

[0073] An electronic device (201) according to one embodiment may be a device that provides a screen based on images of an external environment captured in real time, or a device that provides the external environment as is. For example, the electronic device (201) may be a smartphone or a device configured to be worn on a user's body (e.g., a head-mounted display (HMD) or an augmented reality (AR) glasses device). For example, the electronic device (201) may be configured to combine with an external electronic device, such as a mobile device, and may utilize components of the external electronic device (e.g., the electronic device (102 or 104) of FIG. 1) (e.g., a display module, a camera module, an audio output module, or other components). Not limited thereto, the electronic device (201) may be implemented in various forms including a camera.

[0074] An electronic device (201) according to one embodiment may be configured in an on-device form including an artificial intelligence model (e.g., an AI assistant model and / or a generative AI model in which at least part of the model provides conversational services). Without being limited thereto, the electronic device (201) may be configured to use an artificial intelligence model included in a server (e.g., the server (108) of FIG. 1).

[0075] Referring to FIGS. 2 and FIGS. 3, a processor (210) of an electronic device (201) according to one embodiment (e.g., processor (120) of FIG. 1) receives a user query (e.g., user input information) through an input device comprising at least one of at least one microphone, at least one sensor, or at least one camera, and can verify an operation related to the user query. According to one embodiment, the processor (210) can acquire an image acquired through at least one camera included in the camera circuit (240) as a user query. The processor (210) can acquire audio information including a voice corresponding to a user's speech or a sound source generated at the location where the user is located through at least one microphone included in the microphone circuit (260) as a user query. The processor (210) can obtain information detecting a user's behavior through at least one sensor included in the sensor circuit (280) (e.g., an accelerometer, a gyro sensor, a GPS (global positioning system) sensor, or various sensors embedded in an electronic device) and / or information detected from a device or object within the location where the user is located as a user query. The processor (210) can receive a user query from an external electronic device (e.g., the electronic device (102, 104) of FIG. 1) through the communication circuit (250).

[0076] According to one embodiment, the processor (210) can identify a specific task to be performed based on a user query. According to one embodiment, when the processor (210) receives a user query, it can determine whether the user query is an input related to the task (e.g., a trigger input). If the user query is an input related to the task, the processor (210) can analyze the user query to identify the task to be performed by the specified user and provide a response to the user query related to the identified task. If the processor (210) identifies that the user query is an input not related to the task, it can provide a simple response to the user query. For example, if the processor (210) receives a user query such as "Tell me how to cook ramen," it can identify it as an input related to the task and identify the task meant by the user query as "cooking ramen." For example, when the processor (210) receives a user query such as "Tell me how to change a car tire," it can identify the task as "tire replacement" as an input related to the task. For example, when the processor (210) receives a user query such as "Tell me how to get to Seoul City Hall," it can identify the task as "directions to Seoul City Hall" as an input related to the task. For example, when the processor (210) receives a user query such as "Tell me how to repot a blueberry," it can identify the task as "repotting a blueberry" as an input related to the task. For example, when the processor (210) identifies a task (e.g., "repotting a blueberry"), if it determines based on data related to the user's situation that the user will not perform the task immediately, it can bypass the task so that it does not obtain a response immediately, and later, when the user wishes to perform the task, it can store information about the identified task in memory (220) as reservation information.If the processor (210) determines, based on data related to the user's situation, that the user will not perform the task immediately, it may store a response containing information about the acquired task in memory and output an additional response through an output device (e.g., display and / or speaker) containing simple information related to the task, guidance information to induce the performance of the task and / or information to check whether the task is performed (e.g., "Will you be repotting the blueberry later?", "Would you like an explanation of the blueberry repotting next?", or "Would you like an explanation of the detailed repotting method?"). If the processor (210) determines, based on data related to the user's situation, that the user will not perform the task, it may not acquire a detailed explanation of the task (e.g., "repotting the blueberry"), but instead acquire a response containing information about sub-tasks for multiple steps, and store the acquired response in memory (220) so that it can be output together with the detailed explanation of the task (e.g., "repotting the blueberry") when the task (e.g., "repotting the blueberry") is acquired. According to one embodiment, the processor (210) can obtain data related to the user's situation from an external electronic device (e.g., a wearable electronic device, IoT, or CCTV (closed-circuit television)).

[0077] According to one embodiment, the processor (210) can identify a task to be performed through the task execution module (310) of FIG. 3. Here, the task execution module (310) may be composed of some of the models included in the artificial intelligence model (320) or may be composed of a module (e.g., software configuration, program, or function) included in the memory (220) executed by the processor (210). For example, if a user query includes an image, the processor (210) may classify the objects included in the image into people or things through the task execution module (310) and analyze the classified objects to identify a task to be performed by the user (e.g., inference). For example, if a user query includes text information or voice information, the processor (210) may classify the text information or voice information into words through the task execution module (310) and analyze the classified words to identify a task to be performed by the user based on selected key words (e.g., inference).

[0078] According to one embodiment, the processor (210) identifies a user based on a user query through a task execution module (310), and can identify a task to be performed by the identified user based on context information stored in memory (220) in relation to the identified user and / or user query (e.g., user's personal information, health information, medical information, activity information, biometric information, history information related to tasks, information related to various applications executed on an electronic device, or other various information collected or managed using an artificial intelligence model).

[0079] According to one embodiment, the processor (210) can identify multiple steps of a task through the task execution module (310). For example, when the task execution module (310) receives user input, it may acquire information about the task, including information about subtasks for multiple steps generated through a generative artificial intelligence model (e.g., a large multimodal model (LLM), a large vision model (LVM), or a large multimodal model (LMM)). The processor (210) may store information about the task in memory (220). Here, information about the task may include information related to the execution of the identified task and information about multiple subtasks for the task classified into multiple steps. An electronic device according to one embodiment may acquire information related to the task history (e.g., history information on previously performed tasks and / or history information on the execution of subtasks prior to the current subtask) and / or information related to prohibitions regarding the task as additional information that can be used to obtain a response regarding the task. Here, multiple steps for performing a task can be classified, for example, according to the type of task and user information (e.g., age, gender, occupation, medical information, or other information related to the user).

[0080] According to one embodiment, the processor (210) can acquire data related to the situation of a user performing a task in real time through an input device (e.g., at least one microphone, at least one sensor and / or at least one camera). The electronic device can acquire data related to the user's current situation by monitoring the user's situation in real time.

[0081] According to one embodiment, the processor (210) can identify a specific step corresponding to the user's situation among a plurality of steps based on data related to the user's current situation. The processor (210) can identify a specific step corresponding to the user's situation using an artificial intelligence model (320) (e.g., a first generative artificial intelligence model (LLM 1) (321)).

[0082] According to one embodiment, the processor (210) may obtain a response to a user query that includes information about a task and information about a specific step, based on information about a task and data related to the user situation. The processor (210) may verify the response to the user query generated (e.g., obtained) using an artificial intelligence model (320) (e.g., a second generative artificial intelligence model (LLM 2) (323)). Here, the information about the specific step included in the response to the user query may include identification information of the specific step and information related to the performance of subtasks of the specific step. The information about the specific step may further include additional guidance information regarding subtasks of the specific step or information related to actions prohibited in the specific step.

[0083] According to one embodiment, the processor (210) may output a response through an output interface of an electronic device (201). Here, the output interface may include a display (230), a speaker (270), and / or a haptic module (e.g., the haptic module (179) of FIG. 1). According to one embodiment, the processor (210) may monitor the current user's situation to determine the progress status of a task (e.g., degree of progress or rate of progress) based on data related to the user's situation obtained in real time, and obtain and output a response in response to the determined progress.

[0084] According to one embodiment, the processor (210) can use an artificial intelligence model (320) to identify an additional response including additional information including a detailed description of a sub-task of the current stage based on data related to the user's situation acquired in real time at the current stage, and can display the identified response through a display (230) or output it as audio through a speaker (270).

[0085] According to one embodiment, the processor (210) can determine whether the user's situation does not correspond to a sub-task of a specific step based on data related to the user's situation obtained in real time at a specific step (e.g., a currently performing step). If the processor (210) identifies that the user's current situation is a user action unrelated to a sub-task of a specific step or a pre-specified prohibited action (e.g., an action) at the current step, it can determine that the user's current situation does not correspond to a sub-task of a specific step. For example, if the processor (210) determines that the user is performing a different action instead of performing a sub-task of a specific step provided as a response, it can output a guidance message for the other action (e.g., a first guidance message). The processor (210) can display the guidance message in real time through a display (230) or output it in real time as audio through a speaker (270).

[0086] According to one embodiment, the processor (210) checks the progress status (e.g., degree of progress or progress rate) of a subtask of a currently performing step, and if it identifies that the checked progress status is a situation where the current step is finished (e.g., a situation where a progress rate greater than a specified progress rate is checked), it obtains a guidance message (e.g., a second guidance message) to guide the subtask of the next step based on information about the task, and can provide the guidance message in advance (e.g., output through a display or speaker) before the subtask of the currently performing step is completed. For example, if a time limit (e.g., time limit) is specified for a plurality of steps, the processor (210) checks the progress status (e.g., elapsed time) of the subtask using a timer, and if the time limit is exceeded, it can provide a guidance message.

[0087] According to one embodiment, the processor (210) may provide a guidance message (e.g., a third guidance message) containing information about the required action when the user's situation changes to the next stage. The processor (210) may determine that the user has not performed the required action at the current stage based on the information about the required action included in the response to the user's query and data related to the user's situation.

[0088] According to one embodiment, when the processor (210) provides a response to a user query, it may obtain information related to a specific person (e.g., a chef or a mother) in relation to the task and provide a response to the user query based on the obtained information related to the specific person. For example, if the task to be performed is identified as “Mom’s recipe for making soybean paste stew,” the processor (210) may obtain information about “Mom’s recipe” from memory (220) or an external electronic device, and based on the information about “Mom’s recipe,” verify the response to the user query obtained by the artificial intelligence model (320) and provide the verified response. For example, the processor (210) may provide a response including a guide for performing the task using an artificial intelligence model during the process in which the user performs the task based on information existing in an external electronic device or information registered on a server (e.g., instructions), such as assembling furniture or making toys.

[0089] According to one embodiment, the processor (210) may provide a response to a user query using historical information previously performed by the user regarding a task, historical information of other users regarding a task from an external electronic device, and information obtained from an executable application regarding the performance of a task. For example, if the user query does not specify a particular person and includes a chef's recipe, the processor (210) may provide recommendation information (e.g., information about recommended chefs) through a search application for searching for the chef's recipe (e.g., displayed as text on an execution screen or output as audio). When one of the recommended chefs is selected, the processor (210) may transmit person information about the selected chef and information related to the task to an artificial intelligence model (320) through a task performance module (310). The artificial intelligence model (320) may generate a response to the user query regarding a task to be performed using the recipe for the selected chef.

[0090] According to one embodiment, the processor (210) may acquire images of a video that captures the external environment of the electronic device (201) in real time through at least one camera included in the camera circuit (240) and display the acquired images on the display (230). According to one embodiment, if the electronic device (201) is, for example, a VR (virtual reality) device, the processor (210) may use the camera circuit (240) to control the display (230) so that a real space (e.g., external environment) including the screen is visible through the display (230). According to one embodiment, if the electronic device (201) is, for example, an AR device, the processor (210) may control the display (230) (e.g., a transparent member) so that a real space including the screen is visible through the user's eyes.

[0091] According to one embodiment, the processor (210) may be a hardware component (function) or a software element (program) comprising at least one component provided in the electronic device (201), such as a hardware module or a software module (e.g., an application program). According to one embodiment, the processor (210) may include, for example, one or more combinations of hardware, software, or firmware. The processor (210) may be configured to omit at least some of the components or to include additional components for performing image processing operations in addition to the components.

[0092] According to one embodiment, the memory (220) (e.g., the memory (130) of FIG. 1) may store applications. For example, the memory (220) may store applications (functions or programs) for providing responses related to user queries, responses related to user queries, data obtained by monitoring user situations, and / or information about tasks. The memory (220) may store context information learned using an artificial intelligence model in a designated database (e.g., a context information data database). The memory (220) may store an artificial intelligence model (AI). The memory (220) may store images captured through at least one camera included in an external electronic device or camera circuit (240), audio information detected through two or more microphones, and information detected through at least one sensor. According to one embodiment, the memory (220) may store various data generated during the execution of a program (140), including a program used for function operations (e.g., the program (140) of FIG. 1). For example, the memory (220) may include a program area (140) and a data area (not shown). The program area (140) may store relevant program information for operating the electronic device (201), such as an operating system (OS) (e.g., the operating system (142) of FIG. 1) that boots the electronic device (201). The data area (not shown) may store transmitted and / or received data and generated data according to various embodiments.Additionally, the memory (220) may be configured to include at least one storage medium among flash memory, hard disk, multimedia card micro type memory (e.g., secure digital (SD) or extreme digital (XD) memory), RAM (random access memory), and ROM (read only memory).

[0093] According to one embodiment, a display (230) (e.g., a display module (160) of FIG. 1) may display an execution screen of an application for providing a response to a user query under the control of a processor (210). The display (230) may display a response to a user query. The display (230) may display guidance messages regarding tasks performed by the user. The display (230) may display an image of the user's current situation captured through a camera in relation to the performance of tasks. According to one embodiment, the display (230) may be implemented in the form of a touch screen. When the display (230) is implemented in the form of a touch screen together with an input module, it may display various information generated according to the user's touch actions. According to one embodiment, the display (230) may be composed of at least one of an LCD (liquid crystal display), a TFT-LCD (thin film transistor LCD), an OLED (organic light emitting diodes), an LED, an AMOLED (active matrix organic LED), a flexible display, and a 3-dimensional display. Additionally, some of these displays may be configured to be transparent or light-transmitting so that the outside can be seen through them. This may be configured in the form of a transparent display including a TOLED (transparent OLED). According to one embodiment, in addition to the display (230), other display modules (e.g., an extended display or a flexible display) may be further included.

[0094] According to one embodiment, the camera circuit (240) (e.g., the camera module (180) of FIG. 1) may include at least one camera and may capture images related to the user's situation to acquire data related to the user's situation. The camera circuit (240) may capture images (e.g., 2D images or 3D images) of the external environment so that the actual external environment is displayed through a display in a real space (e.g., virtual reality space, augmented reality space, or mixed reality space) or on a screen corresponding to the real space (e.g., to display a screen). The configuration and operation of at least one camera included in the camera circuit (240) may be the same or similar to the camera configuration of a wearable device such as AR glasses or a head-mounted device (HMD).

[0095] According to one embodiment, the communication circuit (250) (e.g., the communication module (190) of FIG. 1) can communicate with an external electronic device (e.g., the electronic device (102, 104) of FIG. 1) or a server (e.g., the server (108) of FIG. 1). According to one embodiment, the communication circuit (250) may include a cellular module, a WiFi (wireless fidelity) module, a Bluetooth module, or a NFC (near field communication) module.

[0096] According to one embodiment, a microphone circuit (260) (e.g., input module (150) of FIG. 1) includes at least one microphone and can receive data related to a user query or the user's situation (e.g., voice information of the user or sound source information generated in the user's surroundings) through at least one microphone.

[0097] According to one embodiment, the speaker (270) (e.g., the sound output module (155) of FIG. 1) can output audio corresponding to a response to a user query. According to one embodiment, the speaker (270) can output voice information corresponding to a guidance message.

[0098] According to one embodiment, the sensor circuit (280) (e.g., the sensor module (176) of FIG. 1) includes at least one sensor and can detect data related to the user's situation through at least one sensor. The sensor circuit (280) can detect changes in the user's behavior or objects around the user in a plurality of steps, either periodically or non-periodically, through at least one sensor.

[0099] FIG. 4 is a diagram illustrating an example of providing a response to a user query in an electronic device according to one embodiment. FIG. 5a, FIG. 5b, FIG. 5c, and FIG. 5d are diagrams illustrating examples of providing a response to a user query in an electronic device. A specific description of an example of providing a response related to an operation with reference to FIG. 4 and FIG. 5a, FIG. 5b, FIG. 5c, and FIG. 5d will be described with reference to the configuration examples of FIG. 1, FIG. 2, and FIG. 3.

[0100] Referring to FIG. 4, a task execution module (310) of an electronic device (201) according to one embodiment may receive a user query through at least one camera or at least one microphone. In addition, the task execution module (310) may receive a user query through text input. The task execution module (310) may analyze the user query to identify that the user query is an input related to a task and identify the task to be performed by the user. For example, if the task execution module (310) identifies that the user query received through at least one microphone is “Tell me how to repot a blueberry,” it may identify “repotting a blueberry” as the task to be performed.

[0101] According to one embodiment, the processor (210) can automatically determine whether a task (e.g., “repotting blueberries”) has started based on data related to the user’s situation detected through at least one sensor, at least one microphone and / or at least one camera. For example, if a user pulls blueberries out of an existing pot while repotting blueberries and inputs a user query saying “What should I do with the soil?”, the processor can identify that the user is performing the blueberry repotting task, check the subtask currently being performed (e.g., pulling blueberries out of an existing pot), and then provide the user with a response (e.g., guide) obtained by querying an artificial intelligence model (320) (e.g., a generative artificial intelligence model (e.g., LLM)) regarding the necessary action. According to one embodiment, even if the processor (210) does not receive a query corresponding to the task (e.g., repotting blueberries) being performed, it can provide a response appropriate to the user’s situation based on data related to the user’s situation detected in real time, such as the user’s action or the user’s utterance (e.g., “What should I do with the soil?”).

[0102] Referring to FIG. 5a, according to one embodiment, a task execution module (310) can identify subtasks (510) for a task obtained by a generative artificial intelligence model (320). The generative artificial intelligence model (320) can obtain (e.g., generate) subtasks (510) classified into execution steps in the order of performing the task for a user query or a task to be performed (e.g., “repotting blueberries”). The generative artificial intelligence model (320) can obtain subtasks (510) classified into five steps. For example, the generative artificial intelligence model (320) can identify generated subtasks (510) classified into: Step 1, “preparation of necessary materials (new pot, soil, water); Step 2, “carefully removing blueberries from the standard pot”; Step 3, “removing old soil and trimming roots”; Step 4, “putting blueberries into the new pot and filling with new soil”; and Step 5, “watering sufficiently and placing in a bright place”. According to one embodiment, the electronic device (201) may display a first response containing the subtasks (510) in response to a user query through a display (230) or output it through a speaker (270).

[0103] According to one embodiment, the task execution module (310) monitors the user's situation through at least one camera, at least one microphone and / or at least one sensor, and based on data related to the user's situation acquired in real time, checks the subtask of the step currently being performed by the user among the subtasks (510) for a plurality of steps, and checks the progress status (e.g., degree of progress or progress rate) of the subtask of the step currently being performed, and if necessary, changes or newly establishes the subtasks and steps that were previously checked and output to the user, and provides new guidance to the user through an output device. For example, if the task execution module (310) identifies that the subtask of the step being performed by the user is step 2 (e.g., “carefully remove blueberries from an existing flowerpot”), it can identify that the progress status of the subtask being performed by the user (e.g., removing blueberries from an existing flowerpot) is in a state where step 2 has been completed based on data related to the user's situation acquired in real time. Based on identifying that the second step has been completed, the task execution module (310) transmits an input prompt to the artificial intelligence model (320) based on identification information of the subtask to be performed and data related to the user's situation obtained in real time, and the artificial intelligence model (320) can obtain (e.g., generate) a response (521) (e.g., a second response) for the current subtask based on information about the step of the subtask and data related to the user's situation. The task execution module (310) can obtain a response through the artificial intelligence model (320) based on data related to the user's situation obtained in real time, even if no additional query (input information) is received from the user. For example, the artificial intelligence model (320) can generate “Well done, the step of removing blueberries is now complete.” as a response (521) for the second step subtask.If a request for a response to the next step is set in the input prompt delivered to the artificial intelligence model (320), the model may obtain (e.g., generate) a guidance message (523) containing additional information related to the subtask of the next sequence, step 3 (e.g., "Step 3: Remove old soil and tidy up roots") (e.g., "The next step is step 3, removing old soil and tidying up roots. After taking out the blueberries, shake off any old soil remaining on the roots and trim any roots that have grown too long."). According to one embodiment, the electronic device (201) may display the response (521) to the subtask (e.g., a second response) and / or the guidance message (523) through a display (230) or output through a speaker (270). Here, the guidance message (523) may be included in the second response (521) and output as a response to the subtask, or output as separate guidance information.

[0104] According to one embodiment, as illustrated in FIG. 5c, an artificial intelligence model (320) may obtain (e.g., generate) a response (531) containing a detailed description related to a subtask currently being performed or a subtask to be performed next, when a request for additional options (e.g., providing a detailed description) is set in the input prompt transmitted from the task execution module (310) for a user request or for subtasks. According to one embodiment, an electronic device (201) may display the guidance message (531) containing the detailed description through a display (230) or output it through a speaker (270). A guidance message (531) containing detailed instructions may include instructions regarding the performance of the sub-tasks of step 3, such as: “First, please remove old soil and trim the roots. After taking out the blueberries, you need to remove any old soil remaining on the roots. This is to help them adapt well to the new environment, and the method is as follows. First, soak the roots of the blueberries in water. This will help the old soil fall off easily. Next, carefully remove any old soil remaining on the roots using a soft toothbrush or a small brush. Please be careful not to damage the roots during this process. If the roots are too long, it is also good to cut off some of them. This will help the blueberries adapt to the new environment more quickly. Trimming the roots is an important process that helps the blueberries grow healthily. Therefore, careful attention is required at this stage.”

[0105] Referring to FIG. 5d, according to one embodiment, a task execution module (310) can set an input prompt based on information related to the task, data related to the user's situation obtained in real time by monitoring the user's situation, and information related to the subtask being performed by the user identified based on the data (e.g., information about the steps of the subtask and information about the progress status), and transmit the set input prompt to an artificial intelligence model (320). Based on the input prompt, the artificial intelligence model (320) can generate a plurality of subtasks (e.g., information about the steps and the process and method of performing each step of the subtask) (541) and guidance information (542) for the task, and generate information (543) related to the execution of the subtask according to the current user's situation. The artificial intelligence model (320) can obtain (e.g., generate) a response (540) containing a plurality of subtasks (541), guidance information (542), and information (543) related to the performance of the subtasks as a response (e.g., first response) to a user query. According to one embodiment, the electronic device (201) may display a response (540) through a display (230) or output it through a speaker (270), which includes a plurality of sub-tasks (541), guidance information (542) (e.g., “The above steps are approximate and may vary depending on the condition and environment of the plant when actually repotting. Also, since blueberries prefer acidic soil, it is recommended to use blueberry-specific soil or peat moss.”) and information (543) related to the performance of the sub-tasks (e.g., “It appears you are currently in step 3. Step 3 is the step of removing old soil and tidying up the roots. After removing the blueberry, shake off any old soil remaining on the roots and trim any roots that have grown too long.”).

[0106] According to one embodiment, the task execution module (310) monitors the user's situation (e.g., user behavior) at the current stage of the subtask, checks whether the subtask is executed, and provides a response depending on whether the task is executed.

[0107] According to one embodiment, the processor (210) of the electronic device (201) may output a message to ask a user a question using an artificial intelligence model (320), and if it is determined that the verified information is inaccurate based on data related to the user's situation, it may ask the user to obtain accurate information. For example, in a situation where the user is boiling ramen and has put in 3-4 noodles but has not been able to determine the exact number, the processor may ask the user, "How many ramen noodles did you put in?" and obtain an answer from the user to obtain accurate information and update the information regarding the operation.

[0108] According to one embodiment, the processor (210) of the electronic device (201) may provide additional feedback regarding the current subtask from the user while the user is performing the current subtask (e.g., "checking the amount of water added to boil ramen and providing feedback that it would be better to add a little more water"). Subsequently, if the processor (210) confirms that the user has performed an action regarding the additional response based on the acquired data, it may provide feedback on the confirmed user action (e.g., providing feedback that the appropriate amount of water was added). This feedback may be output via audio through the speaker (270), displayed through the display (230) of the electronic device (201), or output through the display of a connected external electronic device. Alternatively, the processor (210) of the electronic device (201) may provide real-time feedback regarding the current subtask to the user while the user is performing the current subtask (e.g., if the user is continuously adding water to boil ramen, it may determine whether the amount is appropriate and provide real-time guidance to the user to stop supplying additional water). This is done by inputting data related to the user situation to an input device (at least It may be possible to acquire in real time through one camera and / or at least one sensor, etc.

[0109] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2) may implement a related software module (e.g., the program (140) of FIG. 1) for providing a response to a user query. The memory of the electronic device (e.g., the memory (130) of FIG. 1 and / or the memory (420) of FIG. 4) may store instructions (e.g., instructions) to implement the software module. At least one processor (e.g., processor (120) of FIG. 1 and / or processor (210) of FIG. 2) can execute instructions stored in memory to implement a software module and can control hardware associated with the function of the software module (e.g., sensor module (176) of FIG. 1, camera module (180), communication module (190) of FIG. 1 and / or communication circuit (250) of FIG. 2, display module (160) of FIG. 1 and / or display (230) of FIG. 2).

[0110] A software module of an electronic device (101, 201) according to one embodiment may be configured to include a kernel (or a hardware abstraction layer (HAL)), a framework (e.g., middleware (144) of FIG. 1), and an application (e.g., application (146) of FIG. 1). At least some of the software modules may be preloaded onto the electronic device (101, 201) or downloadable from a server (e.g., server (108)).

[0111] According to one embodiment, the kernel may include, for example, a system resource manager or a device driver, but may be configured to include other modules, not limited thereto. The system resource manager may perform control, allocation, or reclamation of system resources. The device driver may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WIFI driver, an audio driver, or an IPC (inter-process communication) driver.

[0112] According to one embodiment, the framework may provide various functions to the application through an application programming interface (API) (not shown) to provide functions commonly required by the application or to enable the application to efficiently use limited system resources within the electronic device (101, 201). The framework may include modules that form combinations of various functions of the components. The framework may provide modules specialized for each type of operating system to provide differentiated functions. The framework may dynamically delete some existing components or add new components. According to one embodiment, the application may include an application received from an external electronic device (e.g., a server (108) or an electronic device (102, 104)). According to one embodiment, the application may include a preloaded application or a third-party application downloadable from the server. The components of the software module and the names of the components according to the illustrated embodiment may vary depending on the type of operating system. According to one embodiment, at least a portion of the software module may be implemented in software, firmware, hardware, or a combination of at least two of these. At least a portion of the software module may be implemented (e.g., executed) by a processor (e.g., an application processor (AP)). At least a portion of the software module may include, for example, a module, a program, a routine, a set of instructions, or a process for performing at least one function.

[0113] According to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2) may include a camera circuit including at least one camera (e.g., camera module (180) of FIG. 1 or at least one camera included in camera circuit (240) of FIG. 2), a display (e.g., display module (160) of FIG. 1 or display (230) of FIG. 2), a microphone circuit including a plurality of microphones (e.g., input module (150) of FIG. 1, microphone circuit (260) of FIG. 2), a speaker (e.g., sound output device (155) of FIG. 1, speaker (270) of FIG. 2), at least one processor (processor (120) of FIG. 1 or processor (210) of FIG. 2)), and a memory for storing instructions (e.g., memory (130) of FIG. 1 or memory (220) of FIG. 2).

[0114] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to receive a user query through the input device.

[0115] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify an operation related to the user query based on receiving the user query.

[0116] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify information about the task and a plurality of steps for performing the task.

[0117] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the user to acquire data related to the situation of the user performing the task in real time through the input device.

[0118] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify a specific step among the plurality of steps corresponding to the user's situation based on the data.

[0119] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to output information about the operation and information about the specific step as a response to the user query through the output device.

[0120] According to one embodiment, information regarding the task includes information related to the performance of the identified task and information regarding a plurality of subtasks corresponding to the plurality of steps, and information regarding the specific step includes information related to the performance of the specific step among the plurality of subtasks corresponding to the plurality of steps, and the response to the user query can be obtained through an artificial intelligence model.

[0121] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to obtain additional information including a detailed description of a subtask of the specific step through an artificial intelligence model based on data related to the user's situation, and to output the additional information through the output device.

[0122] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to obtain a first message for guiding that the current situation does not correspond to the information regarding the specific step, based on identifying that the user's current situation does not correspond to the information regarding the specific step based on the data, and to output the first message through the output device.

[0123] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to check the progress status of a subtask of the specific step based on the data, obtain a second message to guide information about a subtask of the next step in sequence before the subtask of the specific step is completed based on the progress status, and output the second message through the output device.

[0124] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to obtain a third message guiding the execution of the essential task at the specific step based on identifying that the specific step is changed to the next step in sequence based on the data, and based on identifying that the essential task at the specific step is not completed, and to output the third message through the output device.

[0125] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to obtain information related to a specific person based on identifying that the task is related to a specific person based on the identified task and the data, and to output information related to the execution of the task obtained based on the information related to the specific person and the data as a response to the user query.

[0126] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to output through the output device an additional response including information that induces the electronic device to perform the task through the artificial intelligence model and information about the task, based on identifying that the electronic device does not immediately perform the task based on the data.

[0127] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to output information about the operation and information about the specific step in real time.

[0128] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to acquire a fourth message regarding the performance of a new task based at least partially on data related to a new situation of the user received in real time, and to output the fourth message through the output device. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify a user based on user authentication information input through the input device, and to output information regarding the task and information regarding the specific step in real time based on the context information of the identified user.

[0129] According to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2)) is an input device comprising at least one of a sensor (e.g., sensor module (176) of FIG. 1, sensor circuit (280) of FIG. 2), a microphone (e.g., input module (150) of FIG. 1, microphone circuit (260) of FIG. 2), or a camera (e.g., at least one camera included in the camera module (180) of FIG. 1 or camera circuit (240) of FIG. 2). It may include an output device comprising at least one of a speaker (e.g., the sound output device (155) of FIG. 1, the speaker (270) of FIG. 2) or a display (e.g., the display module (160) of FIG. 1 or the display (230) of FIG. 2), at least one processor comprising a processing circuit (the processor (120) of FIG. 1 or the processor (210) of FIG. 2), and a memory for storing instructions (e.g., the memory (130) of FIG. 1 or the memory (220) of FIG. 2).

[0130] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to receive user input through the input device, determine whether the user input is a trigger input for performing the task, and if the user input is determined to be a trigger input for performing the task, receive information about the task from the memory or from outside the electronic device, receive data related to the situation of the user performing the task in real time through the input device, and output a guide regarding the performance of the task through the output device based at least on the information about the task and the data related to the situation of the user.

[0131] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to output a guide for the performance of the task in real time through the output device.

[0132] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may output a guide for performing a new task based at least in part on data related to the user's situation received in real time.

[0133] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be able to query a user through the output device and output a guide for performing the task through the output device based at least on the user's answer obtained through the input device, information about the task, and data related to the user's situation.

[0134] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device can identify a user based on user authentication information input through the input device and output a guide for performing the task based on context information of the identified user.

[0135] FIG. 6 is a diagram illustrating an example of a method of operation in an electronic device according to one embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. In the following embodiments, each operation may be understood to be performed in a processor (e.g., processor (120) of FIG. 1 or processor (210) of FIG. 2) of an electronic device (e.g., electronic device (101) of FIG. 1 or electronic device (201) of FIG. 2).

[0136] Referring to FIG. 6, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can verify a user query obtained through an input device comprising at least one microphone (e.g., at least one microphone included in the input module (150) of FIG. 1 or the microphone circuit (260) of FIG. 2), at least one sensor (e.g., at least one sensor included in the sensor module (176) of FIG. 1 or the sensor circuit (280) of FIG. 2), or at least one camera (e.g., at least one camera included in the camera module (180) of FIG. 1 or the camera circuit (240) of FIG. 2) in operation 601. The electronic device according to one embodiment can obtain an image obtained through at least one camera as a user query. The electronic device according to one embodiment can obtain audio information including voice corresponding to a user's speech or a sound source generated at the location where the user is located through at least one microphone included in the microphone circuit (260) as a user query. An electronic device according to one embodiment can obtain information detecting a user's behavior through at least one sensor (e.g., an accelerometer, a gyro sensor, a GPS sensor, or various sensors embedded in the electronic device) and / or information detected from a device or object within the location where the user is located as a user query.

[0137] In operation 603, an electronic device according to one embodiment can analyze a user query to identify a task related to the user query. According to one embodiment, the electronic device can determine, based on the result of analyzing the user query, whether the user query is a simple response input (e.g., first trigger input) or an input for a task (e.g., second trigger input). If the user query is identified as an input for a task, it can provide a response for the task through the operation method of FIG. 6. If the electronic device according to one embodiment identifies that the user input is not an input for a task, it can obtain and provide a simple response unrelated to the task to be performed by the specified user. According to one embodiment, the electronic device can analyze the user query through a task execution module (310) to identify a task to be performed by the user. For example, if the user query includes an image, the electronic device can classify the objects included in the image as people or objects through the task execution module, and analyze the classified objects to identify the task to be performed by the user (e.g., inference). For example, if a user query includes text information or voice information, the processor (210) classifies the text information or voice information into words through the task execution module (310), analyzes the classified words, and can identify (e.g., inference) the task to be performed by the user based on selected key words. Additionally, an electronic device according to one embodiment may identify who is using the electronic device based on user input (e.g., characteristics derived through waveform analysis of the user's voice). That is, based on analysis data of the voices of multiple users (User A, User B, …) previously input through the electronic device, it is possible to identify which user provided the input user query.In addition, an electronic device according to one embodiment includes a camera or a sensor, and can identify and specify which user is currently wearing the electronic device by recognizing user authentication information (e.g., iris, fingerprint) input through the camera or fingerprint sensor. In addition, for example, the electronic device can identify the user based on a user query through a task execution module, and can determine the task to be performed by the identified user based on context information stored in the electronic device's memory (e.g., memory (130) of FIG. 1 or memory (220) of FIG. 2) or a server in relation to the user query and the identified user (e.g., user personal information, health information, medical information, activity information, biometric information, history information related to tasks, information related to various applications executed on the electronic device, or other various information collected or managed using an artificial intelligence model).

[0138] In operation 605, an electronic device according to one embodiment may identify information regarding an identified task and a plurality of steps for performing the task. The information regarding the task may include information related to the performance of the identified task and information regarding a plurality of subtasks for the task classified into a plurality of steps. An electronic device according to one embodiment may obtain, as additional information that can be used to obtain a response regarding the task, history information regarding the task (e.g., history information regarding previously performed tasks and / or history information regarding the performance of subtasks prior to the current subtask) and / or information regarding prohibitions regarding the task. Here, the plurality of steps for performing the task may be classified according to, for example, the type of task and user information (e.g., age, gender, occupation, medical information, or other information related to the user). The electronic device may obtain information regarding the identified task and a plurality of steps for performing the task using the task performance module (310) or artificial intelligence model (320) of FIG. 3.

[0139] 607 In operation, an electronic device according to one embodiment can acquire data related to the situation of a user performing an operation in real time through at least one microphone, at least one sensor and / or at least one camera. The electronic device can acquire data related to the user's current situation by monitoring the user's situation in real time.

[0140] 609 In operation, an electronic device according to one embodiment can identify a specific step corresponding to the user's situation among a plurality of steps based on data related to the user's current situation. The electronic device can identify a specific step corresponding to the user's situation by using an artificial intelligence model.

[0141] In operation 611, the electronic device may output a response to a user query that includes information about a task and information about a specific step. An electronic device according to one embodiment may obtain (e.g., generate) a response to a user query that includes information about a task and information about a specific step based on data related to the user situation obtained in real time using an artificial intelligence model. The electronic device may display the response obtained through the artificial intelligence model via a display (e.g., display module (160) of FIG. 1, display (230) of FIG. 2)) or output it in real time as audio and / or vibration via a speaker (e.g., sound output module (155) of FIG. 1 or speaker (270) of FIG. 2)) and / or a haptic module (e.g., haptic module (179) of FIG. 1). Here, the information about a specific step included in the response to the user query may include identification information of the specific step and information related to the performance of subtasks of the specific step. The information about a specific step may further include additional guidance information regarding subtasks of the specific step or information related to actions prohibited in the specific step.

[0142] FIG. 7 is a diagram illustrating an example of a method of operation in an electronic device according to one embodiment. FIG. 8 is a diagram illustrating an example of providing a response to a user query in an electronic device according to one embodiment. FIG. 9a and FIG. 9b are diagrams illustrating examples of providing a response to a user query in an electronic device according to one embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel. In the following embodiments, each operation may be understood as being performed in a processor (e.g., processor (120) of FIG. 1 or processor (210) of FIG. 2) of an electronic device (e.g., electronic device (101) of FIG. 1 or electronic device (201) of FIG. 2).

[0143] Referring to FIGS. 7, FIGS. 8, FIGS. 9a, and FIGS. 9b, an electronic device (201) according to one embodiment (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can verify a user query obtained through an input device in operation 701. The electronic device (201) according to one embodiment can obtain an image obtained through at least one camera (e.g., at least one camera included in the camera circuit (240) of FIG. 2) as a user query. The electronic device (201) according to one embodiment can obtain audio information including voice corresponding to a user's speech or a sound source generated at the location where the user is located through at least one microphone included in the microphone circuit (e.g., the microphone circuit (260) of FIG. 2) as a user query. An electronic device (201) according to one embodiment may obtain information detected by detecting a user's behavior through at least one sensor (e.g., at least one sensor included in the sensor circuit (280) of FIG. 2) and / or information detected from a device or object within the location where the user is located as a user query. For example, the at least one sensor may include an accelerometer, a gyro sensor, a GPS sensor, or various sensors embedded in the electronic device.

[0144] In operation 703, an electronic device (201) according to one embodiment can identify a task related to the user query by analyzing the user query through a task execution module (310). For example, if the user query includes an image, the electronic device (201) can classify the objects included in the image into people or objects through the task execution module (310), and analyze the classified objects to identify the task to be performed by the user (e.g., inference). For example, if the user query includes text information or voice information, the electronic device (201) can classify the text information or voice information into words through the task execution module (310), and analyze the classified words to identify the task to be performed by the user based on selected key words (e.g., inference). For example, the electronic device (201) can identify a user based on a user query through the task execution module (310), and can identify a task to be performed by the identified user based on context information stored in the electronic device's memory (e.g., memory (130) of FIG. 1 or memory (220) of FIG. 2) or a server in relation to the user query and the identified user (e.g., user's personal information, health information, medical information, activity information, biometric information, history information related to the task, information related to various applications executed on the electronic device, or other various information collected or managed using an artificial intelligence model). For example, as shown in FIG. 8, if the user query is an input such as "Tell me how to cook ramen step by step," the electronic device (201) can analyze the user query by the task execution module (310) and identify the task to be performed as "how to cook ramen" based on the analysis result.

[0145] In the above 703 operation, an electronic device (201) according to one embodiment can identify a plurality of steps for a task (e.g., "how to cook ramen") through a task execution module (310) and obtain information about the task, including subtasks (810) of the plurality of steps, through an artificial intelligence model (320). The information about the task may include information related to the execution of the identified task and information about the subtasks of the plurality of steps. Here, the information about the plurality of subtasks may include information composed of at least one of text, voice, or image for the process, method, or guidance for performing the task. The plurality of subtasks may refer to a process or method for performing the task in order based on the plurality of steps. The electronic device (201) according to one embodiment may obtain information related to the task history (e.g., history information on previously performed tasks and / or history information on the execution of subtasks prior to the current subtask) and / or information related to prohibitions regarding the task as additional information that can be used to obtain a response regarding the task. Here, multiple steps for performing a task may be classified according to, for example, the type of task, user information (e.g., age, gender, occupation, medical information, or other information related to the user). The electronic device may obtain information about the identified task and multiple steps for performing the task using the task execution module (310) and / or artificial intelligence model (320) of FIG. 3. The artificial intelligence model (320) (e.g., generative artificial intelligence model) may obtain (e.g., generate) subtasks (810) classified into steps in the order of performing a task (e.g., "how to cook ramen") based on information about the task and / or data related to the user situation obtained in real time. Here, the subtasks (810) may be classified into six steps, and may be classified into various steps depending on the type of the identified task.For example, the electronic device (201) can identify generated subtasks (810) classified into: Step 1, “Put water in a pot and boil it”; Step 2, “When the water boils, add the noodles”; Step 3, “Add the soup immediately after adding the noodles”; Step 4, “(Optional) Add an egg after 2 minutes”; Step 5, “(Optional) Add green onions and chili peppers”; and Step 6, “Boil for 3 to 4 minutes”.

[0146] In operation 705, an electronic device (201) according to one embodiment can acquire data related to the situation of a user performing an operation in real time through at least one microphone, at least one sensor and / or at least one camera. The electronic device (201) can acquire data related to the user's current situation by monitoring the user's situation in real time. The electronic device (201) can acquire (e.g., collect) data related to the user's situation in real time and store it in memory until an operation to provide a response to a user query is completed.

[0147] In operation 707, an electronic device (201) according to one embodiment can identify a specific step corresponding to the user's situation among a plurality of steps based on data related to the user's current situation. The electronic device (201) can identify a specific step (e.g., Step 2 (911)) corresponding to the user's situation using an artificial intelligence model (320). For example, if the result of analyzing an image acquired through at least one camera confirms that the water in the pot is boiling, or if the water is boiling through at least one microphone or at least one sensor, it can be confirmed that the user is currently performing a sub-task of Step 2 (911) among the sub-tasks. When the electronic device (201) confirms the step currently being performed as Step 2 (911) through the task execution module (310), it can request a response regarding the sub-task currently being performed from the artificial intelligence model (320). The artificial intelligence model (320) can obtain a response (first response) for a sub-task for the second step (911) based on data (901) related to the user situation and information about the second step (911) transmitted through the task execution module (310).

[0148] In operation 709, the electronic device (201) may output a response (920) (e.g., a first response) to a user query that includes information about a task and information about a specific step (e.g., step 2 (911)). According to one embodiment, the electronic device (201) may obtain (e.g., generate) a response to a user query that includes information about a task and information about a specific step (920) based on data related to the user situation obtained in real time using an artificial intelligence model (320).

[0149] In operation 711, an electronic device (201) according to one embodiment may display a response to a user query containing information about a task and information about a specific step (920) through a display (e.g., display module (160) of FIG. 1, display (230) of FIG. 2) or output audio and / or vibration through a speaker (e.g., sound output module (155) of FIG. 1 or speaker (270) of FIG. 2) and / or a haptic module (e.g., haptic module (179) of FIG. 1). Here, the information about a specific step (920) included in the response to the user query may include identification information of the specific step and information (921) related to the performance of a subtask of the specific step. The information about a specific step (920) may further include information (922) regarding the mandatory performance of a subtask of the specific step, information (923) regarding prohibition of performance in the specific step, or additional guidance information.

[0150] In operation 713, an electronic device (201) according to one embodiment can check whether the user's situation has changed to the next step based on data related to the user's situation acquired in real time. If, as a result of the check, it has changed to the next step, the electronic device (201) performs operation 715, and if it has not changed to the next step, the electronic device (201) can perform step 717.

[0151] In operation 715, an electronic device (201) according to one embodiment may obtain a guidance message for the next subtask using an artificial intelligence model based on information for the next subtask and input data obtained in real time, and may display the obtained guidance message through a display (e.g., display module (160) of FIG. 1, display (230) of FIG. 2) or output it as audio and / or vibration through a speaker (e.g., sound output module (155) of FIG. 1 or speaker (270) of FIG. 2) and / or a haptic module (e.g., haptic module (179) of FIG. 1).

[0152] In operation 717, an electronic device (201) according to one embodiment can check whether the execution of the operation has been completed. If, as a result of the check, the execution of all steps of the operation has been completed, the electronic device (201) terminates the operation to provide a response to a user query, and if not, can perform operation 705.

[0153] FIG. 10 is a diagram illustrating an example of providing a response to a user query in an electronic device according to one embodiment.

[0154] Referring to FIG. 10, according to one embodiment, when an electronic device (201) performs the 711 and 713 operations of FIG. 7 described above, it may acquire data related to the user's situation (e.g., image (1010)) acquired in real time at the current stage (e.g., stage 2) by means of a task execution module (310). The electronic device (201) may use an artificial intelligence model (320) to identify an additional response (1020) (e.g., a second response) containing additional information including a detailed description of the subtask of the current stage based on the data related to the user's situation (e.g., image (1010)) acquired in real time at the current stage (e.g., stage 2). The electronic device (201) can display the response (1020) confirmed by the task execution module (310) through the display (230) or output it as audio and / or vibration through the speaker (270) (e.g., the acoustic output module (155) of FIG. 1 or the speaker (270) of FIG. 2) and / or the haptic module (e.g., the haptic module (179) of FIG. 1).

[0155] FIG. 11 is a diagram illustrating an example of providing a response to a user query in an electronic device according to one embodiment.

[0156] Referring to FIG. 11, according to one embodiment, when the electronic device (201) performs the 711 and 713 operations of FIG. 7 described above, it can determine whether the user's situation is not consistent with the sub-task of a specific step (e.g., step 2 (911)) based on data (1110) (e.g., input data) related to the user's situation obtained in real time at a specific step (e.g., step 2 currently being performed) by the task execution module (310). If the electronic device (201) identifies that the user's current situation is a user action not related to the sub-task of a specific step or a pre-specified prohibited action (e.g., behavior) at the current step, it can determine that the user's current situation is not consistent with the sub-task of a specific step. For example, if the electronic device (201) detects that the user is performing a different action (e.g., attempting to add an egg) without performing the two-step subtask provided in response (e.g., "When the water boils, add the noodles"), it may output a guidance message (1120) for the other action (e.g., "You must add the beaten egg after adding and boiling the noodles. Please stop adding the beaten egg."). The electronic device (201) may display the guidance message (1120) (e.g., the first message) in real time through the display (230) or output it as audio in real time through the speaker (270).

[0157] According to one embodiment, the electronic device (201) checks the progress status (e.g., degree of progress or progress rate) of a subtask of a currently performing step, and if it identifies that the confirmed progress status is a situation where the current step is completed (e.g., a situation where a progress rate greater than a specified progress rate is confirmed), it obtains a guidance message (e.g., a second message) to guide the subtask of the next step based on information about the task, and can provide the guidance message in advance (e.g., output through a display or speaker) before the subtask of the currently performing step is completed. For example, if a time limit (e.g., time limit) is specified for a plurality of steps, the electronic device (201) can check the progress status (e.g., elapsed time) of the subtask using a timer, and if the time limit is exceeded, it can provide a guidance message.

[0158] According to one embodiment, the electronic device (201) may provide a message (e.g., a third message) guiding the user to perform an essential task in the current stage if the user has not completed an essential task (an action that the user is required to perform) in the current stage when the user's situation changes to the next stage. The electronic device (201) may determine that the user has not performed an essential action in the current stage based on information regarding the essential task included in the response to the user's query and data related to the user's situation.

[0159] According to one embodiment, the electronic device (201) acquires additional information including information for guiding the sub-task of the next step and information regarding the detailed description of the next step using an artificial intelligence model, and can output the acquired additional information before the sub-task of the step being performed is completed. Here, the additional information can be provided (e.g., output via a display or speaker) according to the progress status of the sub-task of the step being performed, even if no additional query from the user is entered.

[0160] According to one embodiment, when the electronic device (201) provides a response to a user query, it may obtain information related to a specific person (e.g., a chef or a mother) in relation to the task, and provide a response to the user query based on the obtained information related to the specific person. For example, if the task to be performed is identified as “Mom’s recipe for making soybean paste stew,” the electronic device (201) may obtain information about “Mom’s recipe” from a memory (e.g., memory (130) of FIG. 1 or memory (220) of FIG. 2)) or an external electronic device, and based on the information about “Mom’s recipe,” verify the response to the user query obtained by the artificial intelligence model (320) and provide the verified response.

[0161] According to one embodiment, the electronic device (201) can provide a response to a user query using history information previously performed by the user in relation to a task, history information of another user in relation to a task from an external electronic device, and information obtained from an executable application in relation to performing a task.

[0162] According to another embodiment in this document, an electronic device (e.g., the electronic device of FIG. 1 and the electronic device (201) of FIG. 2) may receive guidance to a specific location as a user query or command, and accordingly, may identify "guidance to a destination" as a task related to the user query. For example, if a user utters a user command such as "tell me the way to Seoul City Hall" through an input device, the electronic device may identify the task as "guidance to Seoul City Hall" as an input related to the task. Alternatively, as previously mentioned, user input may be triggered not only by a format such as a "query" or command based on user voice input, but also by other formats of input such as an image, so that the electronic device first queries the user and receives a response from the user to start. For example, if a user carries or wears the electronic device (201) and boards a vehicle that already has information about the electronic device (201), the electronic device (201) becomes aware of it through short-range communication with the vehicle, or becomes aware that the user has boarded the vehicle through image information input via an image input device (e.g., camera) mounted on the electronic device (201), the electronic device (201) may recognize this as a trigger input and first query the user for a destination.

[0163] Additionally, the electronic device may output a message to ask a user a question using an artificial intelligence model (e.g., the artificial intelligence model (320) of FIG. 3), and based on data related to the user's situation, if it is determined that the verified information is inaccurate or that additional information is needed, it may query the user to obtain accurate information or additional information. For example, if the user utters a user command such as "Tell me the way to Seoul City Hall," the electronic device may ask "Is the user's mode of travel by car or on foot?", obtain an answer from the user, and output more appropriate information for the task to the user based on the obtained answer.

[0164] The electronic device can output information regarding multiple sub-tasks to the user as a type of information about the task. For example, it can output information regarding multiple sub-tasks to the user, such as going straight for 500 meters (Step 1; Sub-task 1), turning right at an intersection (Step 2; Sub-task 2), and then going straight for another 300 meters (Step 3; Sub-task 3). In addition, information regarding the user's current location and the direction of movement can be acquired in real time through a GPS sensor and a video input device (e.g., a camera) mounted on the electronic device. This information can be utilized as data related to the situation of the user performing the task.

[0165] In addition, the electronic device can output additional information through an output device that the speed limit on the road where the sub-operation of the first stage is being carried out is 60 km / h, and if it is determined that the current user's movement speed is 80 km / h based on data regarding the user's situation obtained in real time, it can output a first message through an output device in real time to indicate that the prohibition on the operation is not being met.

[0166] Additionally, the electronic device identifies which of the multiple steps corresponds to the user's current situation based on data related to the user's situation acquired in real time, and provides the user with information related to the execution of the task and / or information regarding multiple sub-tasks through an output device. For example, when sub-task 1 of the first step is almost completed, a second message may be output through the output device to provide information regarding sub-task 2 of the second step (turn right at the intersection shortly).

[0167] Additionally, based on data related to the acquired user's situation, the electronic device identifies a situation in which the first step among multiple steps changes to the second step, checks whether the user has performed a right turn (essential task) at an intersection, and if it identifies that a right turn at an intersection has not been performed, it can output a third message guiding the performance of a right turn through an output device.

[0168] Additionally, if necessary, the electronic device may modify or newly establish the subtasks and steps previously verified and output to the user, and provide new guidance to the user through the output device. For example, if the user drives along a different path without performing the right turn required in the subtask of the second step, the electronic device may output a fourth message through the output device that provides new guidance to the user through the output device regarding new subtasks and steps for driving along a newly established path, rather than the previously guided subtasks and steps, based on data related to the user's new situation acquired in real time (e.g., the user's real-time current location).

[0169] FIG. 12 is a perspective view showing the structure of an electronic device according to one embodiment.

[0170] Referring to FIG. 12, an electronic device (1200) according to one embodiment may be an electronic device (101) of FIG. 1, an electronic device (201) of FIG. 2, an electronic device (102 or 104) communicating with the electronic device (101) of FIG. 1, or a device capable of providing services related to virtual reality technology that provides a virtual environment similar to the electronic device (101) of FIG. 1. Virtual reality (VR) technology, which is a technology that provides a virtual environment, may be developed into augmented reality (AR), mixed reality (MR), and / or extended reality (XR) that encompasses these.

[0171] The electronic device (1200) may be a device configured to be worn on a user's body, as illustrated in FIG. 12 (e.g., a head-mounted display (HMD) or a glasses-type AR glasses device). For example, the electronic device (1200) may be configured to combine with an external electronic device, such as a mobile device, and may utilize components of the external electronic device (e.g., a display module, a camera module, an audio output module, or other components). Not limited thereto, the electronic device (200) may be implemented in various forms that can be worn on a user's body.

[0172] According to one embodiment, the electronic device (1200) may configure a real space (e.g., virtual reality space, augmented reality space, or mixed reality space) that displays a real space corresponding to an actual external environment captured in the surrounding environment where the user is located (e.g., augmented reality image) or a virtual image provided (e.g., 2D or 3D image), and may control a display module to display at least one virtual object corresponding to the user and / or at least one virtual object corresponding to an object for user interaction in the real space.

[0173] According to one embodiment, the electronic device (1200) may include a processor (120) shown in FIG. 1, memory (130), display module (160), sensor module (176), camera module (180), charging module (e.g., battery (189) of FIG. 1), and communication module (190). The electronic device (1200) may further include an acoustic output device (155) shown in FIG. 1, an input module (150), or other components shown in FIG. 1. In addition, the electronic device (1200) may be configured to include other components necessary to provide virtual reality functions, augmented reality functions, or mixed reality functions (e.g., services or methods).

[0174] According to one embodiment, the processor (120) is electrically connected to other components and can control other components. The processor (120) can perform various data processing or operations in accordance with the execution of various functions (e.g., operations, services, or programs) provided by the electronic device (1200). The processor (120) can perform various data processing or operations to display at least one virtual object related to real objects included in an image captured in real space and / or a virtual object corresponding to a user (e.g., an avatar) in a virtual reality space. The processor (120) can perform various data processing or operations to express user interaction or movement of the virtual object displayed in the virtual reality space.

[0175] Again, with reference to FIG. 12, an electronic device (1200) according to one embodiment will be described. As described above, the electronic device (1200) is not limited to a glasses-type (e.g., AR glasses) augmented reality device, and can be implemented as various devices capable of providing immersive content (e.g., content based on XR technology) to the user's eyes (e.g., AR head-mounted display type, 2D / 3D head-mounted display device or VR head-mounted display device).

[0176] According to one embodiment, a camera module of an electronic device (1200) (e.g., camera module (180) of FIG. 1 or camera circuit) can capture still images and / or video. According to one embodiment, the camera module may be placed within a lens frame and around a first display (1251) and a second display (1252). According to one embodiment, the camera module may include one or more first cameras (1211-1, 1211-2), one or more second cameras (1212-1, 1212-2), and one or more third cameras (1213). According to one embodiment, images acquired through one or more first cameras (1211-1, 1211-2) may be used for detecting hand gestures by a user, tracking the user's head, and / or spatial recognition. One or more first cameras (1211-1, 1211-2) may be GS (global shutter) cameras or RS (rolling shutter) cameras. One or more first cameras (1211-1, 1211-2) can perform simultaneous localization and mapping (SLAM) operations through depth imaging. One or more first cameras (1211-1, 1211-2) can perform spatial recognition and / or motion recognition for 3DoF (depth of field) and / or 6DoF. According to one embodiment, the first cameras (211-1, 211-2) can periodically or non-periodically transmit information (e.g., trajectory information) related to the user's eyes (e.g., left eye and right eye) or the trajectory of the gaze (e.g., eye tracking) to a processor (e.g., processor (120) of FIG. 1).The first camera (1211-1, 1211-2) can be used to position the center of a virtual image projected onto an electronic device (1200) (e.g., AR glasses) according to the direction in which the user's pupil gazes, and a GS camera can be primarily used to detect the pupil and track rapid pupil movements. The first camera (1211-1, 1211-2) can be configured for the left eye and the right eye, respectively, and the first camera (211-1, 211-2) configured for the left eye and the right eye, respectively, may have the same performance and specifications.

[0177] According to one embodiment, the electronic device (200) may use another camera (e.g., a third camera (1213)) for hand detection and tracking and user gesture recognition. According to one embodiment, at least one of the first camera (1211-1, 1211-2) to the third camera (1213) may be replaced with a sensor module (e.g., a LiDAR sensor). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0178] According to one embodiment, images acquired through one or more second cameras (1212-1, 1212-2) may be used to detect and track the user's pupils. One or more second cameras (1212-1, 1212-2) may be GS cameras. One or more second cameras (1212-1, 1212-2) may correspond to the left eye and the right eye, respectively, and the performance of one or more second cameras (1212-1, 1212-2) may be substantially identical. One or more third cameras (1213) may be cameras with relatively high resolution. One or more third cameras (1213) may perform auto-focusing (AF) and optical image stabilization (OIS) functions. One or more third cameras (1213) may be GS (global shutter) cameras or RS (rolling shutter) cameras. One or more third cameras (1213) may be color cameras.

[0179] According to one embodiment, the electronic device (1200) may include one or more light-emitting elements (1214-1, 1214-2). The light-emitting elements (1214-1, 1214-2) are different from the light source described below, which irradiates light onto a screen output area of ​​a display. According to one embodiment, the light-emitting elements (1214-1, 1214-2) may irradiate light to facilitate pupil detection in detecting and tracking a user's pupil through one or more second cameras (1212-1, 1212-2). According to one embodiment, the light-emitting elements (1214-1, 1214-2) may each include an LED. According to one embodiment, the light-emitting elements (1214-1, 1214-2) may irradiate light in the infrared region. According to various embodiments, light-emitting elements (1214-1, 1214-2) may be attached around the frame of the augmented reality device (1200). According to one embodiment, light-emitting elements (1214-1, 1214-2) may be positioned around one or more first cameras (1211-1, 1211-2) and may assist gesture detection, head tracking, and spatial recognition by one or more first cameras (1211-1, 1211-2) when the augmented reality device (1200) is used in a dark environment. According to one embodiment, light-emitting elements (1214-1, 1214-2) may be positioned around one or more third cameras (1213) and may assist image acquisition by one or more third cameras (1213) when the augmented reality device (1200) is used in a dark environment.

[0180] According to one embodiment, the electronic device (1200) may include a battery (1235-1, 1235-2) (e.g., battery (189) of FIG. 1). The battery (1235-1, 1235-2) may store power to operate the remaining components of the augmented reality device (1200).

[0181] According to one embodiment, a display module of an electronic device (1200) (e.g., a display module (160) of FIG. 1) may include a first display (1251), a second display (1252), one or more input optical members (1253-1, 1253-2), one or more transparent members (1290-1, 1290-2), and one or more screen display portions (1254-1, 1254-2). According to one embodiment, the first display (1251) and the second display (1252) may be light output modules and may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). According to one embodiment, if the first display (1251) and the second display (1252) are composed of a liquid crystal display device, a digital mirror display device, or a silicon liquid crystal display device, the augmented reality device (1200) may include a light source that irradiates light onto a screen output area of ​​the display. According to one embodiment, if the first display (1251) and the second display (1252) can generate light themselves, for example, if they are composed of an organic light-emitting diode or a micro LED, the augmented reality device (1200) may provide a user with a good quality virtual image (e.g., an image of a virtual reality space) without including a separate light source.

[0182] According to one embodiment, one or more transparent members (1290-1, 1290-2) included in the electronic device (1200) may be positioned to face the user's eyes (e.g., left and right eyes) when the user wears the augmented reality device (1200). The one or more transparent members (1290-1, 1290-2) may include at least one of a glass plate, a plastic plate, or a polymer. When the user wears the augmented reality device (e.g., the electronic device (1200)), the user can view the external environment through the one or more transparent members (1290-1, 1290-2).

[0183] According to one embodiment, one or more input optical members (1253-1, 1253-2) included in the electronic device (1200) can guide light generated from a first display (1251) and a second display (1252) to the user's eye. An image based on the light generated from the first display (1251) and the second display (1252) is formed on one or more screen display portions (1254-1, 1254-2) on one or more transparent members (1290-1, 1290-2), and the user can see the image formed on the one or more screen display portions (1254-1, 1254-2).

[0184] According to one embodiment, the electronic device (1200) may include one or more optical waveguides (not shown). The optical waveguides may transmit light generated from the first display (1251) and the second display (1252) to the user's eye. The electronic device (1200) may include one optical waveguide corresponding to each of the left eye and the right eye. According to one embodiment, the optical waveguides may include at least one of glass, plastic, or polymer. The optical waveguides may include a nano-pattern formed on an inner or outer surface, for example, a polygonal or curved grating structure. The optical waveguides may include a free-form prism, in which case the optical waveguides may provide incident light to the user through a reflective mirror. According to one embodiment, the optical waveguide includes at least one of a diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or a reflective element (e.g., a reflective mirror), and can guide display light emitted from a light source to the user's eye using at least one diffractive element or reflective element included in the optical waveguide. According to one embodiment, the diffractive element may include an input / output optical member. According to one embodiment, the reflective element may include a member that causes total internal reflection (TIR) ​​(e.g., a total internal reflection optical element or a total internal reflection waveguide). For example, total internal reflection is a method of guiding light, which may mean creating an angle of incidence such that light (e.g., a virtual image) input through an input grating area is 100% reflected from one surface (e.g., a specific surface) of the waveguide and is transmitted 100% to an output grating area.

[0185] In one embodiment, light emitted from a display (e.g., a first display (1251) and a second display (1252)) may have its light path guided to a waveguide through an input optical member (e.g., an optical waveguide). Light traveling inside the waveguide may be guided toward the user's eye through an output optical member. A screen display may be determined based on the light emitted toward the eye.

[0186] According to one embodiment, the electronic device (1200) may include one or more voice input devices (1262-1, 1262-2, 1262-3) and one or more voice output devices (1263-1, 1263-2).

[0187] According to one embodiment, the electronic device (1200) may include a first PCB (1270-1) and a second PCB (1270-2). The first PCB (1270-1) and the second PCB (1270-2) may transmit electrical signals to components included in the electronic device (1200), such as a first camera (1211-1, 1211-2), a second camera (2112-1, 1212-2), a third camera (1213), a display (1251, 1252), an audio module (e.g., the audio module (170) of FIG. 1), and a sensor module (e.g., the sensor module (176) of FIG. 1). According to one embodiment, the first PCB (1270-1) and the second PCB (1270-2) may be flexible printed circuit boards (FPCB). According to one embodiment, the first PCB (1270-1) and the second PCB (1270-2) may each include a first substrate, a second substrate, and an interposer disposed between the first substrate and the second substrate.

[0188] FIG. 13a is a perspective view showing the structure of an electronic device according to one embodiment.

[0189] Referring to FIG. 13a, an electronic device (300) according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, or the electronic device (1200) of FIG. 12) may be a wearable device such as a head-mounted device (HMD) that can be worn on a user's head to provide an image (e.g., a virtual reality space image) in front of the eyes. The configuration of the electronic device (1300) of FIG. 3 may be all or partly the same as the configuration of the electronic device (1200) of FIG. 12.

[0190] According to one embodiment, the electronic device (1300) may include a housing (1310, 1320, 1330) that can form an exterior and provide a space in which components of the electronic device (1300) can be placed.

[0191] According to one embodiment, the electronic device (1300) may include a first housing (1310) that can surround at least a portion of the user's head. According to one embodiment, the first housing (1310) may include a first surface (1300a) facing the outside of the electronic device (1300) (e.g., in the +X direction).

[0192] According to one embodiment, the first housing (1310) may surround at least a portion of the internal space (I). For example, the first housing (1310) may include a second surface (1300b) facing the internal space (I) of the electronic device (1300) and a third surface (1300c) opposite to the second surface (1300b). According to one embodiment, the first housing (1310) may be combined with a third housing (1330) to form a closed curve shape surrounding the internal space (I).

[0193] According to one embodiment, the first housing (1310) may accommodate at least some of the components of the electronic device (1300). For example, a light output module, a circuit board, and a speaker module may be placed within the first housing (1310).

[0194] According to one embodiment, a display member (1340) corresponding to the left and right eyes of the electronic device (1300) may be included. The display member (340) may be disposed in a first housing (1310). The configuration of the display member (1340) of FIG. 13 may be wholly or partially identical to the configuration of the screen display portion (1254-1, 1254-2) of FIG. 12.

[0195] According to one embodiment, the electronic device (1300) may include a second housing (1320) that can be placed on the face of a user. According to one embodiment, the second housing (1320) may include a fourth surface (1300d) that can face at least partially the face of a user. According to one embodiment, the fourth surface (1300d) may be a surface facing the internal space (I) of the electronic device (1300) (e.g., -X direction). According to one embodiment, the second housing (1320) may be combined with the first housing (1310).

[0196] According to one embodiment, the electronic device (1300) may include a third housing (1330) that can be seated on the back of the user's head. According to one embodiment, the third housing (1330) may be combined with the first housing (1310). According to one embodiment, the third housing (1330) may accommodate at least some of the components of the electronic device (1300). For example, a battery (e.g., the battery (1235-1, 1235-2) of FIG. 2) may be placed within the third housing (1330).

[0197] In order to enhance the user's overall user experience, usage environment, and usability of a head-mounted wearable electronic device (1300), it may be necessary for the sensations felt and experienced by the user in VR (virtual reality), AR (augmented reality), and MR (mixed reality) spaces to be as similar as possible to the sensations of the real world.

[0198] FIGS. 13b and FIGS. 13c are perspective views showing the structure of an electronic device according to one embodiment.

[0199] Referring to FIG. 13b and FIG. 13c, in one embodiment, camera modules (1311, 1312, 1313, 1314, 1315, 1316) and / or a depth sensor (1317) for acquiring information related to the surrounding environment of an electronic device (1300) (e.g., a wearable device) may be disposed on a first surface (1310) of the housing.

[0200] In one embodiment, camera modules (1311, 1312) can acquire images related to the surrounding environment of a wearable electronic device.

[0201] In one embodiment, camera modules (1313, 1314, 1315, 1316) can acquire images while the electronic device (1300) is worn by a user. The camera modules (1313, 1314, 1315, 1316) can be used for hand detection, tracking, and user gesture (e.g., hand movements) recognition. The camera modules (1313, 1314, 1315, 1316) can be used for 3DoF, 6DoF head tracking, position (space, environment) recognition, and / or movement recognition. In one embodiment, camera modules (1311, 1312) may be used for hand detection and tracking and user gestures.

[0202] In one embodiment, the depth sensor (1317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for determining the distance to an object, such as time of flight (TOF). In place of or additionally to the depth sensor (1317), camera modules (1313, 1314, 1315, 1316) may determine the distance to an object.

[0203] According to one embodiment, a face recognition camera module (1325, 1326) (e.g., FT (Face Tracking) camera) and / or a display (1321) (and / or a lens) may be disposed on the second surface (1320) of the housing.

[0204] In one embodiment, a face recognition camera module (1325, 1326) adjacent to the display may be used to recognize the user's face or to recognize and / or track both of the user's eyes. In one embodiment, the lens may serve to adjust the focus so that the screen output to the display (1321) can be seen by the user's eyes, and may be composed of, for example, a Fresnel lens, a Pancake lens, or a multi-channel lens.

[0205] In one embodiment, the display (1321) (and / or lens) may be disposed on a second surface (1320) of the wearable electronic device (1300). In one embodiment, the wearable electronic device (1300) may not include camera modules (315, 316) among a plurality of camera modules (1313, 1314, 1315, 1316). Although not illustrated in FIG. 13b and FIG. 13c, the electronic device (1300) may further include at least one of the configurations illustrated in FIG. 12.

[0206] As described above, according to one embodiment, the electronic device (1300) may have a form factor for being worn on a user's head. The electronic device (1300) may further include a strap and / or a wearing member for being secured on a part of the user's body. The electronic device (1300) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0207] Hereinafter, the electronic device described in this disclosure may be an electronic device that a user can wear on their body (e.g., head), such as a head-mounted display (HMD) device, augmented reality (AR) glasses, and / or a VST device, as described with reference to FIGS. 12, 13a through 13c. Herein, the electronic device may be referred to as a wearable electronic device. The interpretation described in this disclosure may be referred to as “translation” and may be replaced by other terms having the meaning of transferring a language other than the user’s language into the user’s language. The subject of interpretation described in this disclosure may be a person who speaks a language different from the user or a device that outputs a different language. The interpretation described in this disclosure may mean transferring audio information, including the voice of a person who speaks a different language or sound output from a device that outputs a different language, into the form of audio or text in the language used by the user. In this disclosure, the audio information may include sign language as the other language of the subject of interpretation.

[0208] FIG. 14 is a diagram illustrating a generative artificial intelligence system according to one embodiment.

[0209] Referring to FIG. 14, in a generative artificial intelligence system (1400) according to one embodiment, a user query / response interface (1410) (e.g., an input module (150) of FIG. 1 or a microphone circuit (260) of FIG. 2, and / or a display module (160), a display (230) of FIG. 2, a first display (1251) of FIG. 12, a second display (1252), a display member (1340) of FIG. 13a, or a display (1321) of FIG. 13c)) may receive user input. The user input may be in the form of natural language, images and / or videos, but is not limited thereto. Additionally, context information may be transmitted along with the user input. The context information may include various additional information at the time of user input. For example, the additional information may include information about the application currently being used by the user or the user's location information. Additionally, user input may be in a mixed form of the aforementioned natural language, images, sounds, and context information. Additionally, user input may be in a non-natural language form, such as selecting a menu. The user query / response interface (1410) can output results from the generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in a form such as an action requested by the user. The user query / response interface (1410) can output results from the generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in a form such as an action requested by the user.

[0210] An artificial intelligence framework (1440) (e.g., the processor (120) of FIG. 1 or the processor (210) of FIG. 2) can receive input from a user and coordinate and control each component necessary to perform the user's intent based on the user's query.

[0211] User input received from the user query / response interface (1410) can be transmitted to a prompt design component (1441) (e.g., the processor (120) of FIG. 1 or the processor (210) of FIG. 2). The prompt design component (1441) can be used to generate prompts suitable for inputting user input into a large language model (LLM), a large vision model (LVM), or a large multimodal model (LMM). The prompt design component (1441) may be an artificial intelligence component that uses machine learning algorithms or neural networks to develop better prompts over time. The prompt design component (1441) can generate prompts by accessing a knowledge component containing user preference data, a prompt library, and prompt examples based on user input, and can transmit the generated prompts to the LLM, LVM, or LMM.

[0212] An API / Plug-in management component (1442) (e.g., the processor (120) of FIG. 1 or the processor (210) of FIG. 2)) can perform the role of communicating with external information when there is a request for additional information when user input is passed as input to a generative model (e.g., the generative artificial intelligence model (320) of FIG. 3 or a cloud artificial intelligence (AI) model). The API / Plug-in management component (1442) establishes a channel to communicate with the outside of the artificial intelligence framework (1440) via an API, and through the established channel, it can access various data sources (e.g., a knowledge store (1420)) (e.g., the memory (130) of FIG. 1 or the memory (220) of FIG. 2). Additionally, the API / plugin management component (1442) may request the application / service component (1430) (e.g., the processor (120) of FIG. 1 or the processor (210) of FIG. 2) via the API when the application or service needs to perform an action that ultimately performs user input rather than an intermediate result. Information obtained from the outside may be used to generate a prompt in the prompt design component (1441) along with user input, or it may be passed as input to a generative artificial intelligence model (1460) (e.g., the generative artificial intelligence model (320) of FIG. 3 or a cloud artificial intelligence model).

[0213] An output modification component (or refiner component) (1443) (e.g., the processor (120) of FIG. 1 or the processor (210) of FIG. 2) can finely tune the output of a generative artificial intelligence model (1460) (e.g., the generative artificial intelligence model (320) of FIG. 3 or a cloud artificial intelligence model). For example, the output modification component (1443) can verify whether the content generated through LLM, LVM, and / or LMM is irrelevant, contains biased content, or contains harmful content. Additionally, the output modification component (1443) can determine the extent to which the output matches the desired result and, if additional processing is required, proceed with that process. Furthermore, the output modification component (1443) can configure and provide hints to the user to avoid unwanted output.

[0214] A generative AI model (1460) (e.g., the AI ​​model (320) of FIG. 3 or a cloud AI model) can generally refer to an AI neural network that generates new forms of data based on user queries. A generative AI model (1460) may include a model that generates images and / or a model that generates language. Models that generate images include, but are not limited to, GANs (generative adversarial networks) and VAEs (variational autoencoders), and examples include diffusion-based generative models that use VAEs and transformer structures. Models that generate language are models trained to output the most statistically appropriate output value based on input values, and examples include models such as CHAT-GPT 3 and CHAT-GPT 4. Additionally, there are LMMs (large multimodal models) that can recognize various forms of data input, such as text, images, and voice, and generate new data corresponding to them.

[0215] In one embodiment, the artificial intelligence framework (1440) and / or generative artificial intelligence model (1460) may be included within an artificial intelligence module (e.g., including a processing circuit) within the electronic device. For example, the artificial intelligence module may be operatively coupled with at least one processor of the electronic device (e.g., at least one processor (120) of FIG. 1 or processor (210) of FIG. 2). For example, the artificial intelligence module may be operatively coupled with a sensor hub of the electronic device for one or more sensors within the electronic device.

[0216] According to one embodiment, a method of operation in an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, or the electronic device (1200, 1300) of FIG. 12) may include receiving a user query through an input device of the electronic device.

[0217] According to one embodiment, the method may include an operation of identifying an operation related to the user query based on receiving the user query.

[0218] According to one embodiment, the method may include an operation of confirming information about the operation and a plurality of steps for performing the operation.

[0219] According to one embodiment, the method may include an operation of acquiring data related to the situation of a user performing the task through the input device in real time.

[0220] According to one embodiment, the method may include an operation of identifying a specific step corresponding to the user's situation among the plurality of steps based on the data.

[0221] According to one embodiment, the method may include an operation of outputting information about the operation and information about the specific step as a response to the user query through an output device of the electronic device.

[0222] According to one embodiment, information regarding the task includes information related to the performance of the identified task and information regarding a plurality of subtasks corresponding to the plurality of steps, and information regarding the specific step includes information related to the performance of the specific step among the plurality of subtasks corresponding to the plurality of steps, and the response to the user query can be obtained through an artificial intelligence model.

[0223] According to one embodiment, the method may further include the operation of obtaining additional information including a detailed description of a sub-task of the specific step through an artificial intelligence model based on data related to the user's situation, and the operation of outputting the additional information through the output device.

[0224] According to one embodiment, the method may further include the operation of obtaining a first message for guiding that the current situation does not correspond to the information regarding the specific step, based on identifying that the user's current situation does not correspond to the information regarding the specific step based on the data, and the operation of outputting the first message through the output device.

[0225] According to one embodiment, the method may further include an operation to check the progress status of a sub-task of a specific step based on the data, an operation to obtain a second message to provide information about a sub-task of a next step in sequence before the sub-task of the specific step is completed based on the progress status, and an operation to output the second message through the output device.

[0226] According to one embodiment, the method may further include an operation to check whether an essential task has been completed at the specific step based on identifying a situation in which the specific step is changed to the next step in sequence based on the data, an operation to obtain a third message guiding the performance of the essential task at the specific step based on identifying that the essential task has not been completed at the specific step, and an operation to output the third message through the output device.

[0227] According to one embodiment, the method may further include, based on identifying that the task is related to a specific person based on the identified task and the data, an operation of obtaining information related to the specific person and an operation of outputting information related to the execution of the task obtained based on the information related to the specific person and the data as a response to the user query.

[0228] According to one embodiment, the method may further include the operation of outputting through the output device an additional response including information that induces the artificial intelligence model to perform the task and information about the task, based on identifying that the task is not performed immediately based on the data.

[0229] According to one embodiment, a method of operation in an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, or the electronic device (1200, 1300) of FIG. 12) may include: receiving user input through an input device; determining whether the user input is a trigger input for performing a task; if the user input is determined to be a trigger input for performing the task, receiving information about the task from memory or from outside the electronic device; receiving data related to the situation of the user performing the task in real time through the input device; and outputting a guide regarding the performance of the task through an output device based at least on the information about the task and the data related to the situation of the user.

[0230] According to one embodiment, in a non-transient computer-readable storage medium storing one or more programs, the one or more programs may include an executable instruction to cause the electronic device to perform an operation of receiving a user query through an input device of the electronic device when executed by at least one processor (e.g., processor (120) of FIG. 1 or processor (210) of FIG. 2) of an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2 or electronic device (1200, 1300)) of FIG. 12.

[0231] According to one embodiment, the one or more programs may include an executable instruction that, when executed by at least one processor of an electronic device, causes the electronic device to execute an operation that identifies a task related to the user query based on receiving the user query.

[0232] According to one embodiment, the one or more programs may include an executable instruction that causes the electronic device to perform an operation of verifying information about the task and a plurality of steps for performing the task when executed by at least one processor of the electronic device.

[0233] According to one embodiment, the one or more programs may include an executable command that, when executed by at least one processor of an electronic device, causes the electronic device to execute an operation of acquiring data related to the situation of a user performing the task in real time through the input device.

[0234] According to one embodiment, the one or more programs may include an executable instruction that, when executed by at least one processor of an electronic device, causes the electronic device to execute an operation of identifying a specific step among the plurality of steps corresponding to the user's situation based on the data.

[0235] According to one embodiment, the one or more programs may include an executable command that, when executed by at least one processor of an electronic device, causes the electronic device to perform an operation of outputting information about the operation and information about the specific step as a response to the user query through an output device of the electronic device.

[0236] According to one embodiment, the response to the task includes information related to the performance of the identified task and information regarding a plurality of subtasks corresponding to the plurality of steps, and the information regarding the specific step includes information related to the performance of the specific step among the plurality of subtasks corresponding to the plurality of steps, and the response to the user query can be obtained through an artificial intelligence model.

[0237] According to one embodiment, the one or more programs may include executable commands that, when executed by at least one processor of an electronic device, cause the electronic device to execute an operation of obtaining additional information including a detailed description of a sub-task of a specific step through an artificial intelligence model based on data related to the user's situation, and an operation of outputting said additional information through the output device.

[0238] According to one embodiment, the one or more programs may include executable instructions that, when executed by at least one processor of an electronic device, cause the electronic device to execute the operation of obtaining a first message for guiding that the current situation does not correspond to information regarding the specific step, based on identifying that the current situation of a user at the specific step does not correspond to information regarding the specific step based on the data, and the operation of outputting the first message through the output device.

[0239] According to one embodiment, the one or more programs may include executable commands that, when executed by at least one processor of an electronic device, cause the electronic device to perform the following operations: receiving user input through an input device; determining whether the user input is a trigger input for performing the task; receiving information about the task from memory or from outside the electronic device when it is determined that the user input is a trigger input for performing the task; receiving data related to the situation of the user performing the task in real time through the input device; and outputting a guide regarding the performance of the task through the output device based at least on the information about the task and the data related to the situation of the user.

[0240] This document can verify the task regarding the user's query, monitor the user's situation in real time to check the status of the execution of the verified task, and provide a response to the user's query based on the data verified in real time. This document can provide a response to the user's query tailored to the user's situation, and by providing a response appropriate to the user's situation in real time, it enables the user to perform tasks smoothly and reduces errors in task execution by providing guidance messages regarding mistakes. In addition, various effects that can be identified directly or indirectly through this document may be provided. The effects obtainable from this disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which this disclosure belongs from the description below.

[0241] Furthermore, the embodiments disclosed in this document are presented for the purpose of explaining and understanding the disclosed technical content and are not intended to limit the scope of the technology described in this document. Accordingly, the scope of this document should be interpreted to include all modifications or various other embodiments based on the technical concept of this document.

[0242] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0243] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0244] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0245] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0246] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0247] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101, 201), An input device comprising at least one of a sensor, a microphone, or a camera; An output device comprising at least one of a speaker (270) or a display (160, 230); At least one processor (120, 210) including a processing circuit; and It includes memory (130, 220) for storing instructions, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Receiving a user query through the above input device, and Based on receiving the above user query, identify the operation related to the above user query, and Identify information regarding the above task and a plurality of steps for performing the above task, and Data related to the situation of the user performing the above task is acquired in real time through the above input device, and Based on the above data, identify a specific step among the plurality of steps that corresponds to the user's situation, and An electronic device that causes information about the task and information about the specific step to be output as a response to the user query through the output device.

2. In Paragraph 1, Information regarding the above task includes information related to the performance of the identified task and information regarding a plurality of sub-tasks corresponding to the plurality of steps, and The information regarding the specific step includes information related to the performance of the specific step among the plurality of sub-tasks corresponding to the plurality of steps, and An electronic device that obtains the response to the above user query through an artificial intelligence model (320).

3. In claim 1 or 2, when the instructions are executed individually or collectively by the at least one processor, the electronic device: Based on data related to the situation of the user, additional information including a detailed description of the sub-task of the specific step is obtained through the artificial intelligence model, and An electronic device that causes the above additional information to be output through the above output device.

4. In any one of claims 1 to 3, when the instructions are executed individually or collectively by the at least one processor, the electronic device: Based on the above data, identifying that the user's current situation at the above specific step does not correspond to the information regarding the above specific step, and obtaining a first message to provide guidance that the current situation does not correspond to the information regarding the above specific step, An electronic device that causes the above first message to be output through the output device.

5. In any one of claims 1 to 4, when the instructions are executed individually or collectively by the at least one processor, the electronic device: Based on the above data, check the progress status of the sub-tasks of the above specific step, and Based on the above progress status, a second message is obtained to provide information about the sub-task of the next step in sequence before the sub-task of the specific step is completed, and An electronic device that causes the above second message to be output through the output device.

6. In any one of claims 1 to 5, when the instructions are executed individually or collectively by the at least one processor, the electronic device: Based on identifying, based on the above data, that the above specific step is changed to the next step in the sequence, checking whether the essential task to be performed at the above specific step has been completed, and Based on identifying that the essential task is not completed at the specific step mentioned above, a third message is obtained that guides the performance of the essential task at the specific step mentioned above, and An electronic device that causes the above third message to be output through the output device.

7. In any one of claims 1 through 6, when the instructions are executed individually or collectively by the at least one processor, the electronic device: Based on identifying that the task is related to a specific person based on the identified task and the data above, information related to the specific person is obtained, and An electronic device that outputs information related to the aforementioned specific person and information related to the performance of a task obtained based on the aforementioned data as a response to the aforementioned user query.

8. In any one of claims 1 through 7, when the instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes an additional response, including information for inducing the artificial intelligence model to perform the task and information about the task, to be output through the output device based on identifying that the task is not performed immediately based on the data above.

9. In any one of claims 1 through 8, when the instructions are executed individually or collectively by the at least one processor, the electronic device: An electronic device that causes information about the above-mentioned task and information about the above-mentioned specific step to be output in real time.

10. In any one of claims 1 through 9, when the instructions are executed individually or collectively by the at least one processor, the electronic device: Based at least part of the data related to the user's new situation received in real time, a fourth message regarding the performance of a new task is obtained, and An electronic device that causes the above-mentioned fourth message to be output through the output device.

11. In any one of paragraphs 1 through 10, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Identifying a user based on user authentication information input through the above input device, and A sperm device that causes information about the task and information about the specific step to be output in real time based on the context information of the identified user.

12. A method of operation in an electronic device (101, 201), The operation of receiving a user query through an input device of the above electronic device; An action of identifying a task related to the user query based on receiving the user query; An operation to verify information regarding the above task and a plurality of steps for performing the above task; An operation of acquiring data related to the situation of a user performing the above task in real time through the above input device; Based on the above data, an operation to identify a specific step among the plurality of steps corresponding to the user's situation; and A method comprising the operation of outputting information about the operation and information about the specific step as a response to the user query through an output device of the electronic device.

13. In Paragraph 12, Information regarding the above task includes information related to the performance of the identified task and information regarding a plurality of sub-tasks corresponding to the plurality of steps, and The information regarding the specific step includes information related to the performance of the specific step among the plurality of sub-tasks corresponding to the plurality of steps, and A method of obtaining a response to the above user query through an artificial intelligence model (320).

14. In paragraph 12 or 13, the above method is, An action of obtaining additional information including a detailed description of a sub-task of the specific step through the artificial intelligence model based on data related to the situation of the user; and A method further comprising the operation of outputting the above additional information through the above output device.

15. In a non-transient computer-readable storage medium storing one or more programs, the one or more programs, when executed by at least one processor (120, 210) of an electronic device (101, 201), cause the electronic device: The operation of receiving a user query through an input device of the above electronic device; An action of identifying a task related to the user query based on receiving the user query; An operation to verify information regarding the above task and a plurality of steps for performing the above task; An operation of acquiring data related to the situation of a user performing the above task in real time through the above input device; Based on the above data, an operation to identify a specific step among the plurality of steps corresponding to the user's situation; and A non-transient computer-readable storage medium comprising executable commands to perform an operation of outputting information about the operation and information about the specific step as a response to the user query through an output device of the electronic device.

Images