Electronic device and method for executing task on basis of context data in electronic device

Abstract

An electronic device according to one embodiment comprises: a communication circuit; at least one processor; and a memory storing instructions (130 of figure 1, and 230 of figure 2), wherein the instructions, when executed individually or collectively by the at least one processor, may be configured to cause the electronic device to: acquire a request for a task; acquire context data related to the task; acquire a first action set including a first action and a second action corresponding to the task on the basis of at least a part of the context data; execute the first action set on the basis at least in part of each of the first action and the second action being successfully executable; and execute a second action set including a third action instead of at least one action, on the basis at least in part of at least one of the first action or the second action being unable to be successfully executed. In addition, other embodiments may be included.

Description

Electronic device and method of executing tasks based on context data in the electronic device

[0001] The present disclosure relates to an electronic device and a method for executing a task based on context data in the electronic device.

[0002] An AI assistant is artificial intelligence-based software that provides information or performs specific tasks in response to user commands or queries.

[0003] An AI Assistant can use an AI model to generate answers to user queries or perform specific actions, and the AI ​​Assistant can control operations in the environment of electronic devices such as mobile phones and tablets, and

[0004] Users of electronic devices can interact with the AI ​​Assistant through natural language and receive various functions through the AI ​​Assistant.

[0005] An electronic device according to one embodiment may include a communication circuit, at least one processor, and a memory for storing instructions. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may obtain a request for a task. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may obtain context data related to the task. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may obtain a first set of actions including a first action and a second action corresponding to the task based on at least a portion of the context data. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may execute the first set of actions based on at least a portion of the fact that each of the first action and the second action is executable. The instructions according to one embodiment may be configured to execute a second set of actions including a third action instead of the at least one action when executed individually or collectively by the at least one processor, based at least in part on the fact that the first action or at least one of the second actions cannot be successfully executed.

[0006] A method for executing a task based on context in an electronic device according to one embodiment may include an operation of obtaining a request for a task. The method according to one embodiment may include an operation of obtaining context data related to the task. The method according to one embodiment may include an operation of obtaining a first action set including a first action and a second action corresponding to the task based on at least a portion of the context data. The method according to one embodiment may include an operation of executing the first action set based at least a portion of the fact that each of the first action and the second action is executable. The method according to one embodiment may include an operation of executing a second action set including a third action instead of the at least one action based at least a portion of the fact that at least one of the first action or the second action cannot be successfully executed.

[0007] In a non-volatile storage medium storing instructions according to one embodiment, the instructions are configured to cause the electronic device to perform at least one operation when executed by the electronic device, wherein the at least one operation may include an operation of obtaining context data related to the task. According to one embodiment, the at least one operation may include an operation of obtaining a first action set including a first action and a second action corresponding to the task based on at least a portion of the context data. According to one embodiment, the at least one operation may include an operation of executing the first action set based at least a portion of the fact that each of the first action and the second action is executable. According to one embodiment, the at least one operation may include an operation of executing a second action set including a third action instead of the at least one action based at least a portion of the fact that at least one of the first action or the second action cannot be successfully executed.

[0008] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment.

[0009] FIG. 2a is a block diagram of an electronic device according to one embodiment, and FIG. 2b is a block diagram for explaining the configuration of a processor and an AI model according to one embodiment.

[0010] FIG. 3 is a diagram illustrating the operation of executing a task based on context data in an electronic device according to one embodiment.

[0011] FIGS. 4a, FIGS. 4b, FIGS. 4c, FIGS. 4d and FIGS. 4e are drawings for illustrating an operation of executing a task based on context data in an electronic device according to one embodiment.

[0012] FIGS. 5a and 5b are drawings for explaining the operation of executing a task based on context data in an electronic device according to one embodiment.

[0013] FIG. 6 is a diagram illustrating the operation of executing a task based on context data in an electronic device according to one embodiment.

[0014] FIG. 7 is a diagram illustrating the operation of executing a task based on context data in an electronic device according to one embodiment.

[0015] FIG. 8 is a diagram illustrating the operation of training an AI agent in an electronic device according to one embodiment.

[0016] FIGS. 9a and 9b are drawings for explaining the operation of training an AI agent in an electronic device according to one embodiment.

[0017] FIG. 10 is a flowchart illustrating the operation of executing a task based on context data in an electronic device according to one embodiment.

[0018] FIG. 11 is a flowchart illustrating the operation of executing a task based on context data in an electronic device according to one embodiment.

[0019] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to one embodiment. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0020] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0021] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an AI model. The AI ​​model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the AI ​​model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The AI ​​model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the AI ​​model may include a software structure, either additionally or substantially.

[0022] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0023] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0024] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0025] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0026] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0027] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0028] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0029] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0030] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0031] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0032] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0033] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0034] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0035] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0036] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0037] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0038] According to one embodiment, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0039] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0040] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0041] FIG. 2a is a block diagram of an electronic device according to one embodiment, and FIG. 2b is a block diagram for explaining the configuration of a processor and an AI model according to one embodiment.

[0042] Referring to FIGS. 2a and 2b, according to one embodiment, the electronic device (201) may include a processor (220), a memory (230), a display (260), and a communication circuit (290).

[0043] According to one embodiment, the processor (220) can perform overall control operations of the electronic device (201). According to one embodiment, the processor (220) can control at least one other component (e.g., hardware or software component) of the electronic device (201) connected to the processor (220) by executing software (e.g., program (140) of FIG. 1), and can perform data processing or operations based on instructions. According to one embodiment, the instructions may include instructions composed of machine language that can be processed by the electronic device (201) or the processor (220). For example, the instructions may include instructions corresponding to operation instructions used in the program.

[0044] According to one embodiment, the processor (220) can obtain context data related to the task (e.g., first context data) when it checks a first instruction requesting a task received from a user.

[0045] According to one embodiment, the processor (220) can use an AI assistant to check a first command requesting the execution of a task received from a user.

[0046] According to one embodiment, the processor (220) can identify a user's voice command input through the microphone of the electronic device or a user's text command input through the display of the electronic device as a first command.

[0047] According to one embodiment, the processor (220) can use an AI model (231) (e.g., a first AI model (231a)) to analyze a first instruction and identify at least one task classified into action units as context data.

[0048] According to one embodiment, the first AI model (231a) may include an LLM (Large Language Model).

[0049] For example, when the processor (220) confirms the receipt of a first command for performing a task, such as "book a restaurant for a date and send a text message," it can identify at least one task, such as "search for a restaurant" (first task), "book a restaurant" (second task), "summary of reservation details" (third task), and / or "send a text message" (fourth task), by separating the first command into action units.

[0050] For example, the processor (220) can identify a restaurant search (1st task), a restaurant reservation (2nd task), a reservation history summary (3rd task) and / or a text message sending (4th task) as context data.

[0051] According to one embodiment, the processor (220) can use an AI model (231) (e.g., a first AI model (231a)) to identify at least one action data for executing at least one task as context data.

[0052] For example, the processor (220) can identify at least one action data including "preferred restaurant" (first action data), "dateable schedule" (second action data), "person relationship" (third action data), and / or "message recipient" (fourth action data) to execute "restaurant search" (first task), "restaurant reservation" (second task), "reservation history summary" (third task), and / or "message sending" (fourth task).

[0053] For example, the processor (220) can identify a preferred restaurant (first action data), a "dateable schedule" (second action data), a "person relationship" (third action data), and / or a "message recipient" (fourth action data) as context data.

[0054] According to one embodiment, the processor (220) can use an AI model (231) (e.g., a first AI model (231a)) to identify information corresponding to at least one application that can obtain information about action data of at least one of at least one application stored in an electronic device as context data.

[0055] For example, the processor (220) can check the information of a first application corresponding to a first application (e.g., a gallery application) that can obtain information about a "preferred restaurant" (first action data) among at least one application, and the information of a second application corresponding to a second application (e.g., an internet application).

[0056] For example, the processor (220) can check the information of the first application corresponding to the first application (e.g., gallery application) and the information of the second application corresponding to the second application (e.g., internet application) as context data.

[0057] For example, the processor (220) can check information of a third application corresponding to a third application (e.g., a calendar application) that can obtain information about a "dateable schedule (second action data)" among at least one application.

[0058] For example, the processor (220) can check the information of a third application corresponding to a third application (e.g., a calendar application) as context data.

[0059] For example, the processor (220) can check the information of the first application corresponding to the first application (e.g., gallery application), the information of the fourth application corresponding to the fourth application (e.g., phone application), and the information of the fifth application corresponding to the fifth application (e.g., contact application) among at least one application capable of obtaining information about "person relationships (third action data)" among at least one application.

[0060] For example, the processor (220) can check the information of the first application corresponding to the first application (e.g., gallery application), the information of the fourth application corresponding to the fourth application (e.g., phone application), and the information of the fifth application corresponding to the fifth application (e.g., contact application) as context data. For example, the processor (220) can check the information of the sixth application corresponding to the sixth application (e.g., text application) which can obtain information about the "message recipient" (fourth action data) among at least one application.

[0061] For example, the processor (220) can check the information of the sixth application corresponding to the sixth application (e.g., a character application) as context data.

[0062] According to one embodiment, the processor (220) can use an AI model (231) (e.g., a first AI model (231a)) to identify at least one application among at least one applications stored in an electronic device (201) that can obtain at least one action data, and can obtain at least one action data by combining the data obtained from the identified at least one application.

[0063] According to one embodiment, the processor (220) can minimize execution time by using an AI model (231) (e.g., a first AI model (231a)) to prioritize the execution of applications that appear most relevant, without checking all possible applications.

[0064] According to one embodiment, the processor (220) can store data related to at least one task used more than a reference number of times in local storage and use it as a cache to obtain the data without directly executing the application.

[0065] According to one embodiment, the processor (220) can obtain information of at least one external electronic device corresponding to at least one external electronic device from at least one external electronic device associated with the electronic device (201) as context data using an AI model (231) (e.g., a first AI model (231a)).

[0066] According to one embodiment, the processor (220) can identify at least one external electronic device connected wirelessly and / or wired to the electronic device (201) as at least one external electronic device associated with the electronic device (201). According to one embodiment, the processor (220) can identify at least one external electronic device connected wired to the electronic device (201) via a communication circuit (290) as at least one external electronic device associated with the electronic device (201).

[0067] According to one embodiment, the processor (220) can identify at least one external electronic device logged in with the same account as the electronic device (201) as at least one external electronic device associated with the electronic device (201).

[0068] According to one embodiment, the processor (220) can identify at least one external electronic device (e.g., IoT device) connected to the same network as the electronic device (201) as at least one external electronic device associated with the electronic device.

[0069] For example, at least one external electronic device associated with the electronic device (201) may include a wearable device (e.g., watch, AR glasses and / or earbuds), a vehicle navigation system, a tablet, and / or a home appliance (e.g., TV).

[0070] According to one embodiment, the processor (220) may obtain at least some of the type of device, capability information, constraint information, or status information from at least one external electronic device as information of at least one external electronic device. According to one embodiment, the processor (220) may identify the type of component included in at least one external electronic device (e.g., display, sensor, battery and / or network module) as capability information. According to one embodiment, the processor (220) may identify the authorization information of at least one external electronic device and, in addition to the authorization information, identify information that cannot be performed by at least one external electronic device as constraint information.

[0071] According to one embodiment, the processor (220) can check the activation state (active or inactive), operation state (on or off), and / or battery state (remaining amount and / or charge state) of the type of component included in at least one external electronic device as state information.

[0072] According to one embodiment, the processor (220) may acquire at least one of the information of a first application corresponding to a first application among at least one application stored in an electronic device (201) or the information of a second application corresponding to a second application as context data, and may acquire at least a portion of the information of a first external electronic device corresponding to a first external electronic device among at least one external electronic device or the information of a second external electronic device corresponding to a second external electronic device as context data.

[0073] According to one embodiment, the processor (220) may acquire at least some of the first capability information, first constraint information, or first state information of the first external electronic device as information of the first external electronic device. According to one embodiment, the processor (220) may acquire at least some of the second capability information, second constraint information, or second state information of the second external electronic device as information of the second external electronic device.

[0074] According to one embodiment, the processor (220) can identify at least one candidate external electronic device capable of performing a task based on information from at least one external electronic device.

[0075] According to one embodiment, the processor (220) can obtain information of an electronic device as context data by using an AI model (231) (e.g., a first AI model (231a)). According to one embodiment, the processor (220) can obtain at least some of the capability information, constraint information, or state information of the electronic device as information of the electronic device.

[0076] According to one embodiment, the processor (220) can obtain a first action set including a first action and a second action corresponding to a task based on at least a portion of context data by using an AI model (231) (e.g., a first AI model (231a) or a second AI model (231b)).

[0077] According to one embodiment, the processor (220) identifies at least one of at least one task, at least one action data, information corresponding to at least one application and / or information of at least one external electronic device as context data, and uses an AI model (231) (e.g., a first AI model (231a) or a second AI model (231b)) to obtain a first action set including a first action and a second action corresponding to the task based on at least a portion of the context data.

[0078] According to one embodiment, the processor (220) may identify context data including a first task among at least one task, first action data among at least one action data, information of a first application among at least one application, information of a first external electronic device among at least one external electronic device, and / or information of an electronic device, and generate a first action based on the identified context data. According to one embodiment, the processor (220) may identify context data including a second task among at least one task, second action data among at least one action data, information of a second application among at least one application, information of a second external electronic device among at least one external electronic device, and / or information of an electronic device, and generate a second action based on the identified context data. For example, the processor (220) may obtain a first action set including a first action or a second action based on at least one task, at least one action data, information of at least one application, and / or information of at least one external electronic device through artificial intelligence (e.g., LLM).

[0079] According to one embodiment, the second AI model (231b)) may include a LAM (Large Action Model) model.

[0080] According to one embodiment, the processor (220) may use an AI model (231) (e.g., a second AI model (231b)) to set (configure) a first application among at least one application and / or a first external electronic device (e.g., a first device) among at least one external electronic device as a means for executing a first action. According to one embodiment, the processor (220) may use an AI model (231) (e.g., a second AI model (231b)) to set (configure) a second application among at least one application that is different from the first application and / or a second external electronic device (e.g., a second external electronic device) that is different from the first external electronic device (e.g., a first device) among at least one external electronic device as a means for executing a second action.

[0081] According to one embodiment, the processor (220) can verify whether the first action and the second action can be successfully executed based on the availability of first data to be used by a first application set as a means for executing the first action and / or second data to be used by a second application set as a means for executing the second action, using an AI model (231) (e.g., a second AI model (231b)).

[0082] According to one embodiment, the processor (220) identifies the first data and / or second data as information necessary for the execution of the first application and / or second application, and may include, for example, system resource information necessary for the execution of the application (e.g., network status, storage space of the device and / or, battery level).

[0083] According to one embodiment, the processor (220) can generate first planning data for a first action and second planning data for a second action using an AI model (231) (e.g., a second AI model (231b)).

[0084] According to one embodiment, the processor (220) may generate first planning data for performing a first action based on at least one task among at least one task included in a first action, at least one action data among at least one action data, information of at least one application among at least one application, information of at least one external electronic device among at least one external electronic device, and / or information of an electronic device by using an AI model (231) (e.g., a second AI model (231b)). According to one embodiment, the processor (220) may generate second planning data for performing a second action based on at least one task among at least one task included in a second action, at least one action data among at least one action data, information of at least one application among at least one application, information of at least one external electronic device among at least one external electronic device, and / or information of an electronic device by using an AI model (231) (e.g., a second AI model (231b)).

[0085] According to one embodiment, planning data for performing an action includes information necessary for an AI model to perform an action, such as specific plans, conditions, resources, sequence of operations, and / or time constraints for realizing the action, and can serve as an overall roadmap that includes all variables and environment settings necessary for the action to be successfully executed, providing specific instructions necessary to execute a first action.

[0086] According to one embodiment, the processor (220) can verify whether the first action and the second action can be successfully executed based on context data and / or the first planning data and the second planning data by using an AI model (231) (e.g., the second AI model (231b)). According to one embodiment, the processor (220) can verify whether the first action and the second action can be successfully executed by performing a simulation of the first action and the second action based on context data and / or the first planning data and the second planning data by using an AI model (231) (e.g., the second AI model (231b)).

[0087] According to one embodiment, the processor (220) can execute a first action set representing the execution of a task upon receiving a first instruction by using an AI model (231) (e.g., a second AI model (231b)) to verify that each of the first action and the second action included in the first action set can be successfully executed.

[0088] According to one embodiment, the second AI model (231b) may include a first LAM (Large Action Model) model for generating planning data and a second LAM (Large Action Model) model for verifying actions, or may include one LAM (Large Action Model) model for generating planning data and verifying actions.

[0089] According to one embodiment, if the processor (220) confirms through verification that the first action and the second action can be successfully executed, it identifies a device for executing the first action and the second action among an electronic device and / or at least one external electronic device, and can execute the first action and the second action using the identified device.

[0090] According to one embodiment, if there are multiple candidate devices for executing the first action and the second action, or if no device for executing the first action and the second action is identified in the verification operation, the processor (220) may identify at least one device most suitable for processing the task corresponding to the first instruction as at least one device capable of performing the first action and the second action based on context data and / or information of at least one external electronic device.

[0091] According to one embodiment, the processor (220) may identify at least one device suitable for receiving user feedback as at least one device capable of performing the first action and the second action, if there are multiple candidate devices for executing the first action and the second action, or if no device for executing the first action and the second action is identified in the verification operation.

[0092] According to one embodiment, after executing a first set of actions including a first action and a second action, the processor (220) may use an AI model (231) (e.g., a second AI model (231b)) to perform additional verification of whether the first action and the second action were successfully executed.

[0093] According to one embodiment, the processor (220) may execute a third action set including a fourth action (complementary action) that is different from the first action and the second action included in the first action set, based on user feedback information regarding the result of executing the first action set.

[0094] According to one embodiment, the processor (220) may execute a first action set including a first action and a second action, obtain user feedback information regarding the result of executing the first action set including the first action and the second action, obtain additional context data for subsequent supplementation in addition to context data based on at least part of the feedback information, generate a third action set including a fourth action different from the actions included in the first action set (e.g., the first action and the second action) based on at least part of the additional context data, and execute the third action set.

[0095] For example, the processor (220) may, after executing the action of sending a text message, if the recipient of the text message does not receive and read the text message, or if the user of the electronic device checks for feedback information that the user has entered an additional new schedule on the date of the restaurant reservation made through the calendar application, provide a notification to the user of the electronic device to know the current situation based on at least some of the additional context data (e.g., whether the schedule is duplicated, the importance of the newly added schedule, the user's preferred method of scheduling and / or the distance between the reserved restaurant and the additional schedule location, and / or generate a fourth action (complementary action) to adjust the schedule in the calendar application.

[0096] According to one embodiment, the processor (220) can generate a second action set containing a third action instead of at least one action (e.g., the first action and / or the second action) that cannot be executed successfully by using an AI model (231) (e.g., a second AI model (231b)) to verify that at least one of the first action and the second action included in the first action set cannot be executed successfully.

[0097] According to one embodiment, the processor (220) can obtain context data (e.g., second context data) for obtaining a third action by using the AI ​​model (231) (e.g., second AI model (231b)) to verify that at least one of the first action and the second action included in the first action set (e.g., first action and / or second action) cannot be successfully executed.

[0098] According to one embodiment, the processor (220) can obtain a first action set including a first action and a second action based on at least a portion of the first context data among the context data.

[0099] According to one embodiment, the processor (220) can obtain a second action set including a third action based on at least a portion of second context data that is at least partially different from the first context data among the context data.

[0100] According to one embodiment, the processor (220) can use an AI model (231) (e.g., a second AI model (231b)) to set (set) a second application different from an application (e.g., a first application) that corresponds to at least one action (e.g., a first action or a second action) that cannot be executed among at least one application, based on context data (e.g., second context data) and / or information of at least one external electronic device corresponding to at least one external electronic device, as a means for executing a third action.

[0101] According to one embodiment, the processor (220) can use an AI model (231) (e.g., a second AI model (231b)) to set (set) an external electronic device (e.g., a second device) and another external electronic device (e.g., a second device) as means for executing a third action, based on context data (e.g., second context data) and / or information of at least one external electronic device corresponding to at least one external electronic device.

[0102] According to one embodiment, the processor (220) can obtain additional data related to a task generated through at least one external electronic device connected via a communication circuit (290) by using an AI model (231) (e.g., a second AI model (231b)), and can obtain a third action based on the additional data.

[0103] According to one embodiment, the processor (220) may provide an indication to the user of the electronic device (201) that at least one action that cannot be executed (e.g., a first action or a second action) has been replaced by a third action.

[0104] According to one embodiment, the processor (220) can execute a second set of actions representing the execution of a task upon receiving a first instruction if it verifies that a second set of actions including a third action can be successfully executed using an AI model (231) (e.g., a second AI model (231b)).

[0105] According to one embodiment, the processor (220) may, by using an AI model (231) (e.g., a second AI model (231b)), if it verifies that a third action cannot be successfully executed, execute at least a portion of a second action set including a fifth action different from the third action, or create a fourth action set different from the second action set and execute at least a portion of a fourth action set.

[0106] According to one embodiment, if the processor (220) identifies a third external electronic device among at least one external electronic device connected through a communication circuit (290) as a means for executing a second action set including a third action, it may transmit corresponding control information for executing a second action set including a third action to the third external electronic device. According to one embodiment, the processor (220) may transmit corresponding control information to the third external electronic device, which includes at least one part of a command directing the execution of a second action set including a third action, at least one task, at least one action data, or at least one application information.

[0107] According to one embodiment, the processor (220) may update the AI ​​model (231) (e.g., the second AI model (231b)) based on at least a portion of user feedback information regarding the result of an execution operation of one action set successfully executed among a first action set including a first action and a second action, a second action set including a third action, a third action set including a fourth action, or a fourth action set including a fifth action. According to one embodiment, the processor (220) may obtain user feedback information regarding the result of an execution operation of one action set successfully executed among a first action set including a first action and a second action, a second action set including a third action, a third action set including a fourth action, or a fourth action set including a fifth action.

[0108] According to one embodiment, the processor (220) may provide at least a portion of the feedback information to the AI ​​model (231) (e.g., the second AI model (231b)) so that the AI ​​model (231) (e.g., the second AI model (231b)) can be updated based on at least a portion of the user's feedback information. According to one embodiment, the processor (220) may use the AI ​​(231) (e.g., the second AI model (231b) of FIG. 2b) to generate preference information indicating the user's preference based on the user's feedback information, and provide the preference information to the AI ​​model (231) (e.g., the second AI model (231b)) so that the AI ​​model (231) (e.g., the second AI model (231b)) can be updated based on the preference information.

[0109] According to one embodiment, the processor (220) may perform an operation to verify and execute a set of actions based on context data by using an external AI model (281) instead of an AI model (231) (e.g., a first AI model (231a) and / or a second AI model (231b)). According to one embodiment, the processor (220) may perform an operation to verify and execute a set of actions based on context data by using both the AI ​​model (231) (e.g., a first AI model (231a) and / or a second AI model (231b)) and the external AI model (281).

[0110] According to one embodiment, the memory (230) may be implemented substantially identically or similarly to the memory (130) of FIG. 1.

[0111] According to one embodiment, an AI model (231) may be stored in a memory (230), and the AI ​​model (231) may operate on the device inside an electronic device, and the AI ​​model (231) may be configured in various ways, such as including a first AI model (231a) and a second AI model (231b), or including at least one of the first AI model (231a) and the second AI model (231b), or including at least one of the first AI model (231a) and the second AI model (231b) as one identical AI model, or including the first AI model (231a) and the second AI model (231b) as one identical AI model.

[0112] According to one embodiment, at least some of the on-device AI models (231) can be operated as external AI models.

[0113] According to one embodiment, the on-device AI model (231) performs processing of sensitive information such as personal information, and the external AI model (281) can perform processing of information excluding sensitive information such as personal information.

[0114] According to one embodiment, the on-device AI model (231) is an AI model implemented within an electronic device (201) and can provide various functions without a network.

[0115] According to one embodiment, a plurality of AI models may be stored in the memory (230). FIG. 7 is a diagram illustrating an operation to output three-dimensional audio data by controlling an audio input / output unit in a wearable electronic device according to one embodiment.

[0116] According to one embodiment, each of the plurality of AI models may be a model trained based on a specified type of learning algorithm, and may be an AI model implemented to receive various types of data (or content) as input, perform calculations, and output (or obtain) result data.

[0117] According to one embodiment, a plurality of AI models may include generative AI models.

[0118] According to one embodiment, a generative AI model can generate and output new content (e.g., text, images, and / or computer code, etc.) based on what it has learned in response to an input prompt. For example, in an electronic device (601), learning is performed to output specific types of result data as output data using data of specified types based on a machine learning algorithm or a deep learning algorithm, thereby generating multiple AI models (e.g., machine learning models and deep learning models) that are stored in the electronic device (601), or AI models learned from an external electronic device (e.g., an external server) may be transmitted to and stored in the electronic device (601). For example, the electronic device (601) can output input data (input data) as output data of a model learned through artificial intelligence of specified types based on a machine learning algorithm or a deep learning algorithm. Machine learning algorithms include supervised learning algorithms such as linear regression and logistic regression, unsupervised learning algorithms such as clustering, visualization and dimensionality reduction, and association rule learning, and reinforcement learning algorithms, and deep learning algorithms may include Artificial Neural Networks (ANN), Deep Neural Networks (DNN), and Convolutional Neural Networks (CNN), and may further include various learning algorithms not limited to those described.The trained AI model includes at least one operation (e.g., a convolution layer or a pooling layer) for processing input data, and can be implemented to output result data by performing operations on the input data based on at least one operation.

[0119] According to one embodiment, an application that can be connected to an external AI model (281) may be stored in the memory (230).

[0120] According to one embodiment, the external AI model (281) may include each of the first AI model (231a) and / or the second AI model (231b), or include at least one of the first AI model (231a) and / or the second AI model (231b), or include the first AI model (231a) and / or the second AI model (231b) as one identical AI model.

[0121] According to one embodiment, an AI assistant and an AI agent may be stored in the memory (230).

[0122] According to one embodiment, the AI ​​agent can verify and execute a set of actions based on context data using an AI model (231) in the same manner as the operation of the processor (220) under the control of the processor (220). According to one embodiment, when the AI ​​agent receives a first command for task execution from an AI assistant, it can verify a set of actions based on context data using an AI model (231) and execute a verified set of actions corresponding to the task. According to one embodiment, the AI ​​agent may refer to a system that can autonomously perform tasks requested by a user using various available tools, going beyond a simple question-and-answer system. According to one embodiment, an AI agent can acquire data from various environments or sources in various multi-device environments, use a connected device as a tool to perform an action, use another device connected to a user's account, or perform necessary follow-up tasks even after executing an action, thereby helping to enhance the operation of an AI assistant.

[0123] According to one embodiment, the display (260) may be implemented substantially identically or similarly to the display (160) of FIG. 1.

[0124] According to one embodiment, the communication circuit (290) can form a communication connection with an external electronic device (e.g., another electronic device, or a server) using various types of communication methods and transmit and / or receive data. As described above, the communication methods may include a communication method that establishes a direct communication connection such as Bluetooth and / or Wi-Fi Direct, a communication method that uses an access point (AP) (e.g., Wi-Fi communication), or a communication method that uses cellular communication using a base station (e.g., 3G, 4G / LTE, 5G). Since the communication circuit (290) can be implemented as described above in the communication module (190) in FIG. 1, a redundant description is omitted.

[0125] FIG. 3 is a diagram illustrating the operation of executing a task based on context data in an electronic device according to one embodiment.

[0126] According to one embodiment, the electronic device (201) can perform an operation of executing a task based on context data using a processor (e.g., the processor (220) of FIG. 2a to 2b) or an AI agent (Agent) under the control of the processor.

[0127] The operation of FIG. 3 according to one embodiment describes an operation of executing a task based on context data using an AI agent, but a processor (e.g., the processor (220) of FIG. 2a and FIG. 2b) can also perform the same operation as the AI ​​agent.

[0128] Referring to FIG. 3, according to one embodiment, an AI agent (Agent) (270) may include a data acquisition unit (271) and an action generation and execution unit (273).

[0129] According to one embodiment, the AI ​​agent (Agent) (270) may include a first AI agent (Agent) that performs the same function as the data acquisition unit (271) and a second AI agent (Agent) that performs the same function as the action creation and execution unit (273).

[0130] According to one embodiment, the data acquisition unit (271) can receive a first command from an AI assistant.

[0131] According to one embodiment, when the AI ​​assistant confirms a first command (301a and / or 301b) from a user requesting the execution of a task through the microphone or display of the electronic device (201), it can transmit the first command (301a and / or 301b) to the data acquisition unit (271).

[0132] According to one embodiment, the data acquisition unit (271) can detect at least one task classified by action unit.

[0133] According to one embodiment, the data acquisition unit (271) can acquire at least one task (311) classified into action units by analyzing a first command using a first AI model (e.g., the first AI model (231a) of FIG. 2b) (e.g., LLM), at least one action data for executing at least one task, information (313) corresponding to at least one application from which at least one action data can be acquired from an app data source (313a) stored in the electronic device (e.g., internet app, calendar app, email app and / or gallery app), information (315) corresponding to at least one external electronic device related to the electronic device identified from a device data source (315a), and / or at least some of the information of the electronic device identified from a device state DB (317a) of the electronic device's memory as context data.

[0134] For example, the data acquisition unit (271) can use the first AI model (e.g., the first AI model (231a) of FIG. 2b) (e.g., LLM) to check the input of a first command such as “book a restaurant for a date and send a text message” entered by the user, analyze the first command, and based on the analysis of the first command, classify the first command into action units to detect at least one task such as “search for a restaurant,” “book a restaurant,” “summary of reservation details,” and / or “send a text message” (311).

[0135] For example, according to one embodiment, the data acquisition unit (271) can acquire information related to installed S / W, such as restaurant information through internet search, as information of at least one application (313).

[0136] For example, the data acquisition unit (271) can acquire data acquired by sensing from a wearable device such as a smart watch and / or AI glasses using a first AI model (e.g., the first AI model (231a) of FIG. 2b (e.g., LLM)) as information of at least one external electronic device corresponding to at least one external electronic device (315).

[0137] For example, the data acquisition unit (271) can acquire H / W-related information such as network status, battery status, and display status as information of the electronic device by using the first AI model (e.g., the first AI model (231a) of FIG. 2b) (e.g., LLM) (317).

[0138] According to one embodiment, the data acquisition unit (271) can acquire a first action set based on context data using a first AI model (e.g., the first AI model (231a) of FIG. 2b).

[0139] According to one embodiment, the action generation and execution unit (273) can identify a means for executing a first action set based on information of at least one application corresponding to at least one application stored in the electronic device (201) and / or information of at least one external electronic device corresponding to at least one external electronic device associated with the electronic device, using a second AI model (e.g., the second AI model (231b) of FIG. 2b).

[0140] According to one embodiment, the action generation and execution unit (273) can generate planning data for the first action set using the second AI model (e.g., the second AI model (231b) of FIG. 2b) (331).

[0141] According to one embodiment, the second AI model can perform action planning using a Large Action Model (LAM).

[0142] According to one embodiment, the action generation and execution unit (273) can verify whether the first action set can be successfully executed based on the first planning data and the second planning data using the second AI model (e.g., the second AI model (231b) of FIG. 2b) (333).

[0143] According to one embodiment, the action generation and execution unit (273) can use a second AI model (e.g., the second AI model (231b) of FIG. 2b) to determine whether the first action and the second action included in the first action set can be performed on the electronic device or whether additional information is required (333).

[0144] For example, the action generation and execution unit (273) can use the second AI model (e.g., the second AI model (231b) of FIG. 2b) to determine that the first action and the second action included in the first action set cannot be performed on the electronic device or that additional information is needed, and can adjust the first planning data and the second planning data or generate a second action set including a third action (333).

[0145] According to one embodiment, the action generation and execution unit (273), when verifying that the first action set can be successfully executed, can execute the first action set (335) which indicates the execution of a task upon receiving the first command on a device identified as a means for executing the first action set (e.g., electronic device (201), AR glasses (301) or TV (401)).

[0146] According to one embodiment, the action generation and execution unit (273) may execute the first action set by selecting a device to execute the first action set, or terminate the operation of the first action set when the user's request is fully performed (335).

[0147] According to one embodiment, the action generation and execution unit (273) can perform additional verification using a second AI model (e.g., the second AI model (231b) of FIG. 2b) after the first action set is executed (337).

[0148] According to one embodiment, the action generation and execution unit (273) can obtain a third action set for a supplementary action (337) by using a second AI model (e.g., the second AI model (231b) of FIG. 2b) to identify an additional supplementary action after the first action set has been executed.

[0149] According to one embodiment, the action generation and execution unit (273) checks whether the first action set has operated normally and can generate additional action sets if additional supplementation is needed.

[0150] According to one embodiment, the action creation and execution unit (273) can execute a third action set corresponding to an additional supplementary action (e.g., adding a restaurant reservation schedule to a gallery application) (339).

[0151] The operation of Fig. 3 can be explained in detail in Figs. 4a, 4b, 4c, 4d, and 4e below.

[0152] FIGS. 4a, FIGS. 4b, FIGS. 4c, FIGS. 4d and FIGS. 4e are drawings for illustrating an operation of executing a task based on context data in an electronic device according to one embodiment.

[0153] Referring to FIG. 4a, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) (e.g., a processor (e.g., the processor (220) of FIG. 2a to 2b), or an AI agent (e.g., the AI ​​agent (270) of FIG. 3)) can confirm "book a restaurant for a date and send a text message" as a first command (411) requesting a task.

[0154] According to one embodiment, the electronic device can use an AI assistant to confirm "book a restaurant for a date and send a text message" as a first command (411) requesting the execution of a task received from a user.

[0155] According to one embodiment, the electronic device can identify a user's voice command input through the microphone of the electronic device or a user's text command input through the display of the electronic device as a first command (411).

[0156] According to one embodiment, the electronic device can use an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b) or the second AI model (e.g., the second AI model (231b) of FIG. 2b)) to analyze the first command (411) and identify at least one task (413) classified into action units as context data.

[0157] According to one embodiment, the electronic device can use an LLM (Large Language Model) to analyze a first instruction (411) and identify at least one task (413) classified into action units as context data.

[0158] According to one embodiment, the electronic device can use an AI model to analyze the first command (411), "book a restaurant for a date and send a text message," and detect at least one task corresponding to an action unit. According to one embodiment, the electronic device can detect, as at least one task, the first task (413a), "search for a restaurant," the second task (413b), "book a restaurant," the third task (413c), "summarize reservation details," and the fourth task (413d), "send a text message."

[0159] According to one embodiment, the electronic device can use an AI model to identify at least one action data (415) as context data for executing at least one task (413). According to one embodiment, the electronic device can use an AI model to identify a first action data (415a) "preferred restaurant," a second action data (415b) "dateable schedule," a third action data (415c) "person relationships," and / or a fourth action data (415d) "message recipient" for executing a first task (413a) "restaurant search," a second task (413b) "restaurant reservation," a third task (413c) "reservation history summary," and / or a fourth task (413d) "message sending recipient."

[0160] Referring to FIG. 4b, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2a to FIG. 2b) (e.g., a processor (e.g., the processor (220) of FIG. 2a to FIG. 2b)), or an AI agent (e.g., the AI ​​agent (270) of FIG. 3) can identify information corresponding to at least one application that can obtain action data among at least one application stored in the electronic device as context data by using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b) or the second AI model (e.g., the second AI model (231b) of FIG. 2b).

[0161] According to one embodiment, the electronic device can check the information of a first application corresponding to a first application (e.g., a gallery application) (417a) that can obtain information about a "preferred restaurant," which is the first action data (415a) among at least one application stored in the electronic device, and the information of a second application corresponding to a second application (e.g., an internet application) (417b).

[0162] According to one embodiment, the electronic device can execute a first application (e.g., a gallery application) (417a) to identify a restaurant frequently visited by the user as a preferred restaurant and obtain tag information of food photographed at the restaurant.

[0163] According to one embodiment, the electronic device can execute a second application (e.g., an internet application) (417b) to obtain information about famous restaurants based on the user's current location.

[0164] According to one embodiment, the electronic device can check information of a third application corresponding to a third application (e.g., a calendar application) (417c) that can obtain information about a "dateable schedule" which is a second action data (415b) among at least one application.

[0165] According to one embodiment, the electronic device can obtain information about a "person relationship" which is a third action data (415c) among at least one application, information about a fourth application corresponding to a first application (e.g., a gallery application) (417a), information about a fourth application corresponding to a fourth application (e.g., a phone application) (417d), and information about a fifth application corresponding to a fifth application (e.g., a contact application) (417e).

[0166] According to one embodiment, the electronic device can check information of a sixth application corresponding to a sixth application (e.g., a text message application) (417f) that can obtain information about a "message recipient" which is the fourth action data (415d) among at least one application.

[0167] According to one embodiment, the electronic device can minimize execution time by using an AI model to prioritize the execution of applications that appear most relevant, without checking all possible applications.

[0168] According to one embodiment, the electronic device can store data (information) related to at least one task used more than a reference number of times in local storage and use it as a cache to enable the acquisition of data (information) without directly executing an application.

[0169] According to one embodiment, the electronic device may perform an inference operation (419) to obtain information about at least one action data through at least one application using an AI model, and the inference operation (419) may be displayed through the display of the electronic device (e.g., the display (260) of FIG. 2a) or processed as an internal operation without being displayed.

[0170] According to one embodiment, the electronic device can perform a reasoning operation (419) using a Chain of Thought (COT), which is a technique that induces the expression of a logical thought process step by step in an AI model.

[0171] Referring to FIG. 4c, according to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2a to 2b) (e.g., processor (e.g., processor (220) of FIG. 2a to 2b)), or an AI agent (e.g., AI agent (270) of FIG. 3) can obtain information (431) of at least one external electronic device corresponding to at least one external electronic device from at least one external electronic device associated with the electronic device as context data by using an AI model (e.g., AI model (231) of FIG. 2b) (e.g., first AI model (231a) of FIG. 2b) or a second AI model (e.g., second AI model (231b) of FIG. 2b).

[0172] According to one embodiment, the electronic device may identify at least one external electronic device (431a) associated with the electronic device (201) as at least one first external electronic device (431a) that is wirelessly and / or wired (e.g., Bluetooth, Wire, and / or Wi-Fi) connected to the electronic device through a communication circuit (e.g., communication circuit (290) of FIG. 2a).

[0173] According to one embodiment, the electronic device can identify at least one external electronic device (431b) logged in with the same account as the electronic device as at least one second external electronic device associated with the electronic device.

[0174] According to one embodiment, the electronic device may identify at least one third external electronic device (431c) (e.g., IoT device) connected to the same server or the same network (e.g., SmartThings) as the electronic device as at least one external electronic device associated with the electronic device.

[0175] According to one embodiment, the electronic device can obtain information (433) of at least one external electronic device corresponding to at least one external electronic device from at least one external electronic device.

[0176] According to one embodiment, the electronic device may obtain at least some of the device type capability information (e.g., display, sensor, battery and / or network module), constraint information, or status information (e.g., display status (433c), sensor status (433d), battery status (433b) and / or network status (433a)) as information from at least one external electronic device.

[0177] According to one embodiment, the electronic device may perform an operation (435) of inferring data (information) obtainable from at least one external electronic device based on information from at least one external electronic device using an AI model, and the inference operation (435) may be displayed through the display of the electronic device (e.g., the display (260) of FIG. 2a) or processed as an internal operation without being displayed.

[0178] According to one embodiment, the electronic device can perform inference operations by using an AI model, rather than by considering all possible cases, by prioritizing the form of action that is easiest to acquire and easy to operate, and when entering a process to regenerate actions based on user feedback information after the action is executed, new data and a new set of actions can be generated and used.

[0179] Referring to FIG. 4d, according to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2a to 2b) (e.g., processor (e.g., processor (220) of FIG. 2a to 2b)), or an AI agent (e.g., AI agent (270) of FIG. 3) can obtain a first action set (451) including a first action (437) and a second action (439) corresponding to a task based on at least a portion of context data by using an AI model (e.g., AI model (231) of FIG. 2b) (e.g., second AI model (231b) of FIG. 2b).

[0180] According to one embodiment, the electronic device may generate a first action including a network state (433a) among information of at least one first external electronic device (431a) and / or at least one first external electronic device by using an AI model, a first task (413a), a second task (413b), first action data (415a), second action data (415b), a first application (417a), a second application (417b), a third application (417c) and / or a fourth application (417d).

[0181] According to one embodiment, the electronic device may generate a second action including a network state (433a) and a display state (433c) among information of a third task (413c), a fourth task (413d), third action data (415c), fourth action data (415d), a fifth application (417e), a sixth application (417f), at least one third external electronic device (431c) and / or at least one fourth external electronic device, using an AI model.

[0182] According to one embodiment, the electronic device can generate planning data (453) for a first action set (451) using an AI model.

[0183] According to one embodiment, the electronic device can generate first planning data (453a) (e.g., "selecting an appropriate schedule from a calendar, searching for an appropriate restaurant, and contacting it via a phone app to make a reservation") based on context data included in a first action using an AI model, and generate second planning data (453b) (e.g., "identifying a person through contacts and text messages, summarizing the reservation details, and sending a text message. If the other party is a family member living together at home, displaying it immediately on a TV connected to a Smart Home") based on context data included in a second action.

[0184] According to one embodiment, the electronic device may use an AI model to display planning data (453) through a display (e.g., the display of FIG. 2a) or perform the operation internally without displaying it.

[0185] According to one embodiment, the electronic device can use an AI model to display (represent) or perform internal operations on planning data (453) in natural language, coding units and / or functional units.

[0186] Referring to FIG. 4e, according to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2a to 2b) (e.g., processor (e.g., processor (220) of FIG. 2a to 2b)), or an AI agent (e.g., AI agent (270) of FIG. 3) can verify whether a first action set including a first action and a second action can be successfully executed based on context data and / or planning data (453) by using an AI model (e.g., AI model (231) of FIG. 2b) (e.g., second AI model (231b) of FIG. 2b).

[0187] According to one embodiment, an AI model (e.g., the second AI model of FIG. 2b (231b)) can perform a separate learning process or configure a separate prompt for a verification operation to determine whether the first action and the second action can be executed on a device identified as a means for executing the first action and the second action.

[0188] According to one embodiment, the electronic device can verify whether the first action set including the first action and the second action can be successfully executed by performing a simulation of the first action set including the first action and the second action based on planning data (453) using an AI model (455).

[0189] For example, the electronic device can verify that a first set of actions for a restaurant reservation can be successfully executed based on the location information of the user of the electronic device, the location information of a restaurant preferred by the user based on the location information of the user of the electronic device, the user's schedule information and / or date information available for reservation at the restaurant, using an AI model.

[0190] According to one embodiment, if the electronic device verifies that a first action set including a first action and a second action can be executed (457), it identifies a device to be used as a means to execute the first action set and can execute the first action set (459).

[0191] According to one embodiment, if there are multiple candidate devices for executing a first action and a second action, or if a device for executing a first action and a second action is not identified in a verification operation, the electronic device may identify at least one device most suitable for processing a task corresponding to a first command as at least one device capable of performing the first action and the second action based on context data and / or information of at least one external electronic device.

[0192] According to one embodiment, the electronic device can indicate that a user's request corresponding to a first command has been processed normally, and can identify at least one device among at least one external electronic device capable of performing a first action and a second action based on selection criteria that are optimal for receiving user feedback.

[0193] For example, when an action to brief the results of a restaurant reservation is executed as a user requests the electronic device’s AI Assistant regarding a restaurant reservation, if the user is wearing AI Glasses, the electronic device displays (provides) information regarding the restaurant reservation results (e.g., restaurant reservation details, restaurant location, and / or restaurant menu) through the AI ​​Glasses.

[0194] According to one embodiment, if the electronic device verifies that a first set of actions including a first action and a second action cannot be executed (457), it can obtain a second set of actions different from the first set of actions by performing the actions of FIG. 4a, FIG. 4b, FIG. 4c, and FIG. 4d again (471), and then execute the second set of actions after verifying the second set of actions in FIG. 4e.

[0195] According to one embodiment, when the electronic device confirms that all actions indicating the execution of a task according to a first command have been processed normally, it may terminate the verification operation and / or the monitoring operation due to the failure of the verification operation and no longer perform the action repeatedly.

[0196] FIGS. 5a and 5b are drawings for explaining the operation of executing a task based on context data in an electronic device according to one embodiment.

[0197] FIGS. 5a and 5b illustrate, for example, a user wearing AI glasses requesting weather information through an AI assistant of an electronic device (201) while driving a vehicle and operating a navigation connected to the vehicle.

[0198] Referring to FIG. 5a, according to one embodiment, an AI agent (Agent) (270) may include a data acquisition unit (271) and an action generation and execution unit (273).

[0199] According to one embodiment, the data acquisition unit (271) can confirm the first command "Tell me today's weather" received from the AI ​​assistant.

[0200] According to one embodiment, when the AI ​​assistant confirms a first command (501a and / or 501b) from a user requesting the execution of a task through the microphone or display of the electronic device (201), it can transmit the first command (501a and / or 501b) to the data acquisition unit (271).

[0201] According to one embodiment, an AI assistant can convert a user's first command (501a) requesting the execution of a task into text through the microphone of an electronic device (201) using a STT (Speech To Text) function.

[0202] According to one embodiment, the data acquisition unit (271) can use an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b or the second AI model (231b) of FIG. 2b) to analyze the first command and identify at least one task classified into action units as context data (511).

[0203] According to one embodiment, the data acquisition unit (271) can use an AI model to analyze the first command, "Tell me today's weather," and detect the first task (511a), "Search for current weather information," and the second task (511b), "Weather information briefing" (511).

[0204] According to one embodiment, the data acquisition unit (271) can use an AI model to identify at least one action data for executing at least one task as context data.

[0205] According to one embodiment, the data acquisition unit (271) can check "today's weather information" as first action data, "external schedule information" as second action data, and / or "travel route information" as third action data for executing "search current weather information" as first task (511a) and "weather information briefing" as second task (511b).

[0206] According to one embodiment, the data acquisition unit (271) can use an AI model to obtain information about at least one action data from an app data source (513a) stored in an electronic device (e.g., a weather app in the electronic device, a calendar app in the electronic device, and / or a navigation app in the electronic device) and can identify information corresponding to at least one application as context data (513).

[0207] According to one embodiment, the data acquisition unit (271) can use an AI model to obtain information about the first application (e.g., weather application) that is capable of obtaining information about the first action data, "today's weather information", and can check the information of the first application (513).

[0208] According to one embodiment, the data acquisition unit (271) can use an AI model to obtain information about the second application (e.g., a calendar application) that is capable of acquiring information about the second action data, "external schedule information", and can check the information of the second application (513).

[0209] According to one embodiment, the data acquisition unit (271) can use an AI model to obtain information about the third application (e.g., a navigation application) that corresponds to the third application that can acquire information about the "movement path information" of the third action data, which is today's schedule (513).

[0210] According to one embodiment, the data acquisition unit (271) can acquire data of at least one external electronic device (e.g., AI glasses connected via Bluetooth and / or a vehicle connected via Bluetooth) associated with the electronic device from a device data source (515a) using an AI model, and acquire the acquired data as context data (515).

[0211] According to one embodiment, the data acquisition unit (271) can acquire an image (e.g., driving information of an image vehicle) captured through the camera of the first external electronic device (e.g., AI glasses) among at least one external electronic device (515a) associated with the electronic device using an AI model (515).

[0212] According to one embodiment, the data acquisition unit (271) can acquire driving information stored in a navigation connected to a vehicle from a second external electronic device (e.g., a vehicle) among at least one external electronic device (515a) associated with the electronic device using an AI model (515).

[0213] According to one embodiment, the data acquisition unit (271) can acquire state information of an electronic device and / or state information of at least one external electronic device based on information of an electronic device and / or at least one external electronic device stored in a future state DB (517a) using an AI model, and acquire the acquired state information of the electronic device and / or at least one external electronic device as context data (517).

[0214] According to one embodiment, the data acquisition unit (271) can use an AI model to check the network status information (537a) of an electronic device and / or at least one external electronic device among the status information of the electronic device and / or at least one external electronic device, the status information (537b) of whether the speaker of the first external electronic device (e.g., AI glasses) is operating normally, and / or the status information (537c) of the display and / or speaker of the second external electronic device (e.g., vehicle).

[0215] According to one embodiment, the data acquisition unit (271) can acquire a first action set based on context data using an AI model.

[0216] According to one embodiment, the action generation and execution unit (273) can identify a means for executing a first action set based on information of at least one application corresponding to at least one application stored in the electronic device (201) and / or information of at least one external electronic device corresponding to at least one external electronic device associated with the electronic device, using an AI model.

[0217] According to one embodiment, the action generation and execution unit (273) can generate planning data for the first action set using an AI model (531).

[0218] According to one embodiment, the action generation and execution unit (273) can verify whether the first action set can be successfully executed based on the first planning data (531a) (e.g., "briefing weather information by voice") and the second planning data (531b) (e.g., "briefing today's entire hourly weather information by voice") using an AI model (533).

[0219] For example, the action generation and execution unit (273) can verify whether the first action set can be successfully executed by using an AI model to check whether current weather data is available, whether weather data is available by time period, whether the vehicle speaker is operating normally and / or whether the A-glass speaker is operating normally (533).

[0220] According to one embodiment, the action generation and execution unit (273) can perform a verification operation by conducting a simulation using an AI model, and if the simulation for the first action set is successful, it can verify that the first action set can be successfully executed.

[0221] According to one embodiment, the action generation and execution unit (273), upon verifying that the first action set can be executed, can identify a means (device) to execute the first action set by using a second AI model (e.g., the second AI model (231b) of FIG. 2b).

[0222] According to one embodiment, the action generation and execution unit (273) can check how the user is using each device in order to check the means (device) for executing the first action set.

[0223] According to one embodiment, the action generation and execution unit (273) can confirm that the electronic device (201) is not being used as a main device, but the electronic device (201) is currently operating as an interface with an AI assistant, and the user is using the display of a second external electronic device (e.g., a vehicle) where navigation is being executed as the main device, and the first external electronic device (e.g., AI glasses) is being used as an auxiliary device.

[0224] According to one embodiment, the action generation and execution unit (273) can execute a first set of actions verified to be executable (535).

[0225] According to one embodiment, the action generation and execution unit (273) can use an AI model to maintain navigation guidance through the display and speaker of a second external electronic device (e.g., vehicle), output voice information indicating today's weather information, external schedule and / or travel route through the speaker of a first external electronic device (e.g., AI glasses (301)), and can also display data related to today's weather information, external schedule and / or travel route on the display of the electronic device (201) (535).

[0226] For example, the action generation and execution unit (273) can use an AI model to confirm that the vehicle's speaker is being used for navigation guidance, brief weather information through the AI ​​glass's speaker, and after confirming that navigation guidance is being provided on the vehicle's display, display the weather information brought in as voice on the electronic device's display. Referring to FIG. 5b, according to one embodiment, the action generation and execution unit (273) can execute additional verification and supplementary operations of the first action set after the first action set is executed (535) (537).

[0227] According to one embodiment, the action generation and execution unit (273) can confirm that the first action set was successfully executed (537) by using an AI model after the first action set is executed (535) and by performing additional verification operations (537a, 537b) of the first action set.

[0228] For example, the action generation and execution unit (273) can confirm that the first action set has been successfully executed (537) through an additional verification operation (537a) that checks whether a voice message is transmitted to the AI ​​glasses and output to the speaker using an AI model, and an additional detection operation (537b) that checks whether a weather information screen is rendered and displayed on the display of the electronic device.

[0229] According to one embodiment, the action generation and execution unit (273) can obtain a third action set corresponding to a complementary action through a complementary action (537c, 537d) that can be additionally performed after the first action set is executed (535) by using an AI model (537).

[0230] For example, the action generation and execution unit (273) can obtain a third action set corresponding to the supplementary action through a supplementary action (537d) that checks the schedule affected by the weather on the vehicle connected to the supplementary action (537C) that checks the schedule affected by the weather on today's schedule using an AI model.

[0231] According to one embodiment, the action generation and execution unit (273) may acquire a second set of actions corresponding to complementary actions that have not been previously executed but are expected to be used or helpful, based on the acquired context data (537).

[0232] According to one embodiment, the action generation and execution unit (273) can check for information affected by weather in schedule information and information about parts affected by weather (e.g., tires) in information related to a second external electronic device (e.g., vehicle) (537).

[0233] According to one embodiment, the action generation and execution unit (273) can obtain additional information (data) from the data acquisition unit (271) when additional information is needed to obtain a second action set (538).

[0234] According to one embodiment, the action generation and execution unit (273) can generate planning data for a second action set using an AI model (539).

[0235] According to one embodiment, the action generation and execution unit (273) can verify whether a second action set can be successfully executed based on the first planning data (539a) (e.g., voice guidance regarding precautions after checking external schedules and travel routes during rainy times) and the second planning data (539b) ("using the first planning data (539a) and the first planning data (539a) and guidance regarding the drop in tire pressure due to a drop in temperature and adding it to the schedule") using an AI model (539).

[0236] According to one embodiment, the action generation and execution unit (273) can perform a verification operation by conducting a simulation using an AI model, and if the simulation for the second action set is successful, it can verify that the second action set can be successfully executed (551).

[0237] For example, the action generation and execution unit (273) can perform verification operations by simulating operations regarding whether schedule data exists, whether a vehicle's movement path is generated, the external state of the vehicle and / or whether the AI ​​glass speaker operates normally (551).

[0238] According to one embodiment, the action generation and execution unit (273) can identify a means (device) for executing a second action set.

[0239] According to one embodiment, the action generation and execution unit (273) can execute a second set of actions verified to be executable (553).

[0240] According to one embodiment, the action generation and execution unit (273) can use an AI model to maintain navigation guidance through the display and speaker of the second external electronic device (e.g., vehicle), output information regarding schedule precautions (553a) and information regarding precautions regarding tire pressure due to weather (553b) through the speaker of the first external electronic device (e.g., AI glasses (301)), and additionally display information regarding the vehicle inspection schedule (553C) through the display of the electronic device (201) (553).

[0241] FIG. 6 is a diagram illustrating the operation of executing a task based on context data in an electronic device according to one embodiment.

[0242] FIG. 6 illustrates an example in which, when a user wearing AI glasses (301) is looking at a desktop monitor (503), the AI ​​agent of the AI ​​glasses (301) acts as the main device, performs a payment operation using fingerprint authentication through an electronic device (201), and requests to purchase a product through the AI ​​assistant of the AI ​​glasses (301) while the internet application on the desktop monitor (503) is logged in with the same account as the AI ​​glasses and / or the electronic device.

[0243] Referring to FIG. 6, according to one embodiment, an AI agent (Agent) (270) may include a data acquisition unit (271) and an action generation and execution unit (273).

[0244] According to one embodiment, the data acquisition unit (271) can confirm the first command received from the AI ​​assistant, "Please buy the product I am currently viewing."

[0245] According to one embodiment, when the AI ​​assistant confirms a first command (601) from a user requesting the execution of a task through the microphone or display of the AI ​​glasses (301), it can transmit the first command (601) to the data acquisition unit (271).

[0246] According to one embodiment, the data acquisition unit (271) can use an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b or the second AI model (231b) of FIG. 2b) to analyze the first command and identify at least one task classified into action units as context data.

[0247] According to one embodiment, the data acquisition unit (271) can use an AI model to analyze the first command, "Please buy the product I am currently viewing," and detect the first task (611a), "Log in to the site," and the second task (611b), "Search for the product and make a payment" (611).

[0248] According to one embodiment, the data acquisition unit (271) can use an AI model to identify at least one action data for executing at least one task as context data.

[0249] According to one embodiment, the data acquisition unit (271) can verify the first action data, "acquiring a camera-captured image," and the second action data, "extracting product and / or site information (OCR)," for executing the first task (611a), "logging in to the site," and the second task (511b), "product search and payment."

[0250] According to one embodiment, the data acquisition unit (271) can identify information corresponding to at least one application that can acquire information about at least one action data identified from an app data source (613a) of the AI ​​glasses (301) (e.g., camera app of the AI ​​glasses and / or image analysis app within the AI ​​glasses)) using an AI model as context data (613).

[0251] According to one embodiment, the data acquisition unit (271) can use an AI model to obtain information about the second action data, "product and / or site information (OCR) extraction," and can also verify information about the first application corresponding to the first application (e.g., camera application) and information about the second application corresponding to the second application (e.g., image analysis application) (613).

[0252] According to one embodiment, the data acquisition unit (271) can use an AI model to take a picture of a desktop monitor using a first application (e.g., a camera application) and a second application (e.g., an image analysis application), and detect a product through the analysis of the captured image (613).

[0253] According to one embodiment, the data acquisition unit (271) can acquire data (e.g., site access information, site authentication information and / or site payment information) of at least one external electronic device associated with the AI ​​glasses (301) from a device data source (615a) of the AI ​​glasses (301) (e.g., payment app within the electronic device, authentication app within the electronic device and / or internet app within a taskpad logged in as the same user) using an AI model, and acquire the acquired data as context data (615).

[0254] According to one embodiment, the data acquisition unit (271) can acquire user authentication information (e.g., ID and password and / or biometric authentication information) and payment information from the device data source (615a) using the payment application and authentication application of the electronic device (201) (615).

[0255] According to one embodiment, the data acquisition unit (271) can acquire the type of data for which user input is required for product search and payment using an internet application of a first external electronic device (e.g., desktop monitor (503)) from a first external electronic device (e.g., desktop monitor (503)) logged in with the same account from a device data source (615a) using an AI model (615).

[0256] According to one embodiment, the data acquisition unit (271) can acquire status information of at least one external electronic device from the device status DB (617a) using an AI model, and acquire the acquired status information of at least one external electronic device as context data (617).

[0257] For example, the data acquisition unit (271) can acquire the status information of at least one external electronic device, whether payment is possible for the electronic device and / or whether the AI ​​glass speaker is in operation, by using an AI model. According to one embodiment, the data acquisition unit (271) can acquire a first action set based on context data by using an AI model.

[0258] According to one embodiment, the action generation and execution unit (273) can identify a means for executing a first action set based on information of at least one application corresponding to at least one application stored in the AI ​​glasses (301) and / or information of at least one external electronic device corresponding to at least one external electronic device associated with the AI ​​glasses (301), using an AI model.

[0259] According to one embodiment, the action generation and execution unit (273) can use an AI model to generate first planning data (631a) (e.g., "Log in to the site and pay for the product") and second planning data (631b) (e.g., "Purchase history and delivery information briefing") planning data for a first action set (631).

[0260] According to one embodiment, the action generation and execution unit (273) can verify whether the first action set can be successfully executed based on the first planning data (631a) (e.g., "Log in to the site and pay for the product") and the second planning data (631b) (e.g., purchase history and delivery information briefing") using an AI model (633).

[0261] According to one embodiment, the generation and execution unit (273) performs a verification operation by simulating, using an AI model, the operation of logging into a site and paying for a product, the operation of user authentication and / or payment operating normally, and the operation of purchasing a product on the site, based on the first planning data (631a) and the second planning data (631b), and if the simulation for the first action set is successful, it can be verified that the first action set can be successfully executed (633).

[0262] For example, the generation and execution unit (273) can perform a verification operation by simulating an operation regarding whether there is login data, whether there is product information data, whether there is user authentication data and / or whether a product can be purchased on the site, based on a first planning data (631a) such as "log in to the site and make payment for the product" and a second planning data (631b) such as "purchase history and delivery information briefing" (633).

[0263] According to one embodiment, the action generation and execution unit (273), upon verifying that the first action set can be executed, can identify a means (device) to execute the first action set by using a second AI model (e.g., the second AI model (231b) of FIG. 2b).

[0264] According to one embodiment, the action generation and execution unit (273) can execute a first set of actions verified to be executable (635).

[0265] For example, the action generation and execution unit (273) can, when it verifies that the first set of actions can be successfully executed, obtain authentication and / or payment information from the electronic device and provide voice guidance through AI Glasses, execute actions related to purchasing goods through an internet application on a desktop monitor logged in with the same account, display a payment screen through the desktop monitor, and if login or payment is required, proceed with fingerprint authentication through the electronic device (635).

[0266] FIG. 7 is a diagram illustrating the operation of executing a task based on context data in an electronic device according to one embodiment.

[0267] Referring to FIG. 7, according to one embodiment, the first AI agent (270) of the electronic device (201) can confirm the first command (701a, and / or 701b) received from the AI ​​assistant of the electronic device, “book a date restaurant and send a text message.”

[0268] According to one embodiment, the first AI agent (270) can use an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b or the second AI model (231b) of FIG. 2b) to analyze the first command and identify at least one task classified into action units as context data (711).

[0269] According to one embodiment, the first AI agent (270) can use an AI model to identify at least one action data for executing at least one task as context data.

[0270] According to one embodiment, the first AI agent (270) can identify information corresponding to at least one application that can obtain information about at least one action data based on an app data source (713a) stored in an electronic device and / or an external electronic device associated with the electronic device using an AI model (713).

[0271] According to one embodiment, the first AI agent (270) can use an AI model to obtain data of at least one external electronic device related to the electronic device from a device data source (715a) and obtain the obtained data as context data (715).

[0272] According to one embodiment, the first AI agent (270) can obtain status information of at least one external electronic device from the device status DB (717a) using an AI model, and obtain the obtained status information of at least one external electronic device as context data (717).

[0273] According to one embodiment, if the first AI agent (270) determines that it cannot clearly execute a verification operation for the first action set based on data obtained from the device data source (717a) and status information of at least one external electronic device obtained from the device status DB (717a), it may request additional data for a verification operation for the first action set from the second AI agent (370) running in the first application of the electronic device and the third AI agent (470) of the external electronic device connected to the electronic device.

[0274] According to one embodiment, the second AI agent (370) can verify data requested by the first AI agent (270), detect data information that can be generated in the first application (731), create and execute an action to obtain data corresponding to the detected data information (733), obtain additional data for a verification operation for the first action set, and transmit the obtained additional data to the first AI agent (270).

[0275] According to one embodiment, the third AI agent (470) can verify data requested by the first AI agent (270), detect data information that can be generated from an external electronic device (751), generate and execute an action to obtain data corresponding to the detected data information (753), obtain additional data for a verification action for the first action set, and transmit the obtained additional data to the first AI agent (270).

[0276] According to one embodiment, the first AI agent (270) can obtain additional data transmitted from the second AI agent (370) and / or the third AI agent (470)719).

[0277] According to one embodiment, the first AI agent (270) can execute a verification operation for the first action set based on data obtained from at least one external electronic device (715a), information (717a) of at least one external electronic device, and data obtained from the second AI agent (370) and / or the third AI agent (470).

[0278] According to one embodiment, the first AI agent (270) can communicate with the second AI agent (370) and / or the third AI agent (470) using a communication protocol such as PC (Inter Process Communication) and / or HTTP.

[0279] According to one embodiment, the first AI agent (270) may request a verification action for the first action set from the second AI agent (370) and / or the third AI agent (470).

[0280] FIG. 8 is a diagram illustrating the operation of training an AI agent in an electronic device according to one embodiment.

[0281] According to one embodiment, the AI ​​agent may include a data acquisition unit, an action generation and execution unit (273), and an action feedback generation unit (275).

[0282] Referring to FIG. 8, according to one embodiment, the action generation and execution unit (273) can generate planning data for a first action set using an AI model (e.g., the AI ​​model (123) of FIG. 2b (e.g., the second AI model (231b) of FIG. 2b)) (811), verify the execution of the first action set (813), execute the first action set after verifying that the first action set can be executed (815), and perform additional verification and supplementary operations for the execution of the first action set after executing the first action set (817).

[0283] According to one embodiment, the action feedback generating unit (275) can collect user feedback on the set of executed actions (831).

[0284] For example, the action feedback generation unit (275) can verify subsequent actions, such as searching for and reserving a restaurant through the executed action based on the user's first command for task execution, and additionally saving the schedule reserved by the user to the calendar application, as user feedback data.

[0285] According to one embodiment, the action feedback generation unit (275) can assign weights to subsequent actions corresponding to the executed action and user feedback (833).

[0286] For example, the action feedback generation unit (275) may assign the same weight to the first action set or assign a higher weight than the first action set if, after the first action set is executed, the user performs a first action (e.g., screen capture) related to the execution of the first action set within a specified time.

[0287] For example, the action feedback generation unit (275) can assign a lower weight to the second action than to the first action if, after the first action set is executed, the user performs a second action that is not related to the execution of the first action set and then performs a first action that is not related to the execution of the first action set.

[0288] For example, the action feedback generation unit (275) can extract a user pattern after the first action set is executed and assign a weight to the action after the first action set is executed based on the user's pattern.

[0289] According to one embodiment, the action feedback generation unit (275) can configure a prompt for the second AI model (835).

[0290] According to one embodiment, the action feedback generation unit (275) can generate a system context so that the second AI model (e.g., the second AI model (231b) of FIG. 2b) can reference it when generating planning data, and can provide a prompt containing the system context to the second AI model.

[0291] According to one embodiment, the action feedback generation unit (275) can generate a system context so that the second AI model can understand the user's behavioral patterns and generate customized planning data, and can include a guide for actions that the user considers important in the system context.

[0292] According to one embodiment, the action feedback generation unit (275) can update the second AI model based on user feedback information.

[0293] FIGS. 9a and 9b are drawings for explaining the operation of training an AI agent in an electronic device according to one embodiment.

[0294] According to one embodiment, the AI ​​agent may include a data acquisition unit, an action generation and execution unit (273), an action feedback generation unit (275), and a user pattern reinforcement learning unit (277).

[0295] Referring to FIG. 9a, according to one embodiment, the action generation and execution unit (273) can generate planning data for a first action set using an AI model (e.g., the AI ​​model (123) of FIG. 2b (e.g., the second AI model (231b) of FIG. 2b)) (911), verify the execution of the first action set (913), execute the first action set after verifying that the first action set can be executed (915), and perform additional verification and supplementary operations for the execution of the first action set after executing the first action set (917).

[0296] According to one embodiment, the action feedback generating unit (275) can collect user feedback on the set of executed actions (931).

[0297] For example, the action feedback generation unit (275) can verify subsequent actions, such as searching for and reserving a restaurant through the executed action based on the user's first command for task execution, and additionally saving the schedule reserved by the user to the calendar application, as user feedback data.

[0298] According to one embodiment, the action feedback generation unit (275) can assign weights to subsequent actions corresponding to the executed action and user feedback (933).

[0299] For example, the action feedback generation unit (275) may assign the same weight to the first action set or assign a higher weight than the first action set if, after the first action set is executed, the user performs a first action (e.g., screen capture) related to the execution of the first action set within a specified time.

[0300] For example, the action feedback generation unit (275) can assign a lower weight to the second action than to the first action if, after the first action set is executed, the user performs a second action that is not related to the execution of the first action set and then performs a first action that is not related to the execution of the first action set.

[0301] For example, the action feedback generation unit (275) can extract a user pattern after the first action set is executed and assign weights to actions after the first action set is executed based on the user's pattern.

[0302] According to one embodiment, the user pattern reinforcement learning unit (277) can perform reinforcement learning that can reinforce the user's preference based on the user pattern (951).

[0303] According to one embodiment, the user pattern reinforcement learning unit (277) can learn the user's action chaining through reinforcement learning.

[0304] According to one embodiment, chaining refers to a set of actions that are executed in succession, and this can be a set of actions that reflects the user's pattern.

[0305] According to one embodiment, the user pattern reinforcement learning unit (277) can configure a prompt for the second AI model (953).

[0306] According to one embodiment, the user pattern reinforcement learning unit (277) can generate a system context so that the second AI model (e.g., the second AI model (231b) of FIG. 2b) can refer to it when generating planning data, and can provide the second AI model with a prompt containing the system context.

[0307] According to one embodiment, the user pattern reinforcement learning unit (277) can generate a system context so that the second AI model can understand the user's behavioral patterns and generate customized planning data, and can include a guide for actions that the user considers important in the system context.

[0308] According to one embodiment, the user pattern reinforcement learning unit (277) may include system context, user utterance and / or action chaining in the prompt.

[0309] According to one embodiment, the action feedback generation unit (275) can update the second AI model based on user preference information.

[0310] Referring to FIG. 9b, reinforcement learning is described as follows: when an AI agent (270) decides on an action, the environment (970), which is the space where the AI ​​agent (270) operates, can provide a reward for it and update the state. The environment (970) can grant a positive reward to actions with high weight in the user's pattern and a negative reward to actions with low weight. The AI ​​agent (270) proceeds with learning based on these rewards, thereby enabling the second AI model to continuously learn the user's pattern. Subsequently, the learned model can generate prompts including system context, user utterances, and / or action chaining to support the second AI model in learning.

[0311] An electronic device according to one embodiment (101 of FIG. 1; 201 of FIG. 2a and 2b)) may include a communication circuit (190 of FIG. 1; 290 of FIG. 2a), at least one processor (120 of FIG. 1; 220 of FIG. 2a), and a memory for storing instructions (130 of FIG. 1; 230 of FIG. 2a). When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may obtain a request for a task. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may obtain context data related to the task. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may obtain a first set of actions including a first action and a second action corresponding to the task based on at least a portion of the context data. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may execute the first set of actions based at least a portion of the fact that each of the first action and the second action is executable. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to execute a second set of actions including a third action instead of the at least one action based at least a portion of the fact that at least one of the first action or the second action cannot be executed successfully.

[0312] According to one embodiment, the instructions may be configured such that, when executed individually or collectively by the at least one processor, the electronic device sets a first application or a first device as at least part of the means for executing the first action, and sets a second application different from the first application or a second device different from the first device as at least part of the means for executing the second action.

[0313] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may obtain, as at least part of the context data, information of a first application corresponding to the first application, information of a first device corresponding to the first device, information of a second application corresponding to the second application, or information of a second device corresponding to the second device. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may be configured to determine whether the first action or the second action can be successfully executed based on at least part of the information of the first application, the information of the first device, the information of the second application, or the information of the second device.

[0314] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may acquire the first capability information, the first constraint information, or the first state information of the first device as at least part of the first device information, and acquire the second capability information, the second constraint information, or the second state information of the second device as at least part of the second device information. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may be configured to perform an operation of determining whether the first action or the second action is successfully executable based on at least part of the first capability information, the first constraint information, the first state information, the second capability information, the second constraint information, or the second state information.

[0315] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may correspond, respectively, to the first device and the second device, to different first external device and second external device functionally connected to the electronic device.

[0316] The instructions according to one embodiment may be configured to allow the electronic device, when executed individually or collectively by the at least one processor, to determine whether the first action or the second action can be successfully executed, at least in part, based on whether the first data or the second data to be used by the first application or the second application in association with the task, respectively, is available.

[0317] The instructions according to one embodiment may be configured to set, respectively, an application or device corresponding to the at least one action and a different application or device, as at least part of the means for executing the third action, when executed individually or collectively by the at least one processor.

[0318] The instructions according to one embodiment may be configured such that, when executed individually or collectively by the at least one processor, the electronic device obtains the first action or the second action based on at least a portion of the first context data among the context data, and obtains the third action based on at least a portion of the second context data among the context data that is at least partially different from the first context data.

[0319] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may obtain additional data related to the task, generated through an external electronic device wirelessly connected to the electronic device via the communication circuit, as at least part of the context data. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may be configured to obtain the third action based on at least part of the additional data.

[0320] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may be configured to provide an indication to the user that the at least one action has been replaced by the third action.

[0321] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may perform the execution of the second set of actions based further on the fact that the third action can be successfully executed. According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to perform a fourth action different from the third action as at least part of the second set of actions instead of the third action, or as at least part of a set of third actions different from the second set of actions, based further on the fact that the third action cannot be successfully executed.

[0322] The instructions according to one embodiment may be configured to transmit corresponding control information to the external device so that the third action is executed by the external device as at least part of the execution operation of the second action set, when executed individually or collectively by the at least one processor, based at least partly on the fact that the external device wirelessly connected to the electronic device through the communication circuit is set as a means for executing the third action.

[0323] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may obtain feedback information regarding the result of the action to be executed of the second action set. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may obtain additional context data other than the context data based at least partially on the feedback information. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may generate a fourth action set including a fourth action different from the actions included in the second action set, based further on at least partially on the additional context data. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may be configured to execute the fourth action set as at least partially in response to a request for the task.

[0324] The instructions according to one embodiment may be configured such that, when executed individually or collectively by the at least one processor, the electronic device acquires the second action set including the third action using a large action model.

[0325] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may obtain feedback information regarding the result of the operation to be executed of the second action set. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may be configured to provide the at least some information of the feedback information to the large action model so that at least some information of the feedback information is used to update the large action model.

[0326] When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may generate preference information indicating a user's preference based on the feedback information. When the instructions according to one embodiment are executed individually or collectively by the at least one processor, the electronic device may be configured to provide the preference information to the large action model so that the preference information can be used to update the large action model.

[0327] FIG. 10 is a flowchart illustrating the operation of executing a task based on context data in an electronic device according to one embodiment. The operations for executing the task may include operations 1001 through 1009. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, at least two operations may be performed in parallel, or other operations may be added.

[0328] In operation 1001, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can obtain a request for a task.

[0329] According to one embodiment, the electronic device can recognize a first command requesting the execution of a task.

[0330] According to one embodiment, the electronic device can identify a user's voice command input through the microphone of the electronic device or a user's text command input through the display of the electronic device as a first command.

[0331] In operation 1003, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can obtain context data related to the task.

[0332] According to one embodiment, the electronic device can identify (obtain) at least one task classified into action units by analyzing a first command using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b)) as context data.

[0333] According to one embodiment, the electronic device can identify (obtain) at least one action data for executing at least one task as context data by using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b).

[0334] According to one embodiment, the electronic device can identify (obtain) information corresponding to at least one application that can obtain information about action data of at least one of at least one application stored in the electronic device by using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b)).

[0335] According to one embodiment, the processor (220) can identify (obtain) information of at least one external electronic device corresponding to at least one external electronic device from at least one external electronic device associated with the electronic device as context data by using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b).

[0336] According to one embodiment, the electronic device can identify (obtain) information of the electronic device as context data by using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b).

[0337] In operation 1005, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) may obtain a first set of actions including a first action and a second action corresponding to a task based on at least some of the context data.

[0338] According to one embodiment, the electronic device can generate (obtain) a first action set including a first action and a second action corresponding to a task based on at least a portion of context data by using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the second AI model (231b) of FIG. 2b).

[0339] According to one embodiment, the electronic device can generate (acquire) a first action including a first task among at least one task, first action data among at least one action data, information of a first application among at least one application, information of a first external electronic device among at least one external electronic device, and / or information of the electronic device.

[0340] According to one embodiment, the electronic device can generate (acquire) a second action including a second task among at least one task, second action data among at least one action data, information of a second application among at least one application, information of a second external electronic device among at least one external electronic device and / or information of the electronic device.

[0341] In operation 1007, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) may execute a first set of actions based at least in part on the fact that each of the first action and the second action can be successfully executed.

[0342] According to one embodiment, if the electronic device confirms that the first action and the second action can be successfully executed by using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the second AI model (231b) of FIG. 2b)), it identifies a device for executing the first action and the second action among the electronic device and / or at least one external electronic device, and can execute the first action and the second action through the identified device.

[0343] In operation 1009, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) may execute a second set of actions including a third action instead of at least one action, based at least in part on the fact that the first action or at least one of the second actions cannot be successfully executed.

[0344] According to one embodiment, the electronic device can obtain a second action set containing a third action instead of at least one action that cannot be executed (e.g., the first action or the second action) by using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the second AI model (231b) of FIG. 2b)) when verifying that at least one action (e.g., the first action and / or the second action) among the first action and the second action included in the first action set cannot be successfully executed.

[0345] According to one embodiment, the electronic device may acquire context data for acquiring a third action, generate planning data for a second action set including the third action, verify the second action set including the third action, and then execute the second action set.

[0346] FIG. 11 is a flowchart illustrating the operation of executing a task based on context data in an electronic device according to one embodiment. The operations for executing the task may include operations 1101 through 1123. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, at least two operations may be performed in parallel, or other operations may be added.

[0347] In operation 1101, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can recognize a first command requesting the execution of a task.

[0348] According to one embodiment, the electronic device can identify a user's voice command input through the microphone of the electronic device or a user's text command input through the display of the electronic device as a first command.

[0349] In operation 1103, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can analyze the first instruction and identify at least one task separated into action units as context data.

[0350] According to one embodiment, the electronic device can detect at least one task classified into action units by analyzing a first command using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b)).

[0351] In operation 1105, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can identify at least one action data for executing at least one task as context data.

[0352] According to one embodiment, the electronic device can identify at least one action data for executing at least one task as context data by using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b).

[0353] In operation 1107, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can identify information corresponding to at least one application as context data.

[0354] According to one embodiment, the electronic device can identify information corresponding to at least one application that can obtain information about action data of at least one of at least one application stored in the electronic device using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b)) as context data.

[0355] In operation 1109, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can check information from at least one external electronic device as context data.

[0356] According to one embodiment, the processor (220) can obtain information of at least one external electronic device corresponding to at least one external electronic device from at least one external electronic device associated with the electronic device as context data by using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b).

[0357] According to one embodiment, the electronic device can identify at least one external electronic device connected wirelessly and / or wired to the electronic device (201) through a communication circuit (e.g., the communication circuit (290) of FIG. 2a) as at least one external electronic device associated with the electronic device.

[0358] According to one embodiment, the electronic device can identify at least one external electronic device logged in with the same account as the electronic device as at least one external electronic device associated with the electronic device.

[0359] According to one embodiment, the electronic device can identify at least one external electronic device (e.g., an IoT device) connected to the same network as the electronic device as at least one external electronic device associated with the electronic device.

[0360] According to one embodiment, the electronic device can obtain at least some of the device type, capability information, constraint information, or status information from at least one external electronic device as information of at least one external electronic device.

[0361] According to one embodiment, the electronic device can obtain information of the electronic device as context data by using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the first AI model (231a) of FIG. 2b).

[0362] In operation 1111, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) may obtain a first set of actions including a first action and a second action corresponding to a task based on at least a portion of the context data.

[0363] According to one embodiment, the electronic device may obtain a first set of actions including a first action and a second action corresponding to a task based on at least a portion of context data by using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the second AI model (231b) of FIG. 2b).

[0364] According to one embodiment, the electronic device may generate a first action including a first task among at least one task, first action data among at least one action data, information of a first application among at least one application, information of a first external electronic device among at least one external electronic device, and / or information of the electronic device.

[0365] According to one embodiment, the electronic device may generate a second action comprising a second task among at least one task, second action data among at least one action data, information of a second application among at least one application, information of a second external electronic device among at least one external electronic device and / or information of the electronic device.

[0366] In operation 1113, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can generate first planning data for a first action and second planning data for a second action.

[0367] According to one embodiment, the electronic device may generate first planning data for performing a first action based on an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the second AI model (e.g., the second AI model (231b) of FIG. 2b)) among at least one task included in the first action, the first action data among at least one action data, information of the first application among at least one application, information of the first external electronic device among at least one external electronic device, and / or information of the electronic device.

[0368] According to one embodiment, the electronic device may generate second planning data for performing a second action based on an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the second AI model (231b) of FIG. 2b)), a second task among at least one task included in the second action, second action data among at least one action data, information of a second application among at least one application, information of a second external electronic device among at least one external electronic device, and / or information of the electronic device.

[0369] In operation 1115, an electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can verify whether the first action and the second action can be successfully executed based on the first planning data and the second planning data.

[0370] According to one embodiment, the electronic device can verify whether a first action and a second action can be successfully executed based on context data and / or first planning data and second planning data by using an AI model (e.g., AI model (231) of FIG. 2b) (e.g., second AI model (231b) of FIG. 2b).

[0371] According to one embodiment, the electronic device can verify whether the first action and the second action can be successfully executed by performing a simulation of the first action and the second action based on the first planning data and the second planning data using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the second AI model (231b) of FIG. 2b).

[0372] In operation 1117, the electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) can execute a first set of actions including the first action and the second action in operation 1119, provided that the first action and the second action can be successfully executed.

[0373] According to one embodiment, if the electronic device confirms through verification that the first action and the second action can be successfully executed, it identifies a device among the electronic device and / or at least one external electronic device for executing the first action and the second action, and can execute the first action and the second action through the identified device.

[0374] In operation 1121, the electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) may perform additional verification operations and supplementary operations.

[0375] According to one embodiment, after executing a first set of actions including a first action and a second action, the processor (220) may perform additional verification of whether the first action and the second action were successfully executed using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the second AI model (231b) of FIG. 2b).

[0376] According to one embodiment, after executing a first action set including a first action and a second action, the processor (220) may execute a third action set including a fourth action (complementary action) different from the first action and the second action included in the first action set based on user feedback information regarding the result of executing the first action set, using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the second AI model (231b) of FIG. 2b).

[0377] In operation 1117, if the electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (201) of FIG. 2a to 2b) verifies that the first action and the second action cannot be successfully executed, in operation 1123, the second action set including the third action can be obtained.

[0378] According to one embodiment, the electronic device can obtain a second action set containing a third action instead of at least one action that cannot be executed (e.g., the first action or the second action) by using an AI model (e.g., the AI ​​model (231) of FIG. 2b) (e.g., the second AI model (231b) of FIG. 2b)) when verifying that at least one action (e.g., the first action and / or the second action) among the first action and the second action included in the first action set cannot be successfully executed.

[0379] According to one embodiment, the electronic device may obtain context data (e.g., second context data) for obtaining a third action through operations 1107 to 1115, generate planning data for a second action set including the third action, verify the second action set including the third action, and then execute the second action set.

[0380] A method for executing a task based on context in an electronic device (101 in FIG. 1; 201 in FIG. 2a and 2b) according to one embodiment may include an operation of obtaining a request for the task. According to one embodiment, the method may include an operation of obtaining context data related to the task. According to one embodiment, the method may include an operation of obtaining a first action set including a first action and a second action corresponding to the task based on at least a portion of the context data. According to one embodiment, the method may include an operation of executing the first action set based at least a portion of the fact that each of the first action and the second action is executable. According to one embodiment, the method may include an operation of executing a second action set including a third action instead of the at least one action based at least a portion of the fact that at least one of the first action or the second action cannot be successfully executed.

[0381] The method according to one embodiment may include an operation of setting a first application or a first device as at least part of the means for executing the first action. The method according to one embodiment may further include an operation of setting a second application different from the first application or a second device different from the first device as at least part of the means for executing the second action.

[0382] The method according to one embodiment may include an operation of obtaining, as at least a part of the context data, information of a first application corresponding to the first application, information of a first device corresponding to the first device, information of a second application corresponding to the second application, or information of a second device corresponding to the second device. The method according to one embodiment may further include an operation of determining whether the first action or the second action can be successfully executed based on at least a part of the information of the first application, the information of the first device, the information of the second application, or the information of the second device.

[0383] The electronic device according to one embodiment disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic device according to the embodiment of this document is not limited to the aforementioned devices.

[0384] One embodiment of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, each of phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationally,” it means that said component may be connected to said other component directly (e.g., wired), wirelessly, or through a third component.

[0385] The term "module" as used in an embodiment of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to an embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0386] One embodiment of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101) or electronic device (301)). For example, a processor (e.g., processor (520)) of the machine (e.g., electronic device (301)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0387] According to one embodiment, the method according to one embodiment disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0388] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (201 of FIG. 1; FIG. 2a to 2b), Communication circuit (190 in FIG. 1; 290 in FIG. 2a); At least one processor (120 of FIG. 1; 220 of FIG. 2a to 2b); and It includes memory for storing instructions (130 in FIG. 1; 230 in FIG. 2), and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Obtain a request for a task; Obtain context data related to the above task; Based on at least a portion of the above context data, a first action set including a first action and a second action corresponding to the task is obtained; Execute the first set of actions based at least in part on the fact that each of the first action and the second action is successfully executable; and An electronic device configured to execute a second set of actions including a third action instead of the at least one action, based at least in part on the fact that at least one of the first action or the second action cannot be successfully executed.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, A first application or a first device is set as at least part of the means for executing the first action, and a second application different from the first application or a second device different from the first device is set as at least part of the means for executing the second action, Determining whether the first action or the second action can be successfully executed, at least in part based on whether the first data or the second data to be used by the first application or the second application in association with the above task is available, and An electronic device configured to set, respectively, an application or device corresponding to the at least one action and a different application or device as at least part of the means for executing the third action.

3. In Paragraph 1 or 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, As at least part of the above context data, obtaining information of a first application corresponding to the first application, information of a first device corresponding to the first device, information of a second application corresponding to the second application, or information of a second device corresponding to the second device; and An electronic device configured to determine whether the first action or the second action can be successfully executed based on at least part of the information of the first application, the information of the first device, the information of the second application, or the information of the second device.

4. In any one of paragraphs 1 through 3, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Acquiring first capability information, first constraint information, or first state information of the first device as at least part of the first device information, and acquiring second capability information, second constraint information, or second state information of the second device as at least part of the second device information; and It is configured to perform an operation to determine whether the first action or the second action is successfully executable based on at least a portion of the first capability information, the first constraint information, the first state information, the second capability information, the second constraint information, or the second state information, and The first device and the second device are electronic devices corresponding, respectively, to different first external devices and second external devices functionally connected to the electronic device.

5. In any one of paragraphs 1 through 4, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device configured to obtain the first action or the second action based on at least a portion of the first context data among the context data, and to obtain the third action based on at least a portion of the second context data among the context data that is at least partially different from the first context data.

6. In any one of paragraphs 1 through 5, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Acquiring additional data related to the task, generated through an external electronic device wirelessly connected to the electronic device and the communication circuit as at least part of the context data; and An electronic device configured to acquire the third action based on at least part of the additional data.

7. In any one of paragraphs 1 through 6, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device configured to provide a user with an indication that at least one action has been replaced by the third action.

8. In any one of paragraphs 1 through 7, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Performing the execution of the second set of actions based further on the fact that the third action can be successfully executed; and An electronic device configured to execute a fourth action different from the third action as at least part of the second set of actions, or as at least part of the third set of actions different from the second set of actions, instead of the third action, based further on the fact that the third action cannot be successfully executed.

9. In any one of paragraphs 1 through 8, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Based at least in part that an external device wirelessly connected to the electronic device and the communication circuit is set as a means for executing the third action, An electronic device configured to transmit corresponding control information to an external device so that the third action is executed at least as part of the operation to be executed of the second action set.

10. In any one of paragraphs 1 through 9, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Obtaining feedback information regarding the result of the action executed in the second action set; Obtain additional context data other than the context data based on at least part of the above feedback information; Generating a fourth action set including a fourth action different from the actions included in the second action set, based further on at least some of the additional context data; and An electronic device configured to execute the fourth set of actions as at least part of the response to a request for the above task.

11. In any one of paragraphs 1 through 10, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device configured to obtain the second action set including the third action using a large action model.

12. In any one of paragraphs 1 through 11, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Obtaining feedback information regarding the result of the action executed in the second action set; and An electronic device configured to provide at least some of the above feedback information to the large action model so that at least some of the information is used to update the large action model.

13. In any one of paragraphs 1 through 12, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Based on the above feedback information, preference information representing the user's preference is generated, and An electronic device configured to provide the preference information to the large action model so that the preference information can be used to update the large action model.

14. A method for executing a task based on context data in an electronic device (101 of FIG. 1; 201 of FIG. 2a to 2b), Action of obtaining a request for a task; An operation to obtain context data related to the above task; An operation to obtain a first action set including a first action and a second action corresponding to the task, based on at least a portion of the context data; An operation to execute the first set of actions based at least in part on the fact that each of the first action and the second action is successfully executable; and A method comprising executing a second set of actions including a third action instead of the at least one action, based at least in part on the fact that at least one of the first action or the second action cannot be successfully executed.

15. In a non-volatile storage medium storing instructions, the instructions are configured to cause the wearable electronic device to perform at least one operation when executed by the wearable electronic device, wherein the at least one operation is, Action of obtaining a request for a task; An operation to obtain context data related to the above task; An operation to obtain a first action set including a first action and a second action corresponding to the task, based on at least a portion of the context data; An operation to execute the first set of actions based at least in part on the fact that each of the first action and the second action is successfully executable; and A storage medium comprising an operation of executing a second set of actions including a third action instead of the at least one action, based at least in part on the fact that at least one of the first action or the second action cannot be successfully executed.

Images