Electronic device and method for operating artificial intelligence assistant by same

Abstract

An electronic device according to an embodiment may comprise a display, a communication module, a connector, a processor, and a memory for storing instructions executable by the processor, wherein the instructions cause the electronic device to: receive an input signal for calling an AI assistant from an external electronic device connected to the communication circuit or the connector in a state in which an app execution screen of an application is displayed on the display; execute the AI assistant on the basis of the received input signal to display the app execution screen in a first area of the display; display an AI execution screen related to the AI assistant in a second area of the display; display information related to a user query input on the AI execution screen on the basis of the reception of the user query input; and generate an answer to the user query input through the AI assistant on the basis of at least some of the user query input, information related to an application being executed by the electronic device, system-related information, and personalization-related information, and display the answer on the AI execution screen.

Description

Method of operating an electronic device and its artificial intelligence assistant

[0001] The present disclosure relates to an electronic device and a method for operating an artificial intelligence assistant thereof.

[0002] With the advancement of artificial intelligence technology, there is a trend of introducing AI (artificial intelligence) functions into applications of electronic devices. AI assistants understand the user's input intent and support the output of responses or the execution of functions corresponding to that intent.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] In order for a user to receive assistance from AI functions while performing a task using an electronic device (e.g., launching an application), a series of steps are required, including interrupting the currently running task, locating the AI ​​Assistant application, and launching it. For instance, a user might be using a specific application, then switch screens to bring up the application list, launch the AI ​​Assistant application (hereinafter referred to as AI Assistant) from the list, and finally input a query to obtain the desired result.

[0005] However, when switching screens, the application currently in operation is paused, and during this pause, there are cases where the electronic device fails to fully save the previous state of operation. Furthermore, when using an AI assistant, if the user's query is not specific or the AI ​​assistant misunderstands the user's intent and provides an answer, the user is required to go through an additional querying process to obtain an answer within the desired range.

[0006] Various embodiments of the present disclosure aim to propose a method, apparatus, and recording medium for an electronic device to execute an AI assistant using an external electronic device during the execution of an application and to utilize the executed AI assistant.

[0007] The problems to be solved in this disclosure are not limited to those mentioned above, and may be extended in various ways without departing from the spirit and scope of this disclosure.

[0008] An electronic device according to one embodiment may include a display.

[0009] An electronic device according to one embodiment may include a communication module comprising at least one communication circuit.

[0010] An electronic device according to one embodiment may include a connector.

[0011] An electronic device according to one embodiment may include at least one processor including a processing circuitry.

[0012] An electronic device according to one embodiment may include a memory that stores executable instructions that can be executed by the at least one processor.

[0013] According to one embodiment, when the instructions are executed individually or collectively by the processor, the electronic device may receive an input signal calling an AI (artificial intelligence) assistant from an external electronic device connected to the communication circuit or the connector while the application execution screen is displayed on the display.

[0014] Commands according to one embodiment may cause the electronic device to execute an AI assistant based on the received input signal to display the app execution screen in a first area of ​​the display and display an AI execution screen related to the AI ​​assistant in a second area of ​​the display.

[0015] Commands according to one embodiment can cause the electronic device to display information related to the user query input on an AI execution screen based on the reception of the user query input.

[0016] Commands according to one embodiment may enable the electronic device to generate an answer to the user query input through an AI assistant and display it on the AI ​​execution screen, based on at least some of the user query input, information related to an application running on the electronic device, information related to a system, or information related to personalization.

[0017] According to one embodiment, the AI ​​execution screen may include at least one recommendation AI function executable via the AI ​​assistant in relation to the application or app execution screen displayed in the first area, and key input information of the external electronic device executing the recommendation AI function.

[0018] A method for operating an artificial intelligence assistant of an electronic device using an external electronic device according to one embodiment may include the operation of displaying the application execution screen on a display.

[0019] A method according to one embodiment may include the operation of receiving an input signal to call an AI (artificial intelligence) assistant from an external electronic device while the app execution screen is displayed.

[0020] A method according to one embodiment may include the operation of executing an AI assistant based on the received input signal to display the app execution screen in a first area of ​​the display and display an AI execution screen related to the AI ​​assistant in a second area of ​​the display.

[0021] A method according to one embodiment may include an operation of displaying information related to a user query input on an AI execution screen based on the reception of a user query input.

[0022] A method according to one embodiment may include an operation of generating an answer to a user query input through an AI assistant based on at least some of the user query input, information related to an application running on an electronic device, information related to a system, or information related to personalization.

[0023] A method according to one embodiment may include an operation of displaying the generated answer on the AI ​​execution screen.

[0024] The above AI execution screen may include at least one recommendation AI function executable via the AI ​​assistant in relation to the application or app execution screen displayed in the first area, and key input information of the external electronic device for executing the recommendation AI function.

[0025] According to one embodiment of the present disclosure, a non-transitory computer-readable medium storing instructions that cause the electronic device to perform operations when executed by a processor of the electronic device, wherein the operations may include: an operation of displaying an application execution screen on a display; an operation of receiving an input signal calling an artificial intelligence (AI) assistant from an external electronic device while the application execution screen is displayed; an operation of executing the AI ​​assistant based on the received input signal to display the application execution screen in a first area of ​​the display and display an AI execution screen related to the AI ​​assistant in a second area of ​​the display; an operation of displaying information related to the user query input on the AI ​​execution screen based on the reception of the user query input; an operation of generating an answer to the user query input through the AI ​​assistant based on at least some of the user query input, information related to the application running on the electronic device, system information, or personalization information; and an operation of displaying the generated answer on the AI ​​execution screen.

[0026] Various embodiments can execute the AI ​​assistant of an electronic device through input using an external electronic device connected to the electronic device.

[0027] Various embodiments allow an electronic device to improve the productivity of applications by dividing the display area based on input received from an external electronic device during application execution and simultaneously running an AI assistant without interrupting the running application.

[0028] Various embodiments provide the AI ​​assistant with an analysis of an execution screen or an application displayed on a display, along with context related to at least one application running on an electronic device, context related to a system, and personalized user context, when the AI ​​assistant is executed, thereby providing an answer to a user's query input without additional procedures for ambiguous and vague user queries that can be interpreted in various ways.

[0029] Various embodiments may suggest recommendation AI functions that can be supported via AI in relation to a currently running application or a screen displayed on a current display, and provide the user with an input operation guide for executing recommendation AI functions.

[0030] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0031] FIG. 1a is a block diagram of an electronic device in a network environment according to one embodiment.

[0032] FIG. 1b illustrates the configuration of a generative AI system according to one embodiment.

[0033] FIG. 2 illustrates software configurations configured to support an AI assistant service in an electronic device according to one embodiment.

[0034] FIG. 3 illustrates an example of an electronic device and external electronic device configuration according to one embodiment.

[0035] FIG. 4 illustrates a method of operating an artificial intelligence assistant of an electronic device using an external electronic device according to one embodiment.

[0036] FIG. 5 illustrates an example of a user interface for operating an artificial intelligence assistant of an electronic device according to one embodiment.

[0037] FIGS. 6a and 6b illustrate examples of execution screens of an artificial intelligence assistant according to one embodiment.

[0038] FIG. 7 illustrates an example of calling an artificial intelligence assistant of an electronic device using an external electronic device according to one embodiment.

[0039] FIG. 8 illustrates an example of a user interface for operating an artificial intelligence assistant according to one embodiment.

[0040] FIG. 9 illustrates an example of a user interface for operating an artificial intelligence assistant according to one embodiment.

[0041] FIGS. 10a and 10b illustrate an example of a user interface for operating an artificial intelligence assistant in a flexible electronic device according to one embodiment.

[0042] FIG. 11 illustrates an example of a user interface for operating an artificial intelligence assistant according to one embodiment.

[0043] FIGS. 12a to 12d illustrate an example of a user interface for operating an artificial intelligence assistant according to one embodiment.

[0044] FIG. 13 illustrates an example of calling an artificial intelligence assistant of an electronic device using an external electronic device according to one embodiment.

[0045] FIG. 14 illustrates an example of calling an artificial intelligence assistant of an electronic device using an external electronic device according to one embodiment.

[0046] FIGS. 15a and 15b illustrate an example of calling an artificial intelligence assistant of an electronic device using an external electronic device according to one embodiment.

[0047] Each of the embodiments described with reference to the drawings of the present disclosure may be configured independently as a single embodiment. Each of the embodiments described with reference to the drawings of the present disclosure may operate independently as a single embodiment. At least two of the embodiments described with reference to the drawings of the present disclosure may be configured in combination. At least two of the embodiments described with reference to the drawings of the present disclosure may operate in combination. For example, at least a part of the embodiment of FIG. 1a and at least a part of the embodiment of FIG. 2 may operate in combination with each other.

[0048] When at least two of the embodiments described with reference to the drawings of the present disclosure are combined, at least some of the configurations and / or at least some operations included in each embodiment may be omitted.

[0049] The electronic device (101) according to the embodiments disclosed in this document may be of various forms. The electronic device (101) may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0050] FIG. 1a is a block diagram of an electronic device in a network environment according to one embodiment.

[0051] Referring to FIG. 1a, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160) (or display), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0052] The processor (120) includes at least one processing circuitry, and the at least one processing circuitry can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0053] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0054] Memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, software (e.g., program (140)) and input or output data for related instructions. Memory (130) may include volatile memory (132) or non-volatile memory (134). Memory (130) can store instructions executable by the processor (120) or the electronic device (101).

[0055] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0056] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0057] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0058] A display module (160) (or a display) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0059] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0060] The sensor module (176) may include at least one sensor. The sensor module (176) may detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0061] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0062] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0063] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0064] The camera module (180) includes at least one camera and can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0065] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0066] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0067] A communication module (190) includes at least one communication circuit and can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of a processor (120) (e.g., an application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., a LAN (local area network) communication module, or a power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0068] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0069] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0070] According to one embodiment, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0071] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0072] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0073] An electronic device (101) according to one embodiment may support a generative artificial intelligence (AI) function. A generative AI function may refer to a technology that generates new content based on content and can generate a new form of AI content (or generative content) by utilizing given input data or information. Here, AI content may refer to content (e.g., text, objects, images, videos, audio, screen information) that is entirely generated (or reconstructed / edited) or partially generated (or reconstructed / edited) based on generative AI.

[0074] An electronic device (101) according to one embodiment may support generative AI functions in conjunction with a server (108). At least one of the electronic device (101) or the server (108) according to one embodiment may include the configuration of the generative AI system illustrated in FIG. 1b.

[0075] FIG. 1b illustrates the configuration of a generative AI system according to one embodiment.

[0076] Referring to FIG. 1b, the generative AI system may include a user interface (10100), an AI framework (10200), a generative AI model (10300), application and service components (10400), and a database component (10500).

[0077] A user interface (e.g., user query / response interface) (10100) according to one embodiment may receive a user query. The user query input may be in the form of natural language, images, and videos. The user interface (10100) may transmit not only data regarding the user query input but also context information to an AI framework (10200). The context information may include various additional information at the time of user input. Additionally, the user query input may be in a mixed form of the aforementioned natural language, images, sounds, and context information. Furthermore, the user query input may be in a non-natural language form that does not generate natural language, such as a menu selection (e.g., a creation request or a modification request). The user interface (10100) may output results from a generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of actions requested by the user.

[0078] An AI framework (AI framework, 10200) according to one embodiment receives user query input and can coordinate and control each component necessary to perform the user's intent based on the user's query input. Such an AI framework (10200) may include a prompt design component (10210) (e.g., prompt manager), an APIs / Plugins Management component (10230), and an output modification component (10250).

[0079] A user query or action entered in a user interface (10100) according to one embodiment may be transmitted to a prompt design component (10210). The prompt design component (10210) may be used to generate a prompt suitable for input into a large language model (LLM) or a large multimodal model (LMM). The prompt design component (10210) may be an AI component that uses machine learning algorithms or neural networks to develop better prompts over time. The prompt design component (10210) may generate a prompt by accessing a database component (10500) (e.g., a knowledge component) containing user preference data, a prompt library, and prompt examples, and transmit it to the large language model (LLM) or large multimodal model (LMM).

[0080] An API and plugin management component (10230) according to one embodiment can perform the role of communicating with external information when there is a request for additional information when transmitting user input as input to a generative model. The API and plugin management component (10230) establishes a channel to communicate with the outside of application and service components (10400) (e.g., AI Interface) through an application programming interface (API), thereby enabling access to various data sources. Additionally, the API and plugin management component (10230) can request an action through the API if the application or service needs to perform an action that ultimately executes a user query rather than an intermediate result. Information obtained from the outside can be transmitted as input to the generative model along with the user input.

[0081] An output modification component (10250) according to one embodiment can finely tune the output of a generative model. For example, the output modification component (10250) can verify whether the content generated through a language model (LLM) or a large-scale multimodal model (LMM) is irrelevant, contains biased content, or contains harmful content. Additionally, the output modification component (10250) can determine the extent to which the output matches the result desired by the user and, if additional processing is required, proceed with that process. Furthermore, the output modification component (10250) can configure and provide hints to the user to avoid unwanted output.

[0082] A generative AI model (10300) according to one embodiment generally refers to an artificial intelligence neural network that generates new forms of data based on user input information. Models that generate images may include, typically, a generative adversarial network (GAN) and a variational auto encoder (VAE). For example, a generative AI model may be a Diffusion-based generative model using a VAE and a Transformer structure. Additionally, a language-generating model may refer to a model trained to output the statistically most appropriate output value based on input values. Among the generative AI models (10300), a language-generating model may be, for example, models such as CHAT-GPT 3 and CHAT-GPT 4. As another example, a large multimodal model (LMM) may be a model capable of recognizing various forms of data input, such as text, images, and voice, and generating new data corresponding to them.

[0083] An electronic device (101) according to one embodiment may be implemented to provide generative AI functions (or operations / services) using a part of the artificial intelligence (AI) system of FIG. 1b.

[0084] An electronic device (101) according to one embodiment can provide an AI assistant service using a generative AI function. For example, the electronic device (101) can provide the AI ​​assistant service described below using the generative AI system of FIG. 1b.

[0085] According to one embodiment, there are no limitations on the computation and data processing functions that the processor (120) can implement on the electronic device (101); however, the present disclosure describes operations for executing (or calling) the AI ​​assistant of the electronic device (101) and operating the AI ​​assistant using the input of an external electronic device. The operations of the electronic device (101) or the processor (120) described below can be performed by loading instructions stored in the memory (130).

[0086] An electronic device (101) according to one embodiment may include a display (e.g., a display module (160)). An electronic device (101) according to one embodiment may include a communication module (190) including at least one communication circuit. An electronic device (101) according to one embodiment may include a connector. An electronic device (101) according to one embodiment may include at least one processor (120) including a processing circuitry. An electronic device (101) according to one embodiment may include a memory that stores executable instructions by the at least one processor. According to one embodiment, when the instructions are executed individually or collectively by the processor (120), the electronic device (101) may receive an input signal calling an artificial intelligence (AI) assistant from an external electronic device connected to the communication circuit or the connector while the application execution screen is displayed on the display. According to one embodiment, the commands may cause the electronic device to execute an AI assistant based on the received input signal to display the app execution screen in a first area of ​​the display and display an AI execution screen related to the AI ​​assistant in a second area of ​​the display. According to one embodiment, the commands may cause the electronic device to display information related to the user query input on the AI ​​execution screen based on the receipt of the user query input.The commands according to one embodiment may enable the electronic device to generate an answer to the user query input through an AI assistant and display it on the AI ​​execution screen based on at least some of the user query input, information related to an application running on the electronic device, information related to a system, or information related to personalization. The AI ​​execution screen according to one embodiment may include at least one recommendation AI function executable through the AI ​​assistant in relation to the application or app execution screen displayed in the first area, and zero input device information of the external electronic device executing the recommendation AI function.

[0087] According to one embodiment, the external electronic device may be characterized by being connected to the electronic device based on at least one of wired communication, wireless communication, or power line communication.

[0088] According to one embodiment, the external electronic device may be characterized as an external electronic device including a keyboard device that includes an AI-dedicated key for calling an AI assistant.

[0089] The commands according to one embodiment may enable the electronic device to analyze an application executed in the first area and content included in the app execution screen, and to use the analyzed information to generate an answer to the user query input.

[0090] According to one embodiment, the input signal may include at least one of voice-based input, image-based input, text-based input, or physical key-based input.

[0091] Information related to an application currently running according to one embodiment includes context obtained from an application displayed in a first area or an application running in the background, and information related to a system includes context obtained from system information to which the electronic device has allowed access to the AI ​​assistant, and clipboard and history information managed by the electronic device, and information related to personalization may include user preference information or user context that the electronic device has learned and stored in advance.

[0092] The commands according to one embodiment may cause the electronic device to display a text-based interaction UI (user interface) or execute a first type of AI assistant when the input signal received from the external electronic device is a first input, and to display a voice-based interaction UI or execute a second type of AI assistant when the input signal received from the external electronic device is a second input.

[0093] According to one embodiment, the commands may cause the electronic device to display information used when generating the answer or additional information related to the answer on the AI ​​execution screen or output it as voice data when displaying an answer to the user query on the AI ​​execution screen.

[0094] The commands according to one embodiment may enable the electronic device to recommend a second application that can be supported in relation to the answer to the user query, and to guide the user to check whether to execute the second application through the AI ​​execution screen.

[0095] The commands according to one embodiment may cause the electronic device to display a response screen based on the execution of the second application in a third area located between a reduced first area and the second area when user input requesting the execution of the second application is received, or to switch the application execution screen displayed in the first area to a response screen based on the execution of the second application.

[0096] According to one embodiment, the commands may enable the electronic device to request the execution of the AI ​​assistant to the other electronic device when another electronic device registered with the same account as the electronic device is connected, transmit information related to the application running on the electronic device, information related to the system, and information related to personalization to the other electronic device, and receive a response corresponding to a user's query input from the other electronic device.

[0097] According to one embodiment, the commands may cause the electronic device to display at least some of the generated answers on the AI ​​execution screen and to request an output by transmitting information used when generating the answers or additional information related to the answers to the other electronic device.

[0098] The electronic device (101) illustrated in FIG. 1a according to the various embodiments disclosed below can be implemented, and for components that are substantially identical to the configuration disclosed in FIG. 1a above, the same reference number is assigned, and redundant descriptions of the functions may be omitted.

[0099] FIG. 2 illustrates software configurations configured to support an AI assistant service in an electronic device according to one embodiment. The configurations illustrated in FIG. 2 may refer to software (or application) modules operated by a processor (120). For example, the operations of the AI ​​assistant (220) may be executed by the processor (120).

[0100] Referring to FIG. 2, an electronic device (e.g., the electronic device (101) of FIG. 1a) may include an application (210), an AI assistant (e.g., an AI assistant application) (220), and an AI engine (230). Components of the electronic device (101) may be operatively or electrically connected to each other. In this disclosure, the term “AI assistant” may be replaced with “AI agent.”

[0101] The electronic device (101) can store, install, and run various applications (210). The types of applications that can be run on the electronic device (101) (e.g., web browser applications, messaging applications, email applications, map applications, document creation applications, etc.) are not limited. Hereinafter, the term "application" may be used interchangeably with the term "application (app)."

[0102] The AI ​​assistant (220) can use an AI engine (230) (e.g., the generative AI model (10300) of FIG. 1b) to provide answers (e.g., text, audio, image, screen information, objects) corresponding to user queries through natural language conversation (e.g., voice or text) with the user, and can control overall operations related to AI functions / services. The AI ​​assistant (220) can download an AI engine (or AI model) (230) and data of a specific domain from a server and execute them on a device, or use an AI engine (not shown) on a server to generate answers corresponding to user queries.

[0103] According to one embodiment, the AI ​​assistant (220) may include a context collection module (2210), an analysis module (2213), a prompt manager (2215) (e.g., a prompt design component (10210)), an AI function selection module (2217), and a key combination configuration module (2219).

[0104] The context collection module (2210) can collect context information and transmit it to the prompt manager (2215) when the AI ​​assistant is executed or when a context request is received from the AI ​​assistant. The context information may include app context related to the application, system context related to the system, and personalized user context. The context collection module (2210) may include a context engine (not shown) for identifying and / or inferring context related to the electronic device (101) (e.g., spatial context, behavioral context, and / or temporal context). For example, the processor (120) executing the context engine may acquire or manage spatial context information, which is information about places where the user of the electronic device (101) is repeatedly located (e.g., home, workplace, other places frequently visited, car, entry and / or exit from a specific place). To obtain spatial context information, the processor (120) may obtain information related to the connectivity of the GNSS (global navigation satellite system), the electronic device (101), and / or the POI (point of interest).

[0105] The app context may include information obtainable from an application that is running or is running on the electronic device (101) before the AI ​​assistant is executed. The app context information may be information related to at least one of a foreground application or a background application, a currently focused application (e.g., an application with an input function enabled), or a screen currently displayed on the display. For example, the app context information may be information about the current state of the application (e.g., if viewing a photo in the gallery, information about that photo).

[0106] The system context may include information obtainable from a device platform or a system (e.g., an operating system). The system context may include general information of the electronic device (101) (e.g., current time, current location based on GPS, information on physical movement based on motion sensors (e.g., walking, running, driving), record information of the electronic device (101) (e.g., clipboard, search information), and status information of the electronic device (101) (e.g., battery information, information on devices connected to the electronic device (e.g., external electronic devices and network information). For example, information regarding driving may not only confirm driving via motion sensors but also confirm boarding and parking by detecting Bluetooth connections within the vehicle. The system context may include system information that the electronic device (101) has allowed access to the AI ​​assistant, and clipboard and history information managed by the electronic device (101).

[0107] User context may include personalized information / personalized data about the user. User context may include at least one of user activity information (e.g., commuting, shopping, travel), user emotional state, and user current state (e.g., interests, intentions (shopping)). Information about the user may be identified using information registered with the user or application, or using pre-learned personalized data. User context may include data used for determining priorities of content related to AI assistant functions and / or summarizing content. User context may include data related to the user's activities of the electronic device (101) (e.g., staying, moving, exercising, sleeping, taking photos, eating, making calls, listening to music, watching videos, playing games, making reservations, and / or making payments). The user context may include environmental context such as time, date, location, and / or weather, activity and / or behavioral context of the user of the electronic device (101), context regarding content stored in the electronic device (101) (e.g., upcoming events, and / or coupon expiration dates), and / or data representing characteristics of the user of the electronic device (101) (e.g., preferences of said user). The user context may include data for retrieval-augmented generation (RAG). The analysis module (2213) may analyze the screen currently displayed on the display or the currently focused application (e.g., an application being operated by the user, an application with an input function enabled) and transmit it to the context module (2210) or the AI ​​function selection module (2217).

[0108] The prompt manager (2215) can generate a prompt requesting the creation of AI content based on user query input and transmit the generated prompt to the AI ​​engine (230). The prompt manager (2215) can generate a prompt (input value) to be input to the AI ​​engine along with user query input and context information. For example, the prompt manager (2215) can generate a prompt requesting content that can be regenerated by an AI function in relation to an application or screen currently displayed on the display. As another example, the prompt manager (2215) can complete the prompt by adding user query input received through an AI assistant to a preset prompt template. The prompt manager (2215) can complete the prompt by adding context information to the prompt template along with user query input. The context information may be context information associated with the user's input. Here, the “prompt template” may include instructions (or goals) indicating what needs to be done, which an AI engine (e.g., LLM) uses input values ​​added to the template (e.g., user input, context information, additional information related to the instructions).

[0109] The AI ​​function selection module (2217) can select executable AI functions using an AI assistant in relation to the application currently displayed on the display (e.g., the application running in the foreground before screen splitting) and / or the currently displayed execution screen, and provide the selected AI functions to the user interface as recommended AI functions.

[0110] For example, when an AI assistant is launched while a map screen of a map application is displayed on the display, the AI ​​function selection module (2217) combines map screen analysis and collected context information to select content that can be generated by AI in relation to the currently displayed map screen (e.g., travel itinerary, recommendation of famous tourist attractions, etc.) and can suggest this as a recommendation AI function (e.g., would you like me to create a travel itinerary for you? Would you like me to recommend famous tourist attractions?).

[0111] The key combination configuration module (2219) can identify devices and networks connected to the electronic device (101) and configure a key combination to execute an AI function recommended by an AI assistant based on external electronic devices (202) and networks connected to the electronic device (101).

[0112] For example, when the electronic device (101) is connected to a keyboard device, and the AI ​​function selection module (2217) selects a first AI recommendation function (e.g., regenerating a travel itinerary) and a second AI recommendation function (e.g., recommending famous tourist attractions), the first key combination of the keyboard device (e.g., ctrl + Z) can be configured as an input key to execute the first AI recommendation function, and the second key combination (e.g., shift) can be configured as an input key to execute the second AI recommendation function. The key combination may be configured differently depending on the external electronic device (202) connected to the electronic device (101). As another example, if the external electronic device (202) connected to the electronic device (101) is a remote control device, the key combination may be configured as a combination of Channel + Key, Channel - Key, Volume + Key, Volume - Key, and / or Number Key.

[0113] According to some embodiments, the selection of AI functions and the configuration of key combinations may be performed through an AI engine. For example, a prompt manager (2215) may generate a prompt saying, “Select AI functions associated with or recommended in relation to the application currently displayed on the screen, and recommend a key combination using an input key of an external electronic device (202) connected to the current electronic device (101),” and transmit the generated prompt to an AI engine (230) to receive recommended AI functions and an input key combination of an external electronic device (202) for executing said recommended AI functions from the AI ​​engine (230).

[0114] The AI ​​engine (230) (e.g., the generative AI model (10300) of FIG. 1b) may be referred to as an AI model. The AI ​​engine (230) may include, for example, at least one of an LLM (large language model) AI engine (231), an MLLM (multimodal large language model) AI engine (233), and an LVM (large vision model) AI engine (not shown). Based on the amount of data to be processed in relation to the AI ​​assistant function / service, the difficulty of analysis, the load of the electronic device (101), and / or network conditions, the electronic device (101) may use either an on-device AI engine or an AI engine on a server, or the on-device AI engine and the AI ​​engine on a server may collaborate to provide the AI ​​assistant function / service.

[0115] The AI ​​engine (230) can generate an answer (e.g., AI content) corresponding to a prompt (input value) transmitted along with user input and context information transmitted from the prompt manager (2215). The prompt transmitted from the prompt manager (2215) may include instructions (or goals) indicating what the AI ​​engine (230) needs to do. The AI ​​engine (230) can generate and output an answer (e.g., AI content related to the application or execution screen currently displayed) based on the instructions included in the prompt (e.g., recommend an AI function related to the currently displayed application, respond to user input).

[0116] FIG. 3 illustrates an example of an electronic device and external electronic device configuration according to one embodiment.

[0117] Referring to FIG. 3, an electronic device (101) according to one embodiment may receive input functions for operating the electronic device (101) through an external electronic device (202).

[0118] For example, the external electronic device (202) may refer to another electronic device (e.g., mobile device, wearable device, HMD (head mounted display), motion controller, remote control) that can be connected to the electronic device (101) based on at least one of wired communication, wireless communication, or power line communication. In the illustration of FIG. 3, the external electronic device (202) is depicted as a keyboard device, but this is merely an example and is not limited thereto.

[0119] As an example, the keyboard device illustrated may communicate with the electronic device (101) via power line communication. The keyboard device may include a stand part (311) and a keypad part (312) configured to be detachable from / combinable with the electronic device (101). The stand part (311) may include a connector (e.g., a Poco connector) (315) that can be electrically connected to the electronic device (101). When the electronic device (101) is combined with the keyboard device, it may communicate with the electronic device (101) (e.g., power line communication) through the connector (e.g., a Poco connector) (315). The keypad part (312) may include a plurality of input keys that provide input functions. According to some embodiments, the keyboard device may further include a touch pad that supports touch input, but is not limited thereto.

[0120] A keyboard device (e.g., external electronic device (202)) according to one embodiment may include an AI key (320) (e.g., dedicated AI key) for calling an AI assistant, but is not limited thereto.

[0121] When an input selecting an AI key (320) is received while the electronic device (101) and the keyboard device are connected via the connector (315), the keyboard device may transmit an input signal (or trigger signal) to the electronic device (101) to call an AI assistant via the connector (315). The input signal transmitted from the keyboard device via the connector (315) may be transmitted to the processor (120) of the electronic device (101). The processor (120) may execute the AI ​​assistant based on the input signal received from the keyboard device.

[0122] According to some embodiments, a specific key may be assigned to the external electronic device (202) to call the AI ​​assistant. For example, when an electronic device (101) that supports the AI ​​assistant is connected to the external electronic device (202), one of the input keys included in the external electronic device (202) (e.g., “F12” in the case of a keyboard device) may be assigned as the key to call the AI ​​assistant. The electronic device (101) may support actions for assigning the key to call the AI ​​assistant in the external electronic device (202) (e.g., UI screens that show the input keys of the external electronic device to the user when the external electronic device is connected and induce the user to assign / select one of the input keys).

[0123] FIG. 4 illustrates a method of operating an artificial intelligence assistant of an electronic device using an external electronic device according to one embodiment. Each operation illustrated in FIG. 4 may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0124] Referring to FIG. 4, an electronic device (101) according to one embodiment may be connected to an external electronic device (202) in operation 410. For example, the external electronic device (202) may refer to another electronic device (e.g., mobile device, wearable device, HMD (head mounted display), motion controller, remote control) that can be connected to the electronic device (101) based on at least one of wired communication, wireless communication, or power line communication. Wireless communication may be short-range communication, but is not limited thereto.

[0125] In operation 420, the electronic device (101) can execute a first application (hereinafter referred to as the application) under the control of the processor (120) and display an app execution screen corresponding to the first application on the display.

[0126] For example, the first application may refer to an application running in the foreground and / or an application with an input function enabled (e.g., an application currently being operated by the user, a focus application displaying a cursor function). The electronic device (101) may also have at least one other second application (e.g., applications previously running in the background) running in addition to the first application currently being displayed on the display.

[0127] In operation 430, the processor (120) can receive an input signal calling an AI assistant (or an input signal requesting the execution of an AI assistant) from an external electronic device (202) connected to the electronic device (101).

[0128] According to one embodiment, the input for calling an AI assistant through an external electronic device (202) may include at least one of voice input, video input, text input, or physical key input. The video input may include video or image input captured through a camera. Alternatively, the video input may include gesture input recognized through a camera.

[0129] For example, if the external electronic device (202) is a keyboard device including the AI ​​key (320) shown in FIG. 3, the processor (120) can receive an input signal selecting the AI ​​key (320) from the keyboard device.

[0130] According to one embodiment, the processor (120) may execute different AI assistant modes or call (or execute) different AI assistants depending on the type of input signal received from the external electronic device (202). For example, if the input signal received from the external electronic device (202) is a first input signal (e.g., a key input of less than a threshold time, a single touch input signal, an AI key input signal, or an input signal of swiping an AI key in a first direction, etc.), the processor (120) may execute a text-based AI assistant (or an AI assistant of the first company). If the input signal received from the external electronic device (202) is a second input signal (e.g., a key input of more than a threshold time, a double touch input signal, an input signal of a combination of an AI key input signal and another key, or an input signal of swiping an AI key in a second direction, etc.), the processor (120) may execute a voice-based AI assistant (or an AI assistant of the second company). Input signals for running different AI assistant modes or calling (or running) different AI assistants can be set differently depending on the type of electronic device (101), design, or implementable AI function.

[0131] According to one embodiment, the processor (120) may receive an input signal calling an AI assistant (or an input signal requesting the execution of an AI assistant) from an external electronic device (202) after receiving a pen input or touch input to a specific area of ​​the execution screen of the first application currently displayed on the display. In operation 440, the processor (120) may store a list of previously executed applications based on the reception of the input signal calling an AI assistant from the external electronic device (202) and analyze the content of the app execution screen (e.g., currently displayed content) displayed on the current display.

[0132] According to one embodiment, the processor (120) can acquire and store information or application list data of applications that are executed or are running on the electronic device (101) (e.g., first application and second application) until an input signal calling an AI assistant is received from an external electronic device (202).

[0133] For example, applications of the electronic device (101) can be executed using an intent structure. The AI ​​assistant can obtain a list of applications that were executed up to the point of executing the AI ​​assistant by using an intent filter. The AI ​​assistant defines a list of executed applications as an item filtered among the information within the intent object in relation to the intent filter, and can be designed to obtain data of the list of executed applications through the intent filter among the intent information when the AI ​​assistant application is executed.

[0134] According to one embodiment, the processor (120) may obtain information related to the first application or data related to the currently displayed application execution screen based on an Application Programming Interface (API). The API is an interface for providing functions provided between the kernel or middleware to the application, and may include at least one interface or function (e.g., a command) for file control, window control, image processing, or character control.

[0135] For example, if the first application is a media playback application, the processor (120) can obtain data such as a title, description, and thumbnail related to the currently displayed app execution screen (or video content) through the data API of the media playback application.

[0136] According to another embodiment, if the first application does not implement a separate API, the processor (120) can capture and analyze the currently displayed app execution screen through an AI function and obtain data related to the app execution screen based on the analyzed result. For example, the processor (120) can capture a video playback screen and then recognize a landmark included in the captured image, or capture a map screen and then recognize a region focused on the current map.

[0137] In operation 450, the processor (120) can select at least one recommended AI function that is executable in relation to the first application of the first execution screen displayed on the current display through the AI ​​function based on the reception of an input signal calling the AI.

[0138] Operations 450 and 455 may be omitted. According to some embodiments, operations 450 and 455 may be performed in parallel with or independently of operation 440, but may also be performed sequentially after operation 440.

[0139] A processor (120) according to one embodiment can analyze and recommend an AI function executable from a first application and / or a currently displayed app execution screen based on at least one of the characteristics of a first application, information related to the first application, and data obtained through screen analysis.

[0140] For example, if the app execution screen is a map screen based on a map application, the processor (120) can select AI functions related to or associated with the currently displayed map screen, such as “1st recommendation AI function (e.g., recommendation of famous tourist spots)”, “2nd recommendation AI function (e.g., recommendation of travel itineraries)”, and “3rd recommendation AI function (e.g., recommendation of restaurant lists)” as recommendation AI functions related to the currently displayed map.

[0141] In operation 455, the processor (120) can configure an AI function key combination to execute an AI function corresponding to at least one selected AI function.

[0142] According to one embodiment, the processor (120) can determine the configuration of an input device that constitutes an external electronic device (202) connected to an electronic device (101), and can configure a key combination (e.g., a combination of keyboard keys in the case of a keyboard) that can be input through the external electronic device (202) to execute an AI function.

[0143] For example, the processor (120) may configure a “Ctrl+Z” key combination as a trigger input for executing a first recommendation AI function and a “A+Z” key combination as a trigger input for executing a second recommendation AI function. These key combinations may be changed according to the input device of an external electronic device (202) connected to the electronic device (101). As another example, if the external electronic device (202) is a remote control device, the key combination for the recommendation AI function may be composed of a combination of volume buttons, channel change buttons, etc.

[0144] According to one embodiment, the processor (120) may configure various key combinations that trigger (or transmit commands) functions related to the operation of the AI ​​assistant using key combinations of an external electronic device (202), regardless of the recommendation AI function.

[0145] For example, if the external electronic device (202) is a keyboard device, the electronic device (101) can execute a function to switch to a mode where all context information obtainable from the electronic device is transmitted to the AI ​​assistant to generate an answer when an input signal of the “AI key + A key” combination is received. The electronic device (101) can execute a function to switch to a mode where the AI ​​assistant generates an answer without context information when an input signal of the “AI key + O key” combination is received. The electronic device (101) can execute a function to log out the AI ​​assistant or terminate the AI ​​assistant when an input signal of the “Windows key + L key” combination is received.

[0146] According to one embodiment, the electronic device (101) may operate functions related to an AI assistant by combining the function keys and AI keys of an external electronic device (202). For example, the combination of “AI key + F2, F3 keys (screen brightness control)” may be configured to execute a function that automatically adjusts screen brightness based on ambient luminance and / or brightness information frequently set by the user. As another example, the combination of “AI key + F4, F5 keys (volume control)” may be configured to execute a function that automatically adjusts the volume of the electronic device based on ambient sound information and / or volume information frequently set by the user.

[0147] According to one embodiment, the processor (120) can provide at least one selected recommendation AI function and a guide for a recommendation AI function execution key combination configuration to an execution screen of an AI assistant (e.g., an AI execution screen).

[0148] In operation 460, the processor (120) may respond to an input signal from an external electronic device (202) that calls an AI assistant, divide the display area (e.g., multi-window), display a reduced app execution screen in the first area, and display an AI execution screen related to the AI ​​assistant in the second area. The app execution screen and the AI ​​execution screen may be displayed in a multi-window manner, but are not limited thereto. For example, the AI ​​execution screen may be displayed in a pop-up window manner that is superimposed on the app execution screen.

[0149] According to one embodiment, the AI ​​execution screen may display a recommendation object comprising at least one AI recommendation function selected in relation to an application and / or app execution screen displayed in a first area and key combination information of an external electronic device for executing said AI recommendation function, but in some cases, the recommendation object may be omitted.

[0150] According to one embodiment, when an AI assistant is executed while an electronic device (101) is displaying two applications as a split screen (multi-window) of a first area and a second area, the size of only the first area where the application with the currently activated input function (e.g., focus app) is displayed may be reduced, or the size of both the first area and the second area may be reduced, and then an AI execution screen may be displayed in a third area (e.g., multiple multi-screen (e.g., 3-split)). In the case of multiple multi-screen (e.g., 3-split), the AI ​​execution screen may be displayed in the third area adjacent to the focused app, or may be displayed by overlapping (or replacing) another app (second area) that is not the focused app (first area) (in other words, the AI ​​execution screen is displayed in a 2-split along with the focused app).

[0151] In operation 470, the processor (120) can transmit context information related to previously executed applications, systems, and users to the AI ​​assistant.

[0152] Context information may include an app context related to an application, a system context related to a system, and a personalized user context; this distinction is made merely for illustrative purposes, and context information may be applied integrally in this disclosure. The app context, system context, and user context may be applied as described in FIG. 2.

[0153] According to one embodiment, the processor (120) can collect context information through an OS Framework or an AI agent daemon and transmit it to an AI assistant.

[0154] According to one embodiment, the processor (120) can collect all context information (e.g., app context, system context and user context) obtainable from the electronic device (101) to the AI ​​assistant at the time of the first execution of the AI ​​assistant and transmit it to the AI ​​assistant.

[0155] According to another embodiment, the processor (120) checks the user query input described below after the AI ​​assistant is first executed and can collect only context information related to or associated with the user query input (e.g., some of the application, system, and user contexts) and transmit it to the AI ​​assistant.

[0156] According to one embodiment, when an input signal calling an AI assistant (e.g., an external electronic device request or an AI call input of the electronic device) is received after the electronic device (101) has booted up and at least one application is not running, the electronic device (101) may provide an AI recommendation function based on a context (list of previously executed apps) stored before the electronic device is terminated.

[0157] In operation 480, the processor (120) can receive user query input related to the execution of the AI ​​assistant's function.

[0158] In operation 490, the processor (120) can obtain an answer based on user query input and context information from the AI ​​engine.

[0159] According to one embodiment, an AI assistant can generate a prompt requesting an answer to user input, transmit user query input and context information (e.g., app context, system context, user context) to an AI engine, and obtain a product (e.g., AI content, answer) from the AI ​​engine.

[0160] FIG. 5 illustrates an example of a user interface for operating an artificial intelligence assistant of an electronic device according to one embodiment.

[0161] Referring to FIG. 5, according to one embodiment, <501> On the screen, the electronic device (101) may display an app execution screen (510) (or content screen) related to a media playback application (e.g., a first application) in a display area (A) under the control of the processor (120). For example, the app execution screen (510) may be a screen playing “media content related to travel or scenery”.

[0162] <502> On the screen, the electronic device (101) can divide the display area based on receiving an input signal calling an AI assistant from an external electronic device (202), display an app execution screen (510) in a reduced size in the first area (A1), and display an AI execution screen (520) related to the AI ​​assistant by running the AI ​​assistant in the second area (A2).

[0163] <502> The screen is illustrated as an example in which a UI for text-based interaction (e.g., a text input mode object (530) and / or an input field (535)) is displayed on the AI ​​execution screen (520), but this is merely an example. When the electronic device (101) receives an input signal calling a voice mode AI assistant from an external electronic device (202), a UI for voice-based interaction (e.g., a voice input mode object (e.g., the voice input mode object (655) of FIG. 6b) and / or an input field (535)) may be displayed. In one example, the input mode object (e.g., the text input mode object (530) or the voice input mode object (655)) may be omitted, and a guide text “Please enter a query” may be displayed in the input field (535).

[0164] For example, the AI ​​execution screen (520) may include a text input mode object (530)), an input field (535), and recommended AI function items (e.g., summarizing videos, planning trips, viewing related spot locations) (540) in relation to the media playback application currently displayed in the first area (A1) or the screen currently displayed in the first area (e.g., video related to movies). The AI ​​execution screen (520) may also provide content explaining how to use the AI ​​assistant (e.g., AI can remember the previous conversation and perform the next command. Try saying this: “What is your schedule today?” -> “What about the weekend?”) or / and content explaining the user input mode (e.g., please enter text).

[0165] According to one embodiment, AI function items (540) may be omitted.

[0166] When the user selects the input field (535), the electronic device (101) may change to a text input mode (e.g., cursor display) or / and display a keypad interface (not shown). The user may enter a query into the input field (535) via the keypad interface or an external electronic device (202) (e.g., keyboard device). For example, if the user enters the query text “Recommend a 4-night, 5-day travel itinerary” into the input field (535), the electronic device (101) may provide an answer to the query “Recommend a 4-night, 5-day travel itinerary” (e.g., a 4-night, 5-day travel itinerary related to a travel region included in a media video) on the AI ​​execution screen (520).

[0167] For example, the electronic device (101) can provide an answer to the user's inquiry without the user having to separately input or specify additional tourist areas included in the movie media by analyzing a media playback application running in the first area (A1) and / or a video currently displayed in the first area (A1) to identify a country, region, or location, etc., related to the video currently being viewed.

[0168] According to one embodiment, the electronic device (101) can predict the range of user queries regarding the media playback application or video currently being played on the screen by analyzing the media playback application and screen running in the first area (A1) and transmitting the app context, system context, and personalized user context related to the application or video currently being displayed in the first area (A1) to an AI engine, and can provide answers to user queries without additional information or specific information.

[0169] FIGS. 6a and 6b illustrate examples of different forms of an artificial intelligence assistant execution screen according to one embodiment.

[0170] Referring to FIG. 6a, an electronic device (101) according to one embodiment is <601> As illustrated in Fig. 5, <502> Execution key information (610) for executing recommended AI functions can be provided to the AI ​​function items (540) included in the AI ​​execution screen (520) displayed on the screen. The execution key information (610) may refer to an input key (or hot key) that requests the execution of the corresponding AI function using an input device of an external electronic device (202) connected to the electronic device (101), without the user inputting text or voice.

[0171] <601> On the screen, the recommended AI features may be, for example, “Summarize Video,” “Plan a Trip,” and “View Related Spot Locations,” and the execution key to execute “Summarize Video” may be “Ctrl+Z.” The execution key to execute “Plan a Trip” is “Shift,” and the execution key to execute “View Related Spot Locations” may be “Ctrl+A.”

[0172] For example, when a user inputs “ctrl+z” through an external electronic device (202), the electronic device (101) <602> As illustrated in Fig. 5, a query input object (620) for “summarizing the video” (e.g., “summarize the video URL and the content of the video (prompt)”) can be automatically generated and displayed on the AI ​​execution screen (520). As another example, the user may directly input text such as “summarize the URL information and the content of this video” in relation to the video displayed on the app execution screen (510) illustrated in Fig. 5 through the text input window (530).

[0173] The AI ​​assistant can provide answer content (630) summarizing the “video” displayed on the app execution screen (510) to the AI ​​execution screen (520) as an answer to the query input object (620).

[0174] Referring to FIG. 6b, according to one embodiment, an electronic device (101) responds to an input signal received from an external electronic device (202) (e.g., an input signal longer than a threshold time) <603> As illustrated in [Image], a UI for voice-based interaction (e.g., voice input mode object (655) and input field (535)) can be provided. <603> As shown in the image, in the case of an AI execution screen (521) for voice-based interaction, a voice input mode object (655) and an input field (535) may be displayed. In the input field (535) of the AI ​​execution screen (521) for voice-based interaction, content such as “Speak to me in voice” may be displayed. While voice input is being received through a microphone, the electronic device (101) may display a voice reception status object (640) (e.g., a microphone icon) indicating that voice is being received on the AI ​​execution screen (521), and may display text corresponding to the recognized voice input in the input field (535).

[0175] According to one embodiment, even if the electronic device (101) displays a UI for voice-based interaction, if the recommendation AI function can be executed by a key combination of an external electronic device (202), <604> As illustrated in the figure, key combination information (610) for executing each recommended AI function in the AI ​​function items (540) can be provided to the AI ​​execution screen (521).

[0176] According to one embodiment, the electronic device (101) <603> When an input signal of less than the threshold time is received again from an external electronic device (202) on the screen, <601> As illustrated in [Image], it can be converted into a UI for text-based interaction.

[0177] According to some embodiments, an AI key received from an external electronic device may be configured to allow switching between a UI mode for text-based interaction and a UI mode for voice-based interaction.

[0178] FIG. 7 illustrates an example of calling an artificial intelligence assistant of an electronic device using an external electronic device according to one embodiment.

[0179] Referring to FIG. 7, an electronic device (101) according to one embodiment is <701> As illustrated in [Image], an app execution screen (710) associated with the first application can be displayed.

[0180] The electronic device (101) can receive a first input signal calling an AI assistant from an external electronic device (202) while the app execution screen (710) is displayed. For example, the first input signal may be a signal in which a first key (e.g., AI key) designated to call the AI ​​assistant is input for less than a threshold time.

[0181] The electronic device (101) may divide the display area as shown in <702-1> based on the first input signal received from the external electronic device (202) to display an app execution screen (710) in the first area and run an AI assistant in the second area to display an AI execution screen (720) for text-based interaction. The AI ​​execution screen (720) for text-based interaction may include a text input mode object (730), an input field (735), and / or a text mode description object (740). Although not shown in the drawings, the AI ​​execution screen (720) may include recommended AI function items and / or execution key combinations. The AI ​​execution screen (720) shown in FIG. 7 may be of FIG. 5 according to the design implementation. <502> or of Fig. 6a <601> It may be replaced with the AI ​​execution screen (520) shown in the image.

[0182] The electronic device (101) can receive a second input signal calling an AI assistant from an external electronic device (202) while the app execution screen (710) is displayed. For example, the second input signal may be a signal in which a first key designated to call the AI ​​assistant is entered for a period of time longer than a threshold time.

[0183] The electronic device (101) may display an app execution screen (710) in a first area, execute an AI assistant in a second area, and display an AI execution screen (721) for voice-based interaction, as illustrated in <702-3>, based on the second input signal being received from an external electronic device (202). The AI ​​execution screen (721) for voice-based interaction may include a voice input mode object (731) and / or a voice mode description object (750). The AI ​​execution screen (720) for voice-based interaction may also include recommended AI functions and / or execution key combinations depending on the design implementation, and <603> or <604> It may be replaced with the AI ​​execution screen (521) shown in.

[0184] The electronic device (101) can easily run a text-based AI assistant or a voice-based AI assistant according to a signal input from an external electronic device (202).

[0185] FIG. 8 illustrates an example of a user interface for operating an artificial intelligence assistant according to one embodiment.

[0186] Referring to FIG. 8, according to one embodiment, the electronic device (101) of FIG. 5 <501> As illustrated in the image, when an AI assistant is launched while an app launch screen (e.g., an app launch screen that plays video content (510)) (810) is displayed, an AI launch screen (820) related to the AI ​​assistant can be displayed at a designated location (e.g., designated as default, but can be changed by user settings), and the location of the AI ​​launch screen (820) can be changed by user operation.

[0187] For example, the AI ​​execution screen (820) may be located in any one of the top, bottom, left, or right directions of the display area. The AI ​​execution screen (820) may also be displayed as a pop-up window overlapping at least a part of the app execution screen (810). The AI ​​execution screen (820) <801> As illustrated in [Image], it is displayed in a first direction (e.g., right area) relative to the app execution screen (810), or <802> As illustrated in [figure], it can be displayed in a second direction (e.g., upper area), and <803> As illustrated in FIG. 8, it may be illustrated in a third direction (e.g., left direction) or displayed in a fourth direction (e.g., lower area). In FIG. 8, the UI displays a voice input mode object (825), but when the AI ​​assistant in text input mode is executed, a text input mode object may be displayed.

[0188] The user <801> In the screen illustrated in, when an AI execution screen (820) located in the first direction (e.g., right area) is selected and dragged upward, the electronic device (101) [removes] the AI ​​execution screen (820). <802> As shown in [figure], it can be moved to a second direction (e.g., upper area).

[0189] FIG. 9 illustrates an example of a user interface for operating an artificial intelligence assistant according to one embodiment.

[0190] Referring to FIG. 9, according to one embodiment, the electronic device (101) can support a function (e.g., a hiding function) for operating an AI assistant through touch interaction.

[0191] The electronic device (101) of FIG. 5 <501> As illustrated in FIG. 9, when an input signal from an external electronic device (202) calling an AI assistant is received while an app execution screen (e.g., an app execution screen (510) for playing video content) (910) is displayed, the display area can be divided to display the app execution screen (910) in the first area and the AI ​​execution screen (920) in the second area. In the UI illustrated in FIG. 9, a voice input mode object (925) is displayed, but when an AI assistant in text input mode is executed, a text input mode object may be displayed.

[0192] If the user touches the AI ​​execution screen (920) and then drags (or swipes) in the first direction (e.g., to the right) relative to the app execution screen (910), <901> As illustrated in [Image], the size of the AI ​​execution screen (920) may gradually decrease and disappear from the display area (e.g., hidden state). In this case, the size of the app execution screen (910) may increase as the size of the AI ​​execution screen (920) decreases. When the AI ​​execution screen (920) disappears from the display area, the app execution screen (910) may be converted into a full window form. In the hidden state, the execution of the AI ​​assistant may not be terminated.

[0193] If the user touches the display again from the direction where the AI ​​execution screen (920) disappeared and drags (or swipes) to a second direction (e.g., left direction) relative to the app execution screen (910), the electronic device (101) <902> As illustrated in [Image], the AI ​​execution screen (920) is displayed again, and <901> Conversely, it can be displayed again as a visual representation in which the size of the second execution screen (920) gradually increases.

[0194] Although not illustrated in the drawing, if the user moves further in a second direction (e.g., left direction) after touching the AI ​​execution screen (920), the size of the app execution screen (910) may gradually decrease and the size of the AI ​​execution screen (920) may gradually increase, and when the app execution screen (910) disappears from the display area, the AI ​​execution screen (920) may be converted into a full window form.

[0195] FIGS. 10a and FIGS. 10b illustrate an example of a user interface for operating an artificial intelligence assistant in a flexible electronic device according to one embodiment.

[0196] Referring to FIG. 10a and FIG. 10b, an electronic device (101) according to one embodiment may have a deformable structure for a display (e.g., foldable, multi-foldable, sliderable structure).

[0197] According to one embodiment, an electronic device having a deformable structure for a display can support a function of changing and operating a UI / execution screen related to an AI assistant in a relative position depending on the size or orientation of an active area displaying visual information. As part of the deformable structure, the electronic device (101) may have a foldable structure capable of varying the display area of ​​a flexible display through at least two housing structures rotatably coupled to each other. The electronic device (101) illustrated in FIGS. 10a and 10b illustrates an example of a multi-foldable in which a first folding axis and a second folding axis are implemented in an in-folding manner, but is not limited thereto. The display of the multi-foldable in which the first folding axis and the second folding axis are implemented in an in-folding manner may include a first area located in the first housing (1010), a second area located in the second housing (1011), and a third area located in the third housing (1012).

[0198] In the first state (e.g., fully unfolded state) of the electronic device (101), the first display (e.g., front display) has the first area, the second area, and the third area activated, and the electronic device (101) can display an app execution screen (1020) by running an application through the first area, the second area, and the third area.

[0199] For example, an electronic device (101) is based on an input signal received from an external electronic device connected to the electronic device (101). <1001> As illustrated in [Image], an AI execution screen (1030) related to A Assistant can be displayed in the first area. The electronic device (101) <1002> and <1003> As illustrated in [Image], it can be switched from a first state to a second state (e.g., partially folded, partially unfolded state). When the electronic device (101) is in the second state, the first display has a third area activated, and the AI ​​assistant or AI execution screen (1030) can be switched to a hidden state due to the structure in which the first area and the second area are folded so as to face each other.

[0200] As another example, the electronic device (101) <1004> As illustrated in the figure, an app execution screen (1020) can be displayed in a first state (e.g., full expanded state), and an AI execution screen (1030) can be displayed in a first area based on an input signal received from an external electronic device. The electronic device (101) <1005> As illustrated in the figure, when transitioning from a first state to a second state (e.g., a folded state of the first housing (1010) and the second housing (1011), or an unfolded state of the second housing (1011) and the third housing (1022)), the third area can be divided to display an app execution screen (1020) and an AI execution screen (1030). The electronic device (101) <1006> As illustrated in the figure, when the first housing (1010), the second housing (1011), and the third housing (1012) are all folded and switched to a third state, the second display (e.g., rear display) is activated, and the second display can be split to display an app execution screen (1020) and an AI execution screen (1030).

[0201] FIG. 11 illustrates an example of a user interface for operating an artificial intelligence assistant according to one embodiment.

[0202] Referring to Fig. 11, <1101> In the screen, the electronic device (101) can execute an AI assistant based on an input signal received from an external electronic device (202) while an application execution screen (1110) related to the application is displayed, and divide the display area (e.g., multi-window) to display the application execution screen (1110) in the first area and the AI ​​execution screen (1120) related to the AI ​​assistant in the second area.

[0203] When the AI ​​assistant is launched, the electronic device (101) may designate the AI ​​launch screen (1120) as a focus area where input functions are activated (or an area where cursor functions are displayed). For example, when the AI ​​assistant is launched and a UI for text-based interaction is displayed, the electronic device (101) may display an input cursor (e.g., keyboard cursor) in an input field (e.g., input field (530) shown in FIG. 5) included in the AI ​​launch screen (1120) to induce user input. As another example, when the AI ​​assistant is launched and a UI for voice-based interaction is displayed, the electronic device (101) may activate a microphone and display a voice reception status object (e.g., voice reception status object (640) shown in FIG. 6b) indicating the voice reception status to switch to a state where voice input can be received.

[0204] According to one embodiment, <1101> On the screen, the electronic device (101) can display a focus object (1130) on the AI ​​execution screen (1120) indicating that the focus area is a focus area where the input function is activated. The focus object (1130) is represented as a rectangular border object (1130) in FIG. 11, but this is merely an example and is not limited thereto. <1102> On the screen, the user can touch the app launch screen (1110). <1103> Based on the input of touching the app execution screen (1110) on the screen, the focus area and focus object (1130) can be moved to the app execution screen (1110). <1104> When the user touches the AI ​​execution screen (1120) again on the screen, the focus area and focus object (1130) can be moved back to the AI ​​execution screen (1120).

[0205] According to one embodiment, <1105> On the screen, the electronic device (101) can receive a focus switching key input (e.g., alt + tap) from an external electronic device (202) while the focus object (1130) is displayed on the AI ​​execution screen (1120). The electronic device (101) <1106> As shown in the screen, the focus area and focus object (1130) can be moved to the app execution screen (1110). When the electronic device (101) receives a focus switching key input (e.g., alt + tap) again from an external electronic device (202), <1107> As shown in the screen, the focus area and focus object (1130) can be moved back to the AI ​​execution screen (1120).

[0206] FIGS. 12a to 12d illustrate an example of a user interface for operating an artificial intelligence assistant according to one embodiment.

[0207] Referring to FIGS. 12a to 12d, an electronic device (101) according to one embodiment can support a function of providing the user with information regarding the generated answer and the information used in the answer when displaying an answer to a user query input after executing the AI ​​assistant based on receiving an input signal calling the AI ​​assistant from an external electronic device (202), and a guide function for an executable application related to the answer based on the answer and app execution history.

[0208] For example, of Fig. 12a <1201> On the screen, based on an input signal from an external electronic device, an app execution screen (1210) containing content describing Parisian attractions via a web browser is displayed in a first area, and an AI execution screen (1220) may be displayed in a second area. The AI ​​execution screen (1220) is a screen displaying a UI for text-based interaction (e.g., a text input mode object (1223), an input field (1225)), but this is merely an example and is not limited thereto.

[0209] To receive assistance from the AI ​​assistant, the user can enter a query through the input field (1225). For example, the user can enter the text “Plan a one-week trip to Paris next month including the Eiffel Tower and Montmartre Hill” into the input field (1225) and send it to the AI ​​assistant (e.g., by pressing the send button). The AI ​​assistant can display the user’s query input object (1230) on the AI ​​execution screen (1220).

[0210] The electronic device (101) can display an answer object (1240) generated through an AI assistant based on the user's query input on an AI execution screen (1220). When generating an answer to the user's query through an AI assistant, the electronic device (101) can generate an answer by obtaining additional information such as the user's December schedule information, tourist attraction information within the currently displayed screen (e.g., Eiffel Tower, Montmartre Hill), recently searched keywords (e.g., Louvre Museum), and the user's interests and information (e.g., soccer) through information displayed on the app execution screen (1210), information analyzing previously executed apps and / or context information (e.g., app context, system context and personalized user context).

[0211] An electronic device (101) according to one embodiment may provide a recommendation AI function (1250) on an AI execution screen (1220) that recommends a second application (e.g., Excel application) that is additionally executable in relation to the answer to a user's query, but this may be omitted.

[0212] When a user requests the execution of a second application related to the answer (e.g., when “Yes” is entered into the input field (1225) in response to “Would you like me to make a travel plan in Excel?”), the electronic device (101) may reduce the size of the first area as shown in <1202-1> of FIG. 12b, run the Excel application in the third area and display the travel plan (1270) created through the Excel file, or, as shown in <1202-2> of FIG. 12c, switch the first area to the background of the app execution screen (1210) and display the travel plan (1270) created through the Excel file.

[0213] According to one embodiment, the electronic device (101) may provide the user with an explanation regarding the reason for creation or the information used in the answer when the AI ​​assistant generates an answer, separately from the answer object (1240) for the user query. The electronic device (101) may transmit personalized user context (or personalized information) managed by the system to the AI ​​assistant to be used for generating the answer. If the user has travel plans, the electronic device may provide an answer and an explanation regarding the user's query by utilizing the pre-learned context. For example, when providing supplementary explanations in voice, the electronic device (101) can output supplementary explanations in voice, such as, “Looking at the gallery… (omitted)… It seems you enjoy watching soccer while traveling, so I’ve planned a trip from December 4 to 11 so you can watch the Paris Saint-Germain match scheduled for December 7. Tickets are still available at reasonable prices. In particular, I have included the Eiffel Tower and Montmartre Hill, which you showed interest in through recent searches, and the Louvre Museum, which is a popular destination, in the itinerary. Since there is a forecast of rain throughout Paris on December 9, I have included the Louvre Museum, where indoor activities are possible, in the itinerary for the 9th. I have displayed the itinerary on the screen.” As another example, when a user asks the AI ​​assistant, “Are there any hotels available for accommodation here?” while looking at a map app, the electronic device (101) can answer, “Since you usually stay in accommodations under 300,000 won, we have displayed hotels available for booking in that price range sorted by rating. We recommend Le Meridien.”

[0214] According to one embodiment, the electronic device (101) of FIG. 12d <1203> As illustrated in [Image], based on the input (1245) selecting the Louvre Museum from the answer object (1240), an object (1260) that elaborates on the selected information of the Louvre Museum can be displayed on the AI ​​execution screen (1220).

[0215] FIG. 13 illustrates an example of calling an artificial intelligence assistant of an electronic device using an external electronic device according to one embodiment.

[0216] Referring to FIG. 13, according to one embodiment, when an electronic device (e.g., tablet) (101) receives a request to run an AI assistant (e.g., an input signal from an external electronic device) while an app execution screen (1310) displaying an application or content is displayed, it checks if there is another electronic device (e.g., smartphone) (1301) registered with the same account, and if another electronic device (1301) registered with the same account is within a short-range communication range (e.g., a device connected to the same AP, a device located in the Quick Share Discovery area, a device recognized in a video captured through a camera, a device connected via BT with a previously registered account, etc.), the electronic device (101) can request the other electronic device (1301) to run an AI assistant.

[0217] The electronic device (101) can transmit context information (e.g., app context, system context and user context) collected by the electronic device (101) along with an AI assistant execution request to another electronic device (1301).

[0218] According to some embodiments, the electronic device may request the execution of an AI assistant from another electronic device (1301) when at least one of the following occurs: the battery condition of the electronic device (e.g., battery capacity below a threshold), network connection constraints, CPU / memory load condition (e.g., overload), or heat condition.

[0219] Another electronic device (1301) can execute an AI assistant based on an AI assistant execution request transmitted from the electronic device (101) and display an AI execution screen (1320) on the display. The other electronic device (1301) can generate an answer through the AI ​​assistant in response to a user's query input and can provide the generated answer to the user through the speaker of the other electronic device (1301) or the AI ​​execution screen (1320) on the display. As another example, the other electronic device (1301) can transmit the generated answer back to the electronic device (101) so that the electronic device (101) can provide the answer to the user.

[0220] FIG. 14 illustrates an example of calling an artificial intelligence assistant of an electronic device using an external electronic device according to one embodiment.

[0221] Referring to FIG. 14, when a user according to one embodiment runs an AI assistant through an electronic device while wearing an HMD, an AI execution screen can be provided in the HMD environment.

[0222] for example, <1401> As illustrated in the figure, the electronic device (101) can enter a virtual reality environment (1410) (or augmented reality environment) while connected to an HMD device. The virtual reality environment (1410) can display a first execution screen displayed on the electronic device in a first window (1420).

[0223] <402> As illustrated in the illustration, when an electronic device (101) in a virtual reality environment (1410) receives an input calling an AI assistant, it can execute the AI ​​assistant and output an AI execution screen related to the AI ​​assistant to a second window (1430) adjacent to a first window (1420) where a first execution screen is displayed, on top of the virtual reality environment (1410).

[0224] FIGS. 15a and 15b illustrate an example of calling an artificial intelligence assistant of an electronic device using an external electronic device according to one embodiment.

[0225] Referring to FIGS. 15a and 15b, according to one embodiment, <1501> On the screen, the electronic device (101) may display a restaurant reservation execution screen (e.g., an execution screen based on a web browser or a restaurant application) (1510) in a display area (A) under the control of the processor (120). Based on receiving an input signal (e.g., an AI key input signal) calling the AI ​​assistant from an external electronic device (202), the electronic device (101) may display a UI for interaction with the AI ​​assistant (e.g., an AI object or input mode object (1530), an input field (1535)) in at least one area of ​​the task bar (1520). For example, <1501> Guide text (1540), such as “Enter a command to AI (or type to AI),” may be displayed in the input field on the screen. The input mode object (1530) may be an AI object indicating that the AI ​​assistant is running, but may also be an object indicating an AI input mode according to an input signal that calls the execution of the AI ​​assistant.

[0226] The user <1502> As shown in the screen, a user query (“Add dinner to my calendar next Tuesday at 5 pm”) can be entered into the input field (1535). The electronic device (101) can display a query input object (1545) corresponding to the user’s query input in the input field (1535). <1503> On the screen, the electronic device (101) transmits the user's query input to the AI ​​assistant, and when it receives an answer to the query from the AI ​​assistant, it may divide the display area to display a restaurant reservation execution screen (1510) in a reduced size in the first area (A1) and display an AI execution screen (1550) in the second area (A2). The AI ​​execution screen (1550) may include a query input object (1545) and an answer object (e.g., restaurant reservation information) (1550) corresponding to the query input. <1503> In the screen, the input mode object (1530) and input field (1535) are shown as being included in the AI ​​execution screen (1550) and task bar, but the input mode object (1530) and input field (1535) displayed on the task bar may not be displayed.

[0227] As another example, the electronic device (101) may provide recommendation AI functions related to restaurant reservations (e.g., a restaurant reservation change function, or an event addition function, etc.) to the AI ​​execution screen (1550).

[0228] According to one embodiment, the electronic device can graphically process, in relation to an input mode object (1530), a state in which query input is being received and a state in which an answer is being generated through an AI assistant are visually distinguished. For example, <1504> As illustrated in [Image], the “1531” input mode object represents a state where query input is being received, and the “1532” object represents a state where an answer is being generated through the AI ​​assistant.

[0229] A method for operating an artificial intelligence assistant of an electronic device using an external electronic device according to one embodiment may include an operation of displaying an application execution screen on a display. A method according to one embodiment may include an operation of receiving an input signal calling an AI (artificial intelligence) assistant from an external electronic device while the application execution screen is displayed. A method according to one embodiment may include an operation of executing the AI ​​assistant based on the received input signal to display the application execution screen in a first area of ​​the display and displaying an AI execution screen related to the AI ​​assistant in a second area of ​​the display. A method according to one embodiment may include an operation of receiving a user query input through the AI ​​execution screen. A method according to one embodiment may include an operation of generating an answer to the user query input through the AI ​​assistant based on at least some of the user query input, information related to the application running on the electronic device, system information, or personalization information. A method according to one embodiment may include an operation of displaying the generated answer on the AI ​​execution screen. The above AI execution screen may include at least one recommendation AI function executable via the AI ​​assistant in relation to the application or app execution screen displayed in the first area, and input device information of the external electronic device for executing the recommendation AI function.

[0230] According to one embodiment, the AI ​​assistant may be characterized by executing different AI functions depending on the type of input signal that calls the AI ​​assistant from the external electronic device.

[0231] According to one embodiment, the external electronic device may be characterized by being connected to the electronic device based on at least one of wired communication, wireless communication, or power line communication.

[0232] The operation of generating an answer to the user query input according to one embodiment may further include analyzing the application executed in the first area and the content included in the app execution screen, and using the analyzed information to generate an answer to the user query input.

[0233] According to one embodiment, information related to an application running on the electronic device includes context obtained from an application displayed in a first area or an application running in the background, and information related to a system includes context obtained from system information that the electronic device has allowed access to the AI ​​assistant, and clipboard and history information managed by the electronic device, and information related to personalization may include user preference information or user context that the electronic device has learned and stored in advance.

[0234] An operation of displaying an AI execution screen related to the AI ​​assistant according to one embodiment may further include an operation of displaying a text-based interaction UI (user interface) when the input signal received from the external electronic device is a first input, and an operation of displaying a voice-based interaction UI when the input signal received from the external electronic device is a second input.

[0235] The operation of displaying the generated answer on the AI ​​execution screen according to one embodiment is

[0236] It may further include an action of displaying information used when generating the above answer or additional information related to the answer on the AI ​​execution screen or outputting it as voice data.

[0237] The operation of displaying the generated answer on the AI ​​execution screen according to one embodiment may further include recommending a second application that is supported in relation to the answer to the user query and providing a request to the user through the AI ​​execution screen to check whether the second application is executed.

[0238] The operation of providing a user with confirmation of whether the second application is executed through the AI ​​execution screen according to one embodiment may further include one of the following: when input from a user requesting the execution of the second application is received, an operation of displaying an answer screen based on the execution of the second application in a third area located between a reduced first area and the second area; or an operation of switching the app execution screen displayed in the first area to an answer screen based on the execution of the second application.

[0239] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" each may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0240] The term “module” as used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0241] One embodiment of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0242] According to one embodiment, the method according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0243] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the components of the multiple components in the same or similar manner as those performed by the corresponding components among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, display; A communication module comprising at least one communication circuit; Connector; At least one processor including processing circuitry; and It includes a memory that stores executable instructions that can be executed by at least one processor, and When the above instructions are executed individually or collectively by the processor, the electronic device, While the application's app execution screen is displayed on the display, an input signal calling an AI (artificial intelligence) assistant is received from an external electronic device connected to the communication module or the connector, and Based on the received input signal, the AI ​​assistant is executed to display the app execution screen in the first area of ​​the display, and an AI execution screen related to the AI ​​assistant is displayed in the second area of ​​the display. Based on the reception of user query input, information related to the user query input is displayed on the AI ​​execution screen, and Based on the above user query input, information related to an application running on the electronic device, system information, or personalization information, an answer to the above user query input is generated through the AI ​​assistant and displayed on the AI ​​execution screen. An electronic device characterized in that the AI ​​execution screen above includes at least one recommendation AI function executable via the AI ​​assistant in relation to the application or app execution screen displayed in the first area, and input device information of the external electronic device that enables the execution of the recommendation AI function.

2. In Paragraph 1, An electronic device characterized by the fact that the above external electronic device is connected to the electronic device based on at least one of wired communication, wireless communication, or power line communication.

3. In Paragraph 1, An electronic device characterized in that the above external electronic device is an external electronic device including a keyboard device that includes an AI-dedicated key for calling the AI ​​assistant.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the processor, the electronic device, An electronic device that analyzes an application executed in the first area and the content included in the app execution screen, and uses the analyzed information to generate an answer to the user query input.

5. In Paragraph 1, The above input signal is an electronic device comprising at least one of voice-based input, image-based input, text-based input, or physical key-based input.

6. In Paragraph 1, Information regarding applications running on the electronic device includes context obtained from applications displayed in the first area or applications running in the background, and The above system-related information includes system information to which the electronic device has allowed access to the AI ​​assistant, and context obtained from clipboard and history information managed by the electronic device. The above personalization-related information is an electronic device comprising user preference-related information or user context that the electronic device has learned and stored in advance.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the processor, the electronic device, If the input signal received from the above external electronic device is the first input, a text-based interaction UI (user interface) is displayed or a first type of AI assistant is executed, and An electronic device that displays a voice-based interaction UI or executes a second type of AI assistant when the input signal received from the above external electronic device is a second input.

8. In Paragraph 1, When the above instructions are executed individually or collectively by the processor, the electronic device, An electronic device that, when displaying an answer to the above user query on the above AI execution screen, displays information used when generating the answer or additional information related to the answer on the above AI execution screen or outputs it as voice data.

9. In Paragraph 1, When the above instructions are executed individually or collectively by the processor, the electronic device, An electronic device that recommends a second application capable of supporting the answer to the above user query and guides the user to check whether to execute the second application through the AI ​​execution screen.

10. In Paragraph 9, When the above instructions are executed individually or collectively by the processor, the electronic device, When user input requesting the execution of the second application is received, a response screen based on the execution of the second application is displayed in a third area located between the reduced first area and the second area, or An electronic device that switches the app execution screen displayed in the first area above to an answer screen based on the execution of the second application.

11. In Paragraph 9, When the above instructions are executed individually or collectively by the processor, the electronic device, An electronic device that, when another electronic device registered with the same account as the electronic device is connected, requests the execution of the AI ​​assistant to the other electronic device, transmits information related to the application running on the electronic device, information related to the system, and information related to personalization to the other electronic device, and receives an answer corresponding to the user's query input from the other electronic device.

12. In Paragraph 11, The above commands are for the electronic device, An electronic device that displays at least some of the generated answers on the AI ​​execution screen and transmits information used when generating the answers or additional information related to the answers to the other electronic device to request an output.

13. A method for operating an artificial intelligence assistant of an electronic device using an external electronic device, The action of displaying the application's execution screen on the display; The operation of receiving an input signal to call an AI (artificial intelligence) assistant from an external electronic device while the above-mentioned app execution screen is displayed; An operation to execute an AI assistant based on the received input signal to display the app execution screen in a first area of ​​the display and display an AI execution screen related to the AI ​​assistant in a second area of ​​the display; An operation to display information related to the user query input on the AI ​​execution screen based on the reception of the user query input; The operation of generating an answer to the user query input through the AI ​​assistant based on at least some of the user query input, information related to an application running on the electronic device, system information, or personalization information; and It includes an action of displaying the generated answer on the AI ​​execution screen, and The above AI execution screen is characterized by including at least one recommendation AI function executable via the AI ​​assistant in relation to the application or app execution screen displayed in the first area, and input device information of the external electronic device for executing the recommendation AI function. A method characterized in that the AI ​​assistant executes different AI functions depending on the type of input signal calling the AI ​​assistant from the external electronic device.

14. In Paragraph 13, The operation of generating an answer to the above user query input is A method further comprising the operation of analyzing an application executed in the first area and the content included in the app execution screen, and using the analyzed information to generate an answer to the user query input.

15. In Paragraph 13, The action of displaying an AI execution screen related to the above AI assistant is When the input signal received from the above external electronic device is the first input, an operation of displaying a text-based interaction UI (user interface) and A method further comprising an operation to display a voice-based interaction UI when the input signal received from the above external electronic device is a second input.

Images