Electronic device for performing extracted operations from content and operation method therefor

Abstract

Disclosed are an electronic device for performing operations extracted from content and an operation method therefor. The electronic device comprises: a memory for storing instructions; and at least one processor for executing the instructions. The instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to: select, on the basis of a first user input, at least a portion of content loaded on the electronic device so as to be displayed on a display module; determine, on the basis of the at least a portion of the loaded content, candidate operations applicable to the electronic device; map the candidate operations and instructions for causing the electronic device to perform the candidate operations; determine, on the basis of a second user input, at least one target operation among the candidate operations; and perform the at least one target operation on the basis of the at least one target operation and the mapped instructions.

Description

Electronic device and method of operation that perform operations extracted from content

[0001] An electronic device that performs operations extracted from content and a method of operating the same are disclosed.

[0002] Generative AI models can summarize text by reading it and concisely reconstructing its content. By identifying the key information and logical structure of the input text, these models can summarize the text by focusing on important content. Generative AI models, capable of understanding and re-expressing the meaning of text rather than simply extracting it, are being utilized in various technological fields.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] According to one embodiment, the electronic device may include a memory for storing instructions. The electronic device may include at least one processor for executing instructions. When the instructions are executed individually or collectively by the at least one processor, the electronic device may select at least a portion of content loaded into the electronic device to be displayed on a display module based on a first user input. When the instructions are executed individually or collectively by the at least one processor, the electronic device may determine candidate actions applicable to the electronic device based on at least a portion of the loaded content. When the instructions are executed individually or collectively by the at least one processor, the electronic device may map the candidate actions to instructions that cause the electronic device to perform the candidate actions. When the instructions are executed individually or collectively by the at least one processor, the electronic device may determine at least one target action among the candidate actions based on a second user input. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be able to perform the at least one target operation based on instructions mapped to the at least one target operation.

[0005] According to one embodiment, the electronic device may include a memory for storing instructions. The electronic device may include at least one processor for executing the instructions. When the instructions are executed individually or collectively by the at least one processor, the electronic device may select at least a portion of the content loaded into the electronic device to be displayed on a display module based on a first user input. When the instructions are executed individually or collectively by the at least one processor, the electronic device may determine candidate actions applicable to an external electronic device communicating with the electronic device based on at least a portion of the loaded content. When the instructions are executed individually or collectively by the at least one processor, the electronic device may map the actions to instructions that cause the external electronic device to perform the candidate actions. When the instructions are executed individually or collectively by the at least one processor, the electronic device may determine at least one target action among the candidate actions based on a second user input. When the above instructions are executed individually or collectively by the at least one processor, the instructions mapped to the at least one target operation can be transmitted to the external electronic device to enable the external electronic device to perform the at least one target operation.

[0006] According to one embodiment, a method of operating an electronic device may include an operation of selecting at least a portion of content loaded into the electronic device to be displayed on a display module based on a first user input. A method of operating an electronic device may include an operation of determining candidate operations applicable to the electronic device based on at least a portion of the loaded content. A method of operating an electronic device may include an operation of mapping the candidate operations to instructions that cause the electronic device to perform the candidate operations. A method of operating an electronic device may include an operation of determining at least one target operation among the candidate operations based on a second user input. A method of operating an electronic device may include an operation of performing the at least one target operation based on instructions mapped to the at least one target operation.

[0007] According to one embodiment, a method of operation of an electronic device may include an operation of selecting at least a portion of content loaded into the electronic device to be displayed on a display module based on a first user input. A method of operation of an electronic device may include an operation of determining candidate operations applicable to an external electronic device communicating with the electronic device based on at least a portion of the loaded content. A method of operation of an electronic device may include an operation of mapping the operations to commands that cause the external electronic device to perform the candidate operations. A method of operation of an electronic device may include an operation of determining at least one target operation among the candidate operations based on a second user input. A method of operation of an electronic device may include an operation of transmitting commands mapped to the at least one target operation to the external electronic device to cause the external electronic device to perform the at least one target operation.

[0008] According to one embodiment, a non-transient computer-readable recording medium may store one or more computer programs including instructions that execute an operation to select at least a portion of content loaded into the electronic device to be displayed on a display module based on a first user input. A non-transient computer-readable recording medium may store one or more computer programs including instructions that execute an operation to determine candidate operations applicable to the electronic device based on at least a portion of the loaded content. A non-transient computer-readable recording medium may store one or more computer programs including instructions that cause the electronic device to perform the candidate operations and instructions that execute an operation to map the candidate operations. A non-transient computer-readable recording medium may store one or more computer programs including instructions that execute an operation to determine at least one target operation among the candidate operations based on a second user input. A non-transient computer-readable recording medium may store one or more computer programs including instructions that execute an operation to perform the at least one target operation based on instructions mapped to the at least one target operation.

[0009] According to one embodiment, a non-transient computer-readable recording medium may store one or more computer programs including instructions that execute an operation to select at least a portion of content loaded into the electronic device to be displayed on a display module based on a first user input. A non-transient computer-readable recording medium may store one or more computer programs including instructions that execute an operation to determine candidate operations applicable to an external electronic device communicating with the electronic device based on at least a portion of the loaded content. A non-transient computer-readable recording medium may store one or more computer programs including instructions that cause the external electronic device to perform the candidate operations and instructions that execute an operation to map the operations. A non-transient computer-readable recording medium may store one or more computer programs including instructions that execute an operation to determine at least one target operation among the candidate operations based on a second user input. A non-transient computer-readable recording medium may store one or more computer programs including instructions that execute an operation to transmit instructions mapped to the at least one target operation to the external electronic device to cause the external electronic device to perform the at least one target operation.

[0010] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0011] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0012] FIG. 2 is a flowchart for explaining the operation method of an electronic device according to one embodiment.

[0013] FIGS. 3 and 4 are drawings for illustrating the selection of all or part of content based on user input according to one embodiment.

[0014] FIG. 5 is a diagram illustrating a list of candidate operations displayed according to one embodiment.

[0015] FIG. 6 is a flowchart for explaining the determination of whether an alternative operation exists according to one embodiment.

[0016] FIG. 7 is a drawing for explaining a list according to an embodiment in which an alternative operation exists.

[0017] FIG. 8 is a diagram illustrating a user interface in a case where no alternative operation exists according to one embodiment.

[0018] FIG. 9 is a flowchart for explaining the execution of a target operation according to one embodiment.

[0019] FIGS. 10 and FIGS. 11 are drawings for illustrating the screen of an electronic device executing target operations according to one embodiment.

[0020] FIG. 12 is a drawing for explaining an electronic device that extracts operations from content according to one embodiment.

[0021] FIG. 13 is a diagram illustrating the operation of an electronic device and an external electronic device according to one embodiment.

[0022] FIG. 14 is a drawing for explaining the operation of an electronic device and an external electronic device according to one embodiment.

[0023] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0024] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0025] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134).

[0026] The processor (120) may be implemented as a circuitry (e.g., a processing circuit) such as a system on chip (SoC) or an integrated circuit (IC). The processor (120) may include one or more processors. For example, the processor (120) may include a combination of one or more processors such as a CPU, GPU, MPU, AP, and CP. Additionally, the processor (120) may include various processing circuits and / or multiple processors. For example, as used in this specification and claims, the term "processor" may include various processing circuits including at least one processor, and one or more of the at least one processor may be configured to perform the various functions described in this specification, either alone or together in a distributed manner. Where "processor," "at least one processor," or "one or more processors" are described in this specification as being configured to perform various functions, these terms may include, for example without limitation, cases where one processor performs some of the described functions and one or more other processors perform the remaining functions, as well as cases where a single processor performs all of the described functions. Additionally, at least one processor may be a combination of multiple processors that perform the described or disclosed various functions in a distributed manner, etc. At least one processor may execute program instructions to achieve or perform the various functions.

[0027] According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0028] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0029] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0030] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0031] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0032] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0033] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0034] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0035] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0036] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0037] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0038] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0039] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0040] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0041] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0042] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0043] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0044] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0045] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0046] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0047] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0048] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0049] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0050] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0051] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0052] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0053] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0054]

[0055] FIG. 2 is a flowchart for explaining the operation method of an electronic device according to one embodiment.

[0056] The operations described below may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Additionally, some operations may be omitted depending on some embodiments. Operations (210) through (260) may be performed by at least one component (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1).

[0057] According to one embodiment, instructions stored in memory (e.g., memory (130) of FIG. 1) by at least one processor may be executed individually or collectively, and the instructions may cause an electronic device to perform the following operations (210) to operations (260).

[0058] In operation (210), the electronic device can select at least a portion of the content loaded into the electronic device to be displayed on the display module based on the first user input.

[0059] According to one embodiment, content may be loaded to be displayed on a display module (e.g., the display module (160) of FIG. 1). Only a portion of the loaded content may be displayed due to size limitations of the display module. For example, the loaded content may correspond to various media capable of conveying information, such as blogs, search results, and news articles. For example, the loaded content may include information on how to control an electronic device (e.g., how to enable or disable specific functions of the electronic device). For example, the loaded content may include a method to change the background of the electronic device via a blog.

[0060] According to one embodiment, the electronic device can select at least a portion of loaded content based on a first user input. The electronic device can select all or part of the loaded content based on the first user input.

[0061] According to one embodiment, the first user input may be an input that causes the electronic device to select (e.g., capture) at least a portion of the loaded content. A method for selecting at least a portion of the loaded content is described later in FIGS. 3 and 4.

[0062] According to one embodiment, the electronic device may determine, based on a first user input, the user's intention to execute on the electronic device content that is included in at least a portion of the loaded content. For example, the first user input may include executing smart capture while the web page is turned on. For example, the first user input may include copying the content of the web page. For example, if a URL is captured or a portion of text content is captured based on the first user input, the electronic device may display a user interface (e.g., "Apply to my device" menu) asking the user whether to apply information included in at least a portion of the loaded content to the electronic device. The user interface may not be displayed every time. The user interface may be displayed only when at least a portion of the loaded content contains words such as "how to do" and / or "how to do". The user interface may be displayed when it is determined through content analysis of the page (e.g., information analysis) that the content (e.g., information) is determined to contain a method suitable for application to the device.

[0063] According to one embodiment, analysis of at least a portion of loaded content may be performed via a voice assistant (e.g., Bixby). For example, when an electronic device activates a voice assistant and receives a command such as “do this,” it may perform the following actions (220). The command may be any sentence containing an intention to apply information (e.g., content) included in at least a portion of the loaded content to the electronic device. Sentence analysis may be performed via a voice assistant.

[0064] According to one embodiment, the electronic device can determine the user's intention to select at least a portion of the loaded content based on at least a portion of the loaded content. If the electronic device determines that the user's intention is to apply information included in at least a portion of the loaded content to the electronic device, it may display a user interface asking the user whether to apply information included in at least a portion of the loaded content to the electronic device.

[0065] In operation (220), the electronic device can extract information from at least a portion of the selected loaded content.

[0066] According to one embodiment, an electronic device can extract information from at least a portion of loaded content based on a user's selection. If all of the loaded content is selected based on the user's selection, the electronic device can extract the entire content. If only a portion of the loaded content is selected based on the user's selection, the electronic device can extract only the content included in that portion of the loaded content.

[0067] In operation (230), the electronic device can determine candidate operations applicable to the electronic device based on at least a portion of the loaded content.

[0068] According to one embodiment, the electronic device can determine candidate actions from the extracted information.

[0069] According to one embodiment, an electronic device can identify actions for at least a portion of loaded content selected based on a first user input. The electronic device can identify actions for at least a portion of loaded content using a generative artificial intelligence model. Based on the identified actions, the electronic device can determine candidate actions applicable to the electronic device. A method for determining candidate actions applicable to the electronic device based on the identified actions is described later in FIG. 6.

[0070] In operation (240), the electronic device can map the candidate operations to instructions that cause the electronic device to perform the candidate operations.

[0071] According to one embodiment, the electronic device can determine instructions that cause the electronic device to perform candidate operations. The electronic device can map the instructions to corresponding candidate operations.

[0072] In operation (250), the electronic device can determine at least one target operation among candidate operations based on the second user input.

[0073] According to one embodiment, the electronic device may determine at least one target action based on a second user input. The second user input may be a selection command for any one of the candidate actions. A method for determining at least one target action is described later in FIG. 5.

[0074] In operation (260), the electronic device can control to perform at least one target operation based on commands mapped to at least one target operation.

[0075] According to one embodiment, the electronic device may sequentially execute at least one target operation based on instructions mapped to at least one target operation. The execution of the target operation is described later in FIGS. 9 to 11.

[0076] The following describes how to select at least a portion of the loaded content.

[0077]

[0078] FIGS. 3 and 4 are drawings for illustrating the selection of all or part of content based on user input according to one embodiment.

[0079] FIG. 3 illustrates a screen (300) for explaining a method of selecting the entire loaded content according to one embodiment.

[0080] According to one embodiment, the screen (300) may display a portion of loaded content. Only a portion of the loaded content may be displayed due to the size limitations of the display module (e.g., the display module (160) of FIG. 1). The loaded content may include various media capable of conveying information, such as blogs, search results, and news articles. The loaded content may include information on how to control an electronic device (e.g., the electronic device (101) of FIG. 1) (e.g., how to enable or disable specific functions of the electronic device). For example, the loaded content may include information on tips for the initial setup of the smartphone when purchasing the smartphone for the first time.

[0081] According to one embodiment, when a generative artificial intelligence model that supports multimodal is used, the loaded content may include images or videos. The images or videos may include information about a method for controlling an electronic device (e.g., a method for enabling or disabling specific functions of the electronic device). For example, the videos may include information about initial setup tips for a smartphone when purchasing the smartphone for the first time.

[0082] According to one embodiment, the electronic device may receive a capture command. For example, the electronic device may perform a capture when it receives a user's capture command for a capture icon (310) displayed in a quick menu. However, the above-described capture method is merely an example and should not be interpreted as limiting the embodiments of the present disclosure. For example, it is obvious to those skilled in the art that the description of the present disclosure may be applied to various capture methods other than the quick menu-based capture method.

[0083] According to one embodiment, the electronic device may activate the screen (320) to request the user to specify an area to perform a capture. For example, the electronic device may indicate that the screen (320) is activated by adding a layer to the screen (300) to make the screen appear dim. The electronic device may receive user input (e.g., gestures) from the user through the activated screen (320). User input may include various inputs, such as inputs for drawing a specific area and inputs for dragging specific content.

[0084] According to one embodiment, user input may be a selection command (325) for a URL. The electronic device may select the entire loaded content based on the selection command (325) for a URL (uniform resource locator). The electronic device may display a screen (330) in which the entire loaded content is selected. On the screen (330), the electronic device may show the user the area selected by the user. For example, the electronic device may indicate that the URL has been selected based on user input by displaying the area containing the URL differently from other areas. For example, the electronic device may display the color of the entire loaded content differently from before the reception of user input to indicate that the entire loaded content has been selected based on the selection of the URL.

[0085] According to one embodiment, the electronic device may determine the entire loaded content as the subject of analysis when the entire loaded content is selected. The electronic device may extract and analyze information included in the entire loaded content. Through the analysis of the entire loaded content, the electronic device may determine the intention of the user who selected the entire loaded content. The electronic device may determine that the user's intention is to apply the content included in the entire loaded content to the electronic device.

[0086] According to one embodiment, if the electronic device contains words such as "~how to," "~how to," and "~and" repeatedly throughout the loaded content, it can determine that the user's intention is to apply the content included throughout the loaded content to the electronic device.

[0087] According to one embodiment, an electronic device can determine a user's intent by using an artificial intelligence model (e.g., a generative artificial intelligence model or an artificial assistant model). The electronic device can determine whether the user's intent is to apply content included in the entire loaded content to the electronic device by inputting the entire loaded content into the artificial intelligence model.

[0088] According to one embodiment, if it is determined that the user's intention is to apply the content included in the entire loaded content to the electronic device, the electronic device may display a user interface (335). The electronic device may receive input regarding whether to apply the content included in the entire loaded content to the electronic device through the user interface (335). The user interface (335) may receive whether to apply to the electronic device from the user through a confirmation button (331) and a cancellation button (333).

[0089] For example, when the electronic device receives user input for the confirmation button (331), it may perform operations to apply the contents included in the entire loaded content to the electronic device. The operations to apply the contents included in the entire loaded content to the electronic device are described later in FIGS. 5 to 9.

[0090] According to one embodiment, the electronic device may receive user input through a voice assistant as well as a user interface (335). For example, the electronic device may use an AI assistant model to play a voice saying, "Would you like to apply this to my device?" The electronic device may receive user input from the user, such as "Do it like this" and "Yes, apply it," in response to the playback of the voice.

[0091] According to one embodiment, the electronic device may include user input such as a user's voice command. For example, a voice assistant may be executed while loaded content is displayed on a display module. The electronic device may receive a voice command from the user, such as "Do as shown on this page," using the voice assistant. The electronic device may select at least a portion of the loaded content based on the voice command. For example, if the electronic device receives a voice command containing content referring to a specific portion, such as "Apply this part of the content," it may select a portion of the loaded content and apply only the actions included in that portion of the loaded content to the electronic device. For example, if the electronic device receives a voice command containing content referring to all parts, such as "Apply everything," it may apply actions included in the entire loaded content to the electronic device.

[0092] FIG. 4 illustrates a screen (400) (e.g., screen (300) of FIG. 3) for explaining a method of selecting a portion of loaded content according to one embodiment.

[0093] According to one embodiment, the screen (400) may display a portion of loaded content. Only a portion of the loaded content may be displayed due to the size limitations of the display module (e.g., the display module (160) of FIG. 1). For example, the loaded content may include various media capable of conveying information, such as blogs, search results, news articles, etc. The loaded content may include information on how to control an electronic device (e.g., the electronic device (101) of FIG. 1) (e.g., how to enable or disable specific functions of the electronic device). For example, the loaded content may include information on tips for the initial setup of the smartphone when purchasing the smartphone for the first time.

[0094] According to one embodiment, when a generative artificial intelligence model that supports multimodal is used, the loaded content may include images or videos. The images or videos may include information about a method for controlling an electronic device (e.g., a method for enabling or disabling specific functions of the electronic device). For example, the videos may include information about initial setup tips for a smartphone when purchasing the smartphone for the first time.

[0095] According to one embodiment, an electronic device may receive a capture command. For example, the electronic device may perform a capture when it receives a user's capture command for a capture icon (410) displayed in a quick menu (e.g., the capture icon (310) of FIG. 3). However, the above-described capture method is merely an example and should not be interpreted as limiting the embodiments of the present disclosure. For example, it is obvious to those skilled in the art that the description of the present disclosure may apply to various capture methods other than the quick menu-based capture method.

[0096] According to one embodiment, the electronic device may activate a screen (420) (e.g., the screen (420) of FIG. 4) to request the user to specify an area to be captured. For example, the electronic device may indicate that the screen (420) is activated by adding a layer to the screen (400) to make the screen appear dim. The electronic device may receive user input (e.g., gestures) from the user through the activated screen (420). User input may include various inputs, such as inputs for drawing a specific area and inputs for dragging specific content.

[0097] According to one embodiment, user input may be a selection command (425) for a specific area of ​​loaded content. The electronic device may select a portion (437) of the loaded content based on the selection command (425). The electronic device may display a screen (430) (e.g., screen (330) of FIG. 3) in which the portion (437) of the loaded content is selected. The electronic device may show the portion (437) of the loaded content to the user. For example, the electronic device may indicate that the portion (437) of the loaded content has been selected based on user input using an object such as a box. For example, the electronic device may display the portion (437) of the loaded content differently in color from the unselected area. Referring to the screen (430), the electronic device may select and display only the portion describing "2. Changing the lock screen and setting widgets" and the image portion thereof.

[0098] According to one embodiment, the electronic device can analyze the loaded content portion (437) as the loaded content portion (437) is selected. The electronic device can extract information contained in the loaded content portion (437) as the loaded content portion (437) is selected. The electronic device can extract all information (e.g., text, images and / or videos) contained in the loaded content portion (437) and convert it into text. One or more artificial intelligence models may be used for text conversion of images and videos.

[0099] According to one embodiment, the electronic device can re-extract parts related to the user's intent (e.g., actions to be applied to the electronic device) after texting. For example, parts unrelated to the user's intent, such as advertising content, can be excluded from the texted content.

[0100] According to one embodiment, the electronic device can determine the user's intention to select a portion of loaded content by analyzing a portion of loaded content (437). The electronic device can determine that the user's intention is to apply the content included in the portion of loaded content (437) to the electronic device.

[0101] According to one embodiment, if the electronic device contains words such as "~how to," "~how to," and "~and" repeatedly throughout the loaded content, it can determine that the user's intention is to apply the content included throughout the loaded content to the electronic device.

[0102] According to one embodiment, an electronic device can determine a user's intent by using an artificial intelligence model (e.g., a generative artificial intelligence model or an artificial assistant model). The electronic device can determine whether the user's intent is to apply content included in the entire loaded content to the electronic device by inputting the entire loaded content into the artificial intelligence model.

[0103] According to one embodiment, if it is determined that the user's intention is to apply the content included in the loaded content part (437) to the electronic device, the electronic device may display a user interface (435) (e.g., the user interface (335) of FIG. 3). The electronic device may receive input regarding whether to apply the content included in the loaded content part (437) to the electronic device through the user interface (435). The user interface (435) may receive whether to apply to the electronic device from the user through a confirmation button (431) (e.g., the confirmation button (331) of FIG. 3) and a cancellation button (433) (e.g., the cancellation button (333) of FIG. 3).

[0104] For example, when the electronic device receives user input for the confirmation button (331), it may perform operations to apply the content included in the loaded content portion (437) to the electronic device. The operations to apply the content included in the loaded content portion (437) to the electronic device are described later in FIGS. 5 to 9.

[0105] According to one embodiment, the electronic device may receive user input through a voice assistant as well as a user interface (435). For example, the electronic device may use an AI assistant model to play a voice saying, "Would you like to apply this to my device?" The electronic device may receive user input from the user in response to the playback of the voice, such as "Do it like this" and "Yes, apply it."

[0106]

[0107] FIG. 5 is a diagram illustrating a list of candidate operations displayed according to one embodiment.

[0108] Referring to FIG. 5, a list (510) is shown containing candidate actions (530, 540, 550, 560, 570) extracted from at least a portion of loaded content in response to user input requesting application to an electronic device (e.g., electronic device (101) of FIG. 1). The list (510) may display the candidate actions (530, 540, 550, 560, 570) sequentially.

[0109] According to one embodiment, the electronic device may receive an application command to apply information included in at least a portion of content loaded through a user interface (e.g., the user interface (335) of FIG. 3 or the user interface (435) of FIG. 4) in FIG. 3 or FIG. 4 to the electronic device. At least a portion of the loaded content may be selected based on user input.

[0110] According to one embodiment, upon receiving an application command, the electronic device can identify actions for application to the electronic device from at least a portion of the loaded content. The electronic device can input into a generative artificial intelligence model to identify actions for application to the electronic device from at least a portion of the content. The generative artificial intelligence model can identify actions from at least a portion of the content.

[0111] For example, referring to the screen (500), actions identified by a generative AI model from at least part of the content are shown. The screen (500) is shown only to explain the output of the generative AI model, and the screen (500) may not actually be displayed on the electronic device as the generative AI model identifies the actions in the background.

[0112] According to one embodiment, the electronic device can determine candidate operations (530, 540, 550, 560, 570) based on identified operations. A method for determining candidate operations (530, 540, 550, 560, 570) is described later in FIGS. 6 and 7.

[0113] According to one embodiment, the electronic device may determine instructions that cause the electronic device to perform candidate operations (530, 540, 550, 560, 570). The electronic device may map the candidate operations (530, 540, 550, 560, 570) to the instructions that cause the electronic device to perform the candidate operations (530, 540, 550, 560, 570).

[0114] According to one embodiment, commands that cause an electronic device to perform candidate actions (530, 540, 550, 560, 570) may include a location on a screen corresponding to the candidate action that must be selected and a selection command for the location. The location may include coordinates on the screen corresponding to the candidate action. The selection command may include touch, long touch, double touch, and drag.

[0115] According to one embodiment, when candidate actions are determined, the electronic device may display a list (510) containing candidate actions (530, 540, 550, 560, 570) on a display module (e.g., the display module (160) of FIG. 1). By displaying the list (510), the electronic device may provide the candidate actions (530, 540, 550, 560, 570) to the user.

[0116] According to one embodiment, candidate actions (530, 540, 550, 560, 570) may have a dependent relationship. For example, a candidate action for the next action may have a dependent relationship with a candidate action for the current action. If any candidate action in the middle is not performed, the next candidate action may not be performed. For example, if candidate action 3 (550) in the list (510) is not performed, candidate action 4 (560) and candidate action 5 (570) may not be performed. However, this is merely an example and should not be interpreted as limiting the embodiments of the present disclosure. For example, there may be no dependent relationship between the candidate actions (530, 540, 550, 560, 570).

[0117] According to one embodiment, the list (510) may further include an edit button (515). The electronic device may provide modifications to the list (510) through the edit button (515).

[0118] According to one embodiment, the electronic device may provide the addition of candidate actions through an edit button (515). For example, if the candidate actions (530, 540, 550, 560, 570) included in the list (510) are in a dependent relationship, the electronic device may provide only the modification of adding the last candidate action through the edit button (515).

[0119] According to one embodiment, the electronic device may provide an edit of a candidate action through an edit button (515). The electronic device may provide an edit for a specific action of the candidate action through the edit button (515). For example, the electronic device may modify candidate action 4 (560), "proceed with brightness editing," to "proceed with brightness editing to 50%" through the edit button (515). Depending on the modification of candidate action 4 (560), the electronic device may edit the brightness to 50% without separate user input for brightness editing when performing candidate action 4 (560). For example, if candidate action 4 (560) is "proceed with brightness editing," the electronic device may display a setting screen corresponding to brightness editing progress and request a command related to brightness modification from the user. For example, if candidate action 4 is modified to "proceed with brightness editing to 50%," the electronic device may control the brightness to 50% without needing to request a command related to brightness modification from the user.

[0120] According to one embodiment, the electronic device may provide the user with a selection of at least some of the candidate actions (530, 540, 550, 560, 570) through a list (510). The electronic device may receive user input to determine at least one target action among the candidate actions (530, 540, 550, 560, 570) based on user input. Only at least one target action determined based on user input may be performed.

[0121] For example, the list (510) may include a guidance message such as, "The following actions are required to perform the function. Is there a function you do not want?" The user may select unwanted candidate actions based on the guidance message.

[0122] According to one embodiment, when user input for determining a target action is received, a user interface (520) that warns that the candidate action for which user input was received and subsequent candidate actions are not performed may be displayed.

[0123] For example, if candidate action 3 (550) is selected based on user input, the electronic device may display a user interface (520) containing a warning message saying, "You have selected action 3. Actions after 3 will also be disabled based on the selection of action 3."

[0124] According to one embodiment, the user interface (520) may include a confirmation button (521) or a cancellation button (523). The electronic device may receive user input through the confirmation button (521) or the cancellation button (523) of the user interface (520). If the confirmation button (521) is selected, the electronic device may perform at least one target action determined based on the user input. If the cancellation button (523) is selected, the electronic device may provide a determination of at least one target action again.

[0125] Below, we will explain how to determine candidate actions.

[0126] FIG. 6 is a flowchart for explaining the determination of whether an alternative operation exists according to one embodiment.

[0127] The operations described below may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Additionally, some operations may be omitted depending on some embodiments. Operations (610) to (650) may be performed by at least one component (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1).

[0128] According to one embodiment, instructions stored in memory (e.g., memory (130) of FIG. 1) by at least one processor may be executed individually or collectively, and the instructions may cause an electronic device to perform the following operations (610) to operations (650).

[0129] In operation (610), the electronic device can determine whether an identified operation corresponding to the current operation among the identified operations can be performed on the electronic device.

[0130] According to one embodiment, only candidate actions that can be performed by the electronic device (e.g., the list (510) of FIG. 5) may be displayed in the list (e.g., the candidate actions (530, 540, 550, 560, 570) of FIG. 5). The electronic device may extract information from at least a portion of the loaded content and identify actions from the extracted information. The electronic device may determine whether the identified actions are actions that can be performed by the electronic device. For each of the identified actions, the electronic device may determine whether they are actions that can be performed by the electronic device.

[0131] According to one embodiment, the electronic device can determine whether each of the actions identified from at least a portion of the loaded content is a first action that can be performed on the electronic device. The first action is an action identified from the loaded content that can be performed on the electronic device and may be included as a candidate action in a list. The electronic device can determine whether the actions identified can be performed on the electronic device sequentially, starting from the first action.

[0132] According to one embodiment, an electronic device can determine whether an identified operation is a first operation that can be performed by the electronic device based on the specification information of the electronic device. The specification information may include the current software version of the electronic device, the manufacturer, versions and model names of applications, etc. The electronic device can determine whether an identified operation is a first operation that can be performed by the electronic device based on an artificial intelligence model (e.g., a generative artificial intelligence model) that has learned the operations that the electronic device can perform. The artificial intelligence model may be a model that has learned the manual of the electronic device based on the specification information.

[0133] According to one embodiment, an identified action may be determined to be a second action that cannot be performed on an electronic device for various reasons. The second action may not be included in the list as an action identified from loaded content that cannot be performed on an electronic device. For example, if an action identified from at least a portion of the loaded content is an action provided only in a different version, the identified action may be determined to be a second action. For example, if an action identified from at least a portion of the loaded content is for an electronic device of a different manufacturer, the identified action may be determined to be a second action. For example, if an action identified from at least a portion of the loaded content is for a different operating system (OS), the identified action may be determined to be a second action. For example, if an action identified from at least a portion of the loaded content is for a heterogeneous electronic device, the identified action may be determined to be a second action.

[0134] According to one embodiment, the electronic device may perform an operation (650) if the identified operation is determined to be a first operation that can be performed by the electronic device. The electronic device may perform an operation (620) if the identified operation is determined to be a second operation that cannot be performed by the electronic device.

[0135] In operation (620), the electronic device can determine whether there is an alternative operation.

[0136] According to one embodiment, if the electronic device determines that an identified operation is a second operation that cannot be performed by the electronic device, it may use an artificial intelligence model to determine whether there exists a third operation that is an alternative operation capable of replacing the second operation. The third operation is an operation that replaces the second operation so that it can be performed by the electronic device, and may be included as a candidate operation in a list.

[0137] According to one embodiment, the artificial intelligence model may be a model further trained to provide alternative actions, which is an artificial intelligence model that has learned actions that the electronic device of the action (610) can perform. The artificial intelligence model further trained to provide alternative actions may determine whether the identified action can be performed by the electronic device, and may output an alternative action if the identified action cannot be performed. Various LLM models may be used as the artificial intelligence model. For example, LLM models such as ChatGPT and Gemini may be used as the artificial intelligence model. Version information of the electronic device and / or the app used by the electronic device may be transmitted to the LLM model via a prompt. For example, information of the electronic device (e.g., OneUI version information) and / or the version information of the app may be transmitted to the LLM model via a prompt.

[0138] According to one embodiment, if a third operation, which is an alternative operation, exists, the electronic device can perform the operation (640). If a third operation, which is an alternative operation, does not exist, the electronic device can perform the operation (630).

[0139] In operation (630), the electronic device may display a user interface indicating that it cannot perform the identified operations.

[0140] According to one embodiment, the electronic device may display a user interface indicating that identified actions cannot be performed on the electronic device due to specific contents of the loaded content. The user interface is described later in FIG. 8.

[0141] In operation (640), the electronic device can replace the identified operation determined to be the second operation with a third operation that is an alternative operation.

[0142] According to one embodiment, a third action, which is an alternative action, may be included in a list of candidate actions (e.g., the list (510) of FIG. 5).

[0143] In operation (650), the electronic device can determine whether the identified operation corresponding to the current operation is the last identified operation.

[0144] According to one embodiment, the electronic device may terminate the operations according to FIG. 6 if the identified operation corresponding to the current operation corresponds to the last identified operation among the identified operations. The electronic device may terminate the operations according to FIG. 6 after determining whether the operations can be performed sequentially from the first identified operation to the last identified operation among the identified operations and / or whether an alternative operation exists.

[0145] According to one embodiment, the electronic device may perform an operation (610) if the identified operation corresponding to the current operation does not correspond to the last identified operation among the identified operations. The electronic device may perform the operation (610) by making the next operation the current operation.

[0146] For example, in operation (650), when it is determined that an identified operation corresponding to the current operation (e.g., a second identified operation) is performable by the electronic device, the electronic device may determine whether the identified operation corresponding to the current operation (e.g., a second identified operation) is the last identified operation. If the last operation among the identified operations is the fifth identified operation, the electronic device may perform operation (610) by making the next operation (e.g., a third identified operation) the current operation.

[0147] According to one embodiment, if the identified action corresponding to the current action is the last identified action, the electronic device may display a list.

[0148] According to one embodiment, it is assumed that all identified operations are determined to be performable by the electronic device. The identified operations (e.g., a first operation) may be determined as candidate operations and included in a list.

[0149] According to one embodiment, it is assumed that some of the identified operations are determined to be impossible to perform in the electronic device. If there is an alternative operation (e.g., a third operation) for an identified operation that is impossible to perform in the electronic device (e.g., a second operation), the identified operation (e.g., a first operation) and the alternative operation (e.g., a third operation) determined to be possible to perform in the electronic device may be determined as candidate operations and included in a list.

[0150] According to one embodiment, all actions included in the list may be candidate actions. Candidate actions may include identified actions (e.g., a first action) and / or alternative actions (e.g., a third action) determined to be performable on an electronic device based on at least a portion of the loaded content. The electronic device may determine target actions from among the candidate actions based on user input. Target actions may refer to actions to be applied to the selected electronic device based on user input.

[0151] Below, a list is described regarding cases where there is an alternative action for an identified action determined to be a second action.

[0152]

[0153] FIG. 7 is a drawing for explaining a list according to an embodiment in which an alternative operation exists.

[0154] Referring to FIG. 7, a list (710) containing alternative actions (e.g., the list (510) of FIG. 5) is shown. The list (710) may include candidate actions (720, 730, 740, 750, 760) (e.g., the candidate actions (530, 540, 550, 560, 570) of FIG. 5. Among the candidate actions (720, 730, 740, 750, 760), it is assumed that candidate action 3 (750) is an alternative action. The remaining candidate actions (720, 730, 740, 760), excluding candidate action 3 (750), are assumed to be actions identified from at least a portion of the loaded content determined to be a first action.

[0155] According to one embodiment, the electronic device can determine whether each of the actions identified from at least a portion of the loaded content based on the method illustrated in FIG. 6 is a first action that can be performed by the electronic device (e.g., the electronic device (101) of FIG. 1). For example, the electronic device can determine the identified actions determined to be the first actions as candidate actions (e.g., remaining candidate actions (720, 730, 740, 760)). If there is a second action among the identified actions that is not performable by the electronic device, the electronic device can determine whether there is a third action that can replace the second action. For example, the electronic device can determine a third action for the identified action determined to be the second action as a candidate action (e.g., candidate action 3 (750)).

[0156] According to one embodiment, the electronic device can determine whether each of the identified operations is a first operation that the electronic device can perform, based on an artificial intelligence model that has learned the operations that the electronic device can perform. The electronic device can determine whether there exists a third operation that can replace the second operation, based on the artificial intelligence model.

[0157] According to one embodiment, if a second action exists and a third action capable of replacing the second action exists, the electronic device can determine candidate actions by replacing the second action with the third action among the identified actions.

[0158] According to one embodiment, an alternative action may include another action having the same function as the identified action determined to be impossible to perform. For example, it may be determined that entry into the third-party application is impossible to perform on the electronic device because a third-party application, such as a photo-taking application, is not stored on the electronic device. The alternative action may include entry into the first-party application as a photo-taking application.

[0159] According to one embodiment, an alternative operation may include another operation having a function similar to the identified operation that is determined to be impossible to perform. For example, if it is determined that brightness adjustment for an image is impossible to perform on the electronic device, the alternative operation may include saturation adjustment for the image.

[0160] According to one embodiment, the alternative operation may include other operations that perform the same operation but have different names. For example, the identified operation may be "entering the screen settings menu," but the electronic device may only have "screen settings menu." The electronic device may determine that "entering the screen settings menu" is an operation that cannot be performed by the electronic device because the names are different. The electronic device may determine "entering the screen settings menu," which can perform the same operation but has different names, as the alternative operation.

[0161] According to one embodiment, with reference to the screen (700), the determination of a generative artificial intelligence model regarding whether a third action capable of replacing a second action exists is illustrated. The screen (700) is shown only to explain the output of the generative artificial intelligence model, and in reality, the generative artificial intelligence model determines the existence of a replacement action in the background, so the screen (700) may not be displayed on the electronic device.

[0162] According to one embodiment, the electronic device may display a user interface indicating that an alternative action is included among the candidate actions included in the list (710). For example, the electronic device may display an alternative action (e.g., candidate action 3 (750)) differently from the remaining candidate actions (720, 730, 740, 760).

[0163] According to one embodiment, the electronic device may display a user interface indicating that an alternative action is included in the list (710). The electronic device may provide the user with the reason why the alternative action is included through the user interface.

[0164] The following describes cases where no alternative action exists.

[0165]

[0166] FIG. 8 is a diagram illustrating a user interface in a case where no alternative operation exists according to one embodiment.

[0167] According to one embodiment, the electronic device can determine whether each of the operations identified from at least a portion of the loaded content based on the method shown in FIG. 6 is a first operation that can be performed on the electronic device (e.g., the electronic device (101) of FIG. 1). If there is a second operation among the identified operations that is not performable on the electronic device, the electronic device can determine whether there is a third operation that can replace the second operation.

[0168] According to one embodiment, a third action may not exist. If an alternative action does not exist, it may be difficult to achieve the user's intention. The electronic device may display a user interface (810) on the screen (800) indicating that actions identified from at least a portion of the loaded content cannot be performed. The screen (800) may include at least a portion of the loaded content. For example, the electronic device may display a user interface (810) on the screen (800) including the phrase "This portion cannot be applied to the device."

[0169] According to one embodiment, the electronic device may highlight at least a portion of the loaded content that makes the performance of identified actions impossible. For example, the electronic device may highlight the portion that makes the performance of identified actions impossible by displaying a highlight (820).

[0170] The execution of at least one target action determined by user input to a list (e.g., the list (510) of FIG. 5 or the list (710) of FIG. 7) is described below. According to one embodiment, the actions included in the list may all be candidate actions (e.g., candidate actions of FIG. 5 (530, 540, 550, 560, 570) and candidate actions of FIG. 7 (720, 730, 740, 750, 760)). The candidate actions may include identified actions and / or alternative actions determined to be performable on an electronic device determined based on at least a portion of the loaded content. The electronic device may determine target actions from among the candidate actions based on user input. Target actions may mean actions to be applied to the selected electronic device based on user input.

[0171]

[0172] FIG. 9 is a flowchart for explaining the execution of a target operation according to one embodiment.

[0173] The operations described below may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel. Additionally, some operations may be omitted depending on some embodiments. Operations (910) through (970) may be performed by at least one component (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1).

[0174] According to one embodiment, instructions stored in memory (e.g., memory (130) of FIG. 1) by at least one processor may be executed individually or collectively, and the instructions may cause an electronic device to perform the following operations (910) to operations (970).

[0175] In operation (910), the electronic device can determine whether user input is required to perform the current target operation.

[0176] According to one embodiment, the electronic device can sequentially perform at least one target operation determined based on user input. The electronic device can sequentially perform at least one target operation from the first target operation to the last target operation.

[0177] According to one embodiment, user input may be required in the target action. For example, if the target action is logging into a website, user input for entering login information may be required. If the target action is selecting an image, user input for selecting an image may be required.

[0178] According to one embodiment, the electronic device may perform an operation (920) when user input is required to perform the current target operation. The electronic device may perform an operation (940) when user input is not required to perform the current target operation.

[0179] In operation (920), the electronic device can display a screen corresponding to the current target operation and a user interface requesting user input.

[0180] According to one embodiment, a user of an electronic device can perform input on a screen according to a user interface. For example, the user can enter login information. For example, the user can select an image.

[0181] In operation (930), the electronic device can receive input from the user and perform the current target operation.

[0182] According to one embodiment, the electronic device can receive a current target operation based on user input. For example, it can perform a login based on received login information. For example, it can select an image based on the user's choice.

[0183] In operation (940), the electronic device can perform the current target operation.

[0184] According to one embodiment, the electronic device can perform a current target operation based on a command mapped to the current target operation. For example, the electronic device can select a specific location on a screen corresponding to the current target operation based on a command mapped to the current target operation.

[0185] In operation (950), the electronic device can determine whether the current target operation is the last target operation.

[0186] According to one embodiment, if the current target operation is not the last target operation, the electronic device can perform operation (960). If the current target operation is the last target operation, the electronic device can perform operation (970).

[0187] In operation (960), the electronic device may display a screen corresponding to the next target operation. The electronic device may perform operation (910) by making the next target operation the current target operation.

[0188] According to one embodiment, the electronic device displays a screen corresponding to a sequential current target operation from the first target operation to the last target operation of at least one target operation based on operations (910) to operations (970), and can determine whether user input is required.

[0189] The following describes the execution of a target operation using the screen of an electronic device.

[0190]

[0191] FIGS. 10 and FIGS. 11 are drawings for illustrating the screen of an electronic device executing target operations according to one embodiment.

[0192] Referring to FIG. 10, a screen (1000) (e.g., screen (330) of FIG. 3 or screen (430) of FIG. 4) is shown that displays at least a portion of loaded content selected based on user input. It is assumed that at least a portion of the loaded content is a blog containing a method for setting automatic spacing as an initial setup tip following the first purchase of an electronic device (e.g., electronic device (101) of FIG. 1).

[0193] According to one embodiment, it is assumed that at least one target operation includes 1. selecting a keyboard option in the settings, 2. selecting a smart input option, 3. selecting an automatic spacing option, and 4. turning on the Korean language option.

[0194] According to one embodiment, a screen (1010) corresponding to the first target action, which is the current target action, may be displayed. The electronic device may control the screen (1010) based on a command mapped to the first target action. The command mapped to the first target action may include a selection location and a selection command. For example, the selection location may include information about the location of a keyboard option on the screen (1010), and the selection command may include a selection command (e.g., a single touch) to enter the next target action. The electronic device may display an object (1015) on the selection location to show which option has been selected. The electronic device may further display a user interface at the bottom of the screen (1010) indicating that a target action is being performed.

[0195] According to one embodiment, when a first target action is performed on a screen (1010), a screen (1020) corresponding to a second target action, which is the current target action, may be displayed. The electronic device may control the screen (1020) based on a command mapped to the second target action. The command mapped to the second target action may include a selection location and a selection command. For example, the selection location may include information about the location of a smart input option on the screen (1020), and the selection command may include a selection command (e.g., a single touch) to enter the next target action. The electronic device may display an object (1025) on the selection location to show which option has been selected. The electronic device may further display a user interface at the bottom of the screen (1020) indicating that a target action is being performed.

[0196] According to one embodiment, when a second target action is performed on the screen (1020), a screen (1030) corresponding to the third target action, which is the current target action, may be displayed. The electronic device may control the screen (1030) based on a command mapped to the third target action. The command mapped to the third target action may include a selection location and a selection command. For example, the selection location may include information about the location of an automatic spacing option on the screen (1030), and the selection command may include a selection command (e.g., a single touch) to enter the next target action. The electronic device may display an object (1035) on the selection location to show which option has been selected. The electronic device may further display a user interface at the bottom of the screen (1030) indicating that a target action is being performed.

[0197] According to one embodiment, when a third target action is performed on the screen (1030), a screen (1040) corresponding to the fourth target action, which is the current target action, may be displayed. The electronic device may control the screen (1040) based on a command mapped to the fourth target action. The command mapped to the fourth target action may include a selection location and a selection command. For example, the selection location may include information about the location of the Korean option on the screen (1040), and the selection command may include a selection command (e.g., a single touch) to enter the next target action. The electronic device may display an object (1045) on the selection location to show which option has been selected. The electronic device may further display a user interface at the bottom of the screen (1040) indicating that a target action is being performed.

[0198] According to one embodiment, the electronic device can provide at least one target operation to a user who operates along a certain path by sequentially displaying screens (1010, 1020, 1030, 1040) corresponding to at least one target operation.

[0199] According to one embodiment, a command mapped to at least one target action may include only a selection location. For example, it may not include a selection command. Based on the selection location, the electronic device may display an object representing the selection location on a screen corresponding to at least one target action. The electronic device may request a selection command for the object representing the selection location from the user. For example, the electronic device may request a selection command for the object representing the selection location from the user through a user interface. Only after receiving a selection command for the object representing the selection location from the user can the device proceed to the next target action. By requesting a selection command from the user, the user can easily remember the path by which at least one target action is performed.

[0200] According to one embodiment, a command mapped to at least one target action may include a command (e.g., code) that causes an electronic device to perform the target action immediately. An artificial intelligence model that has learned the code of an application included in the electronic device may be used to determine the command that causes the actions to be performed. The artificial intelligence model may generate a command (e.g., code) that causes the electronic device to perform the target action immediately, which is a command mapped to the target action. For example, assume that a command mapped to 1. selecting a keyboard option in a setting includes a command that causes the electronic device to perform the target action immediately. The electronic device can perform the selection of a keyboard option immediately without needing to locate the keyboard option according to the command that causes the electronic device to perform the target action immediately and to select the keyboard option according to the selection command.

[0201] According to one embodiment, the objects (1015, 1025, 1035, 1045) may be determined differently depending on the selection command. For example, if the selection command is a single touch, the object may be a circle. For example, if the selection command is a double touch, the object may be the shape of two circles placed at the same center but with different radii. If the selection command is a drag from left to right, the object may be an arrow from left to right. The shapes of the objects described above are merely examples and should not be interpreted as limiting other embodiments.

[0202] The following describes cases where user input is required to perform a target action.

[0203] Referring to FIG. 11, a screen (1100) (e.g., screen (330) of FIG. 3, screen (430) of FIG. 4, or screen (1000) of FIG. 10) is shown that displays at least a portion of loaded content selected based on user input. It is assumed that at least a portion of the loaded content is a blog containing a method for setting the wallpaper of a smartphone of an electronic device (e.g., electronic device (101) of FIG. 1).

[0204] According to one embodiment, at least one target action is assumed to include 1. selecting an image in a gallery application, 2. selecting an option button, 3. selecting to set as wallpaper, and 4. selecting the home screen.

[0205] According to one embodiment, a screen (1110) corresponding to the first target action, which is the current target action, may be displayed. The screen (1110) may be a screen that requires the selection of an image. For example, since the background cannot be set using just any image, the selection of an image to set as the background may be required. The electronic device may display a user interface (1115) that requests user input. For example, it may display a user interface (1115) that includes the phrase "Please select a photo." When user input selecting an image is received on the screen (1110), the electronic device may perform the next target action.

[0206] According to one embodiment, when a first target action is performed on a screen (1110), a screen (1120) corresponding to a second target action, which is the current target action, may be displayed. The electronic device may control the screen (1120) based on a command mapped to the second target action. The command mapped to the second target action may include a selection location and a selection command. For example, the selection location may include information about the location of an option button on the screen (1120), and the selection command may include a selection command (e.g., a single touch) to enter the next target action. The electronic device may display an object (1125) (e.g., object (1015) of FIG. 10, object (1025) of FIG. 10, object (1035) of FIG. 10, or object (1045) of FIG. 10) on the selection location to show which option has been selected. The electronic device may further display a user interface at the bottom of the screen (1120) indicating that a target action is being performed.

[0207] According to one embodiment, when a second target action is performed on the screen (1120), a screen (1130) corresponding to the third target action, which is the current target action, may be displayed. The electronic device may control the screen (1130) based on a command mapped to the third target action. The command mapped to the third target action may include a selection location and a selection command. For example, the selection location may include information about the location of the option to set as the background on the screen (1130), and the selection command may include a selection command (e.g., a single touch) to enter the next target action. The electronic device may display an object (1135) (e.g., object (1015) of FIG. 10, object (1025) of FIG. 10, object (1035) of FIG. 10, or object (1045) of FIG. 10) on the selection location to show which option has been selected. The electronic device may further display a user interface at the bottom of the screen (1130) indicating that a target action is being performed.

[0208] According to one embodiment, when a third target action is performed on the screen (1130), a screen (1140) corresponding to the fourth target action, which is the current target action, may be displayed. The electronic device may control the screen (1140) based on a command mapped to the fourth target action. The command mapped to the fourth target action may include a selection location and a selection command. For example, the selection location may include information about the location of the home screen option on the screen (1140), and the selection command may include a selection command (e.g., a single touch) to enter the next target action. The electronic device may display an object (1145) (e.g., the object (1015) of FIG. 10, the object (1025) of FIG. 10, the object (1035) of FIG. 10, or the object (1045) of FIG. 10) on the selection location to show which option has been selected. The electronic device may further display a user interface at the bottom of the screen (1140) indicating that a target action is being performed.

[0209] An electronic device for performing the operations of the present disclosure is described below.

[0210]

[0211] FIG. 12 is a drawing for explaining an electronic device that extracts operations from content according to one embodiment.

[0212] Referring to FIG. 12, an electronic device (1200) (e.g., the electronic device (101) of FIG. 1) performing the operations of the present disclosure is illustrated. The electronic device may include an AI framework (e.g., the AI ​​Framework (1320) of FIG. 13) having on-device AI processing capabilities. The AI ​​Framework (e.g., the AI ​​Framework (1320) of FIG. 13) may generate and learn a response to said user input using resources within the device, instead of sending or additionally sending user input received through a User Interface (e.g., the User Interface (1310) of FIG. 13) operating on the same device (e.g., the electronic device (101)) to a Generative AI Model (e.g., the Generative AI Model (1330) of FIG. 13) operating on an external device (e.g., a server (108)). The AI ​​Framework (e.g., the AI ​​Framework (1320) of FIG. 13)) may include a cross-application action module It may include a module (1210), a personal data managing module (1220), an AI module (1240), and an orchestration module (1280).

[0213] According to one embodiment, the cross-application action module (1210) determines one or more additional applications required for the operation of an executed application (e.g., an assistant app) and may connect or suggest operations between the app and at least one additional application, or between a plurality of additional applications. For example, the cross-application action module (1210) may execute one or more additional applications to be used to respond to a user request through the assistant app sequentially or at least partially and simultaneously. Additionally, the cross-application action module (1210) may communicate with the additional applications so that the result of the execution of one additional application (e.g., content) can be shared with other additional applications.

[0214] According to one embodiment, the personal data managing module (1220) may provide personal information (e.g., schedule, contact, or message information) about a user of the application (e.g., assistant app) or the additional application running on the device (e.g., electronic device (101)) or other related individuals (e.g., family or friends) to another module of the AI ​​Framework (e.g., AI Framework (1320) of FIG. 13) or a related module (e.g., Generative AI Model (1330) of FIG. 13) running on another device.

[0215] According to one embodiment, the AI ​​module (1240) may include at least one model among one or more AI models (e.g., GAN, VAE, LLM, LMM, LVM, or LAM) operated on an external device (e.g., server (108)) or a corresponding lightweight AI model. Additionally, for said model or said lightweight model, there may be, for example, a LoRA adaptor. The AI ​​module (1240) may be an on-device AI model.

[0216] According to one embodiment, the orchestration module (1280) may select one or more AI models to be used to obtain a response to user input (e.g., user query). For example, the orchestration module (1280) may select one or more AI models from among an AI module (1240), an AI model operating on an external device (e.g., server (108)) (e.g., Generative AI Model (1330) of FIG. 13), or a third AI model (not shown) operating on another external device. When multiple AI models are selected, the orchestration module (1280) may communicate with the selected models or devices so that the operation between the selected AI models and the processing of the results thereof can be coordinated between the relevant models or devices.

[0217] According to one embodiment, two or more modules of the electronic device (1200) (e.g., a cross-application action module (1210) and an orchestration module (1280)) may be implemented as a single module to maintain the same functionality. Various variations are possible.

[0218] According to one embodiment, the electronic device (1200) may further include a plurality of modules (1230, 1250, 1260, 1270). Only components related to the embodiments of the present disclosure are shown in the electronic device (1200). Accordingly, it is obvious to those skilled in the art that general-purpose modules other than the plurality of modules (1230, 1250, 1260, 1270) shown in FIG. 12 may be included.

[0219] In addition, the functions of the multiple modules (1230, 1250, 1260, 1270) described below do not necessarily have to be performed in each module. For example, there may be a module among the multiple modules (1230, 1250, 1260, 1270) that performs two or more functions. Therefore, it should be understood that the description of each module described below does not necessarily correspond one-to-one with each module.

[0220] According to one embodiment, the user UI module (1230) can determine at least a portion of the loaded content based on user input regarding the loaded content. The user UI module (1230) can receive input of a user's intention to apply at least a portion of the content included in the loaded content to an electronic device. Based on user input, the user UI module (1230) can determine how much of the content included in the loaded content will be applied to the electronic device. For example, the user UI module (1230) can receive a user's capture command to determine at least a portion of the loaded content.

[0221] According to one embodiment, the user intent analysis module (1250) can analyze at least a portion of the loaded content to determine whether the user's intent in selecting at least a portion of the loaded content is to apply it to an electronic device. The user intent analysis module (1250) can analyze at least a portion of the loaded content to determine whether the user input is intended to apply the content included in at least a portion of the loaded content to an electronic device. For example, the user intent analysis module (1250) can determine the user's intent using an AI module (1240). If it is determined that the user's intent is to apply at least a portion of the loaded content to an electronic device, the user UI module (1230) can be made to display a user interface (e.g., the user interface (335) of FIG. 3 or the user interface (435) of FIG. 4).

[0222] According to one embodiment, the content analysis module (1260) can identify actions from at least a portion of the loaded content by utilizing the AI ​​module (1240). The content analysis module (1260) can determine whether each of the identified actions can be performed on an electronic device by utilizing the AI ​​module (1240). The content analysis module (1260) can generate a list (e.g., the list (510) of FIG. 5 or the list (710) of FIG. 7) by determining candidate actions (e.g., candidate actions of FIG. 5 (530, 540, 550, 560, 570) and candidate actions of FIG. 7 (720, 730, 740, 750, 760)) by utilizing the AI ​​module (1240). The candidate actions may be actions that can be performed on an electronic device. Among the candidate actions, target actions that are determined to be applied to the electronic device can be performed by an artificial intelligence model (e.g., AI module (1240)).

[0223] According to one embodiment, the electronic device (1200) can generate commands mapped to candidate actions by utilizing an AI module (1240). The AI ​​module (1240) may include an AI model trained to generate commands mapped to candidate actions. The AI ​​model included in the AI ​​module (1240) may have learned the location (e.g., coordinates) of a button or menu that must be selected on the screen to perform the candidate action.

[0224] According to one embodiment, the control module (1270) can perform at least one target action determined based on a list generated by the content analysis module (1260). At least one target action can be determined through the list. The control module (1270) can perform at least one target action based on commands mapped to at least one target action. The control module (1270) can perform at least one target action based on a selection location and a selection command included in the commands mapped to at least one target action. If at least one target action requires user input, the control module (1270) can display a user interface (e.g., the user interface (1115) of FIG. 11) that requests user input.

[0225] According to one embodiment, the control module (1270) can apply target actions to the electronic device according to a list using the AI ​​module (1240). For example, depending on the selection location and selection command included in the command, the AI ​​module (1240) can touch a button or menu that must be touched to perform the target action.

[0226] According to one embodiment, the AI ​​module (1240) may be used in the user intent analysis module (1250), the content analysis module (1260), and the control module (1270). For example, the AI ​​module (1240) may be used to determine whether an action identified in the content analysis module (1260) can be used in the electronic device and / or whether an alternative action exists. For example, the AI ​​module (1240) may be used when analyzing at least a portion of the loaded content, when determining whether an identified action can be performed in the electronic device, when finding whether there is another version or other alternative if the identified action cannot be performed in the electronic device, and / or when finding a location to be selected to apply the target action to the electronic device.

[0227] According to one embodiment, the AI ​​module (1240) may include a plurality of AI models. The plurality of AI models may include generative artificial intelligence models. For example, the AI ​​module (1240) may include an AI model that analyzes user intent, an AI model that identifies actions from at least part of the content, an AI model that determines whether the identified actions can be performed on an electronic device, an AI model that determines alternative actions, and an AI model that generates commands mapped to candidate actions so that the candidate actions can be performed on an electronic device.

[0228] According to one embodiment, an AI model for determining whether an identified action can be performed on an electronic device and / or an AI model for determining an alternative action may be a model trained on the manual of the electronic device. The AI ​​model for determining whether an identified action can be performed on an electronic device may output whether the action is possible on the electronic device when it receives the specification information of the electronic device and the identified action as input. The AI ​​model for determining an alternative action may output an alternative action that is possible on the electronic device when it receives the specification information of the electronic device and the identified action as input.

[0229] According to one embodiment, the AI ​​model (1241) may be configured to change to a command system available for use in the electronic device. The AI ​​model (1241) may be configured to be a model trained on manuals of electronic devices made by the same manufacturer. Once the sequence of identified operations is determined, the analyzed sequence of operations, along with the current model name and / or version, may be input into the AI ​​model (1241). The AI ​​model (1241) determines whether the identified operations can be performed on the electronic device, and if the operations cannot be performed and no alternative operations exist, it may inform the user that the operations cannot be performed.

[0230] However, the AI ​​models described above are merely examples and should not be interpreted as limiting the embodiments. For example, it is obvious to those skilled in the art that there may exist AI models that perform two or more of the functions described above.

[0231] According to one embodiment, an AI model (1241) may be connected to a plurality of adapters (1243). Adapters (e.g., LoRA (low-Rank Adaptation)) may enable the AI ​​model (1241) to incorporate new data without retraining the AI ​​model (1241). For example, if a new version of an electronic device (1200) is deployed, the existing AI model (1241) may not know which operations the electronic device can or cannot perform in the new version. By adding an adapter (1243) for the new version to the AI ​​model (1241), the AI ​​model can utilize data for the new version without new training.

[0232] According to one embodiment, the adapter (1243) may be used to analyze operations that are not present in the AI ​​model (1241). For example, the information found by the user may be information about a higher version than the current version of the electronic device (1200), information about a different model from the electronic device (1200), or information about an electronic device from another manufacturer. The above information may not exist at the time of the release of the electronic device (1200), so the AI ​​model may not have such information. Rather than retraining the AI ​​model (1241) to reflect the above information, the adapter (1243) may be added to allow the AI ​​model (1241) to reflect new data without retraining.

[0233] According to one embodiment, as will be described later in FIG. 14, an electronic device (1200) can control other electronic devices. For example, a user's new electronic device (e.g., TV, tablet PC, air conditioner, etc.) can be registered to the electronic device (1200). When a new electronic device is registered, the electronic device (1200) can add an adapter (1243). For example, when an air conditioner is purchased and registered in a management application (e.g., smart things), the electronic device (1200) can download and install an adapter (1243) related to the air conditioner from a server. Once the adapter (1243) is installed, an AI model (1241) can analyze the operation of the air conditioner and generate commands related to the order of operation, etc.

[0234] According to one embodiment, the electronic device (1200) can determine a target operation for an external electronic device (e.g., the external electronic device (104) of FIG. 1) and control the external electronic device by transmitting a command mapped to the target operation to the external electronic device. The control of the external electronic device is described later in FIG. 14. When a new external electronic device is connected, the AI ​​model (1241) may not know which operations the new external electronic device can or cannot perform. By downloading an adapter (1243) for the new external electronic device from a server (e.g., the server (108) of FIG. 1) and adding it to the AI ​​model (1241), the AI ​​model (1241) can identify operations that are possible or impossible to perform on the new external electronic device.

[0235]

[0236] FIG. 13 is a generative artificial intelligence (AI) system (1300) according to one embodiment.

[0237] Referring to FIG. 13, a generative artificial intelligence system (1300) may include a User Interface (1310), an AI Framework (1320), a Generative AI Model (1330), a Knowledge Repository (1340), and an Application / Service Module (1350). These components may be operated on one or more of an electronic device 101, an external electronic device (102 or 104), or a server (108). For example, the User Interface (1310) and the AI ​​Framework (1320) may be operated on the electronic device (101), and the Knowledge Repository (1340) and the Generative AI Model (1330) may be operated on the server (108).

[0238] According to one embodiment, the User Interface (1310) may receive user input (e.g., user query). User input may be received in the form of text, images, voice (e.g., natural language), video, menu selection, or a combination thereof. The User Interface (1310) may include various context information (e.g., running application or user location) related to the generative artificial intelligence system (1300) at the time the user input is received, in addition to or instead of the user input. The User Interface (1310) may provide the user input or the context information to the AI ​​Framework (1320) and provide the result of processing therefrom to the user, for example, through the AI ​​Framework (1320). According to one embodiment, in addition to user input, the electronic device may provide context information obtained using information included on the screen to the AI ​​Framework (1320). The result may be provided in the form of text, images, voice, video, an action requested by the user (e.g., launching a specified function or app), or a combination thereof.

[0239] According to one embodiment, the AI ​​Framework (1320) can identify (e.g., estimate) a user intent based on at least part of user input or context information received from the User Interface (1310), control each of the relevant modules (e.g., 1321, 1323, or 1325) to perform a function or action corresponding to the identified user intent, and coordinate collaboration between two or more modules. The AI ​​Framework (1320) may include a Prompt Design Module (1321), an API / Plug-in Management Module (1323), and an Output Modification Module (1325), as illustrated in FIG. 2.

[0240] According to one embodiment, the Prompt Design Module (1321) can generate a prompt to be input to the Generative AI Model (1330) based at least partially on user input or context information received from the User Interface (1310). For example, the Prompt Design Module (1321) can generate a prompt using user preferences, a prompt library, or prompt examples stored in the Knowledge Repository (1340) based at least partially on user input or context information.

[0241] According to one embodiment, the API / Plug-in Management Module (1323) may communicate, for example, via an API, with various resources (e.g., Knowledge Repository (1340)) that provide said additional information when there is a request for said additional information in relation to user input. Additionally or generally, when a specified action (e.g., function, app, or service) is performed in response to said user input, the API / Plug-in Management Module (1323) may request the Application / Service Module (1350) to perform said specified action via a corresponding API. The API / Plug-in Management Module (1323) may provide information obtained from the Knowledge Repository (1340), the Application / Service Module (1350), or another external resource to the Prompt Design Module (1321). That obtained information may be used by the Prompt Design Module (1321) to generate a prompt along with the user input, or provided to the Generative AI Model (1330).

[0242] According to one embodiment, the Output Modification Module (1325) can fine-tune the results obtained through the Generative AI Model (1330) as at least part of the response to user input (e.g., user query). For example, the Output Modification Module (1325) can determine whether the content of the response obtained through the Generative AI Model (1330) is appropriate as a response to a request made by the user input. For example, the Output Modification Module (1325) can determine the degree of relevance, degree of bias (e.g., political or social bias), or degree of harmfulness (e.g., sexual or profanity) of the difference between the response obtained through the Generative AI Model (1330) and the user input. Additionally or generally, the Output Modification Module (1325) can request that additional AI processing be performed on the obtained response, or provide the user with a hint to avoid unwanted output. For example, additional prompts can be generated through the Prompt Design Module (1321) to obtain a response again through the Generative AI Model (1330).

[0243] According to one embodiment, the Generative AI Model (1330) may form at least part of an artificial intelligence neural network and may include a model that generates images or a model that generates language. The image generation model may include, for example, a generative adversarial network (GAN), a variational autoencoder (VAE), or a Diffusion-based model using a VAE and a Transformer. The language generation model may include, for example, a large language model (LLM), a large multimodal model (LMM), a large vision model (LVM), or a large action model (LAM). The LAM may automatically generate actions for an environment (e.g., a robot, a car, an electronic device (101), or a program (140)). Additionally, for at least some AI models (e.g., LLM), there may be a low-rank adaptation (LoRA) adaptor that is fine-tuned for, for example, a specific task or a specific situation.

[0244]

[0245] FIG. 14 is a drawing for explaining the operation of an electronic device and an external electronic device according to one embodiment.

[0246] Referring to FIG. 14, an electronic device (1401) (e.g., the electronic device (101) of FIG. 1) and an external electronic device (1404) (e.g., the external electronic device (104) of FIG. 1) are shown.

[0247] According to one embodiment, an electronic device (1401) can control an external electronic device (1404). The external electronic device (1404) may include various computing devices such as mobile phones, smartphones, tablets, e-book devices, laptops, personal computers, desktops, workstations, or servers; various wearable devices such as smart watches, smart glasses, or HMDs (head-mounted displays); various home appliances such as smart speakers, smart TVs, or smart refrigerators; smart cars, smart kiosks, IoT (Internet of Things) devices, WADs (Walking Assist Devices), drones, or robots. For example, if the electronic device (1401) is a smartphone and the external electronic device (1404) is a TV, the electronic device (1401) can control the TV by determining a target action based on a blog describing a method for controlling the TV.

[0248] Prior to operation (1410), it may be necessary to register the external electronic device (1404) to the electronic device (1401). Once the external electronic device (1404) is registered to the electronic device (1401), the electronic device can download an adapter for the external electronic device (1404) (e.g., the adapter (1223) of FIG. 12) through a server (e.g., the server (108) of FIG. 1). The electronic device can add the downloaded adapter to multiple AI models (e.g., the AI ​​model (1221) of FIG. 12).

[0249] According to one embodiment, the electronic device can select an external electronic device (1404) from a list of external electronic devices.

[0250] In operation (1410), the electronic device (1401) may request information about the external electronic device from the external electronic device (1404). The information about the external electronic device may include the software version, manufacturer, versions and model names of the applications as specification information of the external electronic device.

[0251] In operation (1420), the external electronic device (1404) can transmit information of the external electronic device to the electronic device (1401).

[0252] In operation (1430), the electronic device can select at least a portion of the content loaded into the electronic device to be displayed on the display module based on the first user input.

[0253] According to one embodiment, the loaded content may include a method for controlling an external electronic device (1404). A detailed description of the method for selecting at least a portion of the content is omitted as described above in FIGS. 3 and FIGS. 4.

[0254] According to one embodiment, the electronic device can determine the user's intention based on at least a portion of loaded content containing information about an external electronic device. If the electronic device determines that the user's intention is to control the external electronic device, it may display a user interface asking whether to apply it to the external electronic device. For example, the electronic device may display "Do you want to apply it to another registered device?". The electronic device may perform an operation (1440) when it receives a confirmation command through a confirmation button on the user interface.

[0255] In operation (1440), the electronic device can determine candidate operations applicable to an external electronic device communicating with the electronic device based on at least a portion of the loaded content.

[0256] According to one embodiment, the electronic device can extract identified actions from at least a portion of the loaded content. The electronic device can determine whether each of the identified actions is a first action that can be performed by the external electronic device based on an artificial intelligence model that has learned actions that the external electronic device can perform (e.g., the AI ​​module (1240) of FIG. 12).

[0257] According to one embodiment, if there is a second operation among the identified operations that cannot be performed by an external electronic device, the electronic device can determine whether there is a third operation that can replace the second operation based on a learned artificial intelligence model.

[0258] The method for determining candidate actions is described in detail in FIGS. 5 to 7, so a detailed explanation is omitted.

[0259] In operation (1450), the electronic device can map commands and operations to cause an external electronic device to perform candidate operations.

[0260] According to one embodiment, an electronic device can generate instructions that cause an external electronic device to perform candidate actions based on an artificial intelligence model (e.g., the AI ​​module (1240) of FIG. 12). The electronic device can map the candidate actions to the instructions.

[0261] In operation (1460), the electronic device can determine at least one target operation among candidate operations based on the second user input. A detailed explanation of the method for determining at least one target operation is omitted as described above in FIG. 5.

[0262] In operation (1470), the electronic device can transmit commands mapped to at least one target operation to an external electronic device (1404).

[0263] In operation (1480), the external electronic device can perform at least one target operation based on commands mapped to at least one target operation.

[0264] According to one embodiment, the detailed description of the performance of at least one target operation is omitted as described above in FIGS. 9 to 11. According to one embodiment, the electronic device can control an external electronic device based on loaded content according to operations (1410) to (1480).

[0265]

[0266] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may include a memory (e.g., the memory (130) of FIG. 1) that stores instructions. The electronic device may include at least one processor (e.g., the processor (120) of FIG. 1) that executes instructions. When the instructions are executed individually or collectively by at least one processor, the electronic device may select at least a portion of the content loaded into the electronic device to be displayed on a display module (e.g., the display module (160) of FIG. 1) based on a first user input. When the instructions are executed individually or collectively by at least one processor, the electronic device may determine candidate actions applicable to the electronic device based on at least a portion of the loaded content. When the instructions are executed individually or collectively by at least one processor, the electronic device may map the candidate actions to instructions that cause the electronic device to perform the candidate actions. When the instructions are executed individually or collectively by at least one processor, the electronic device may determine at least one target operation among candidate operations based on a second user input. When the instructions are executed individually or collectively by at least one processor, the electronic device may perform at least one target operation based on instructions mapped to at least one target operation.

[0267] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device may display a list in which candidate operations are sequentially displayed (e.g., the list (510) of FIG. 5 or the list (510) of FIG. 7). When instructions are executed individually or collectively by at least one processor, the electronic device may determine at least one target operation among the candidate operations based on a second user input to the list.

[0268] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device may be able to select all or part of the loaded content to extract candidate actions based on a first user input.

[0269] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device may be able to extract identified actions from at least a portion of loaded content. When instructions are executed individually or collectively by at least one processor, the electronic device may be able to determine whether each of the identified actions is a first action that can be performed by the electronic device. When instructions are executed individually or collectively by at least one processor, the electronic device may be able to determine whether there is a third action that can replace a second action if there is a second action among the identified actions that is an action that cannot be performed by the electronic device.

[0270] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device may determine whether each of the identified operations is a first operation that the electronic device can perform based on an artificial intelligence model that has learned operations that the electronic device can perform. When instructions are executed individually or collectively by at least one processor, the electronic device may determine whether there exists a third operation that can replace a second operation based on an artificial intelligence model.

[0271] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device may determine candidate operations by replacing the second operation with the third operation among the identified operations, where a second operation exists and a third operation capable of replacing the second operation exists.

[0272] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device may display a user interface indicating that a second operation exists and has been replaced by a third operation.

[0273] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device may display a user interface (e.g., user interface (810) of FIG. 8) on loaded content indicating that the identified operations cannot be performed when the second operation does not exist.

[0274] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device may be configured to sequentially display screens corresponding to at least one target operation (e.g., the screens of FIG. 10 (1010, 1020, 1030, 1040) or the screens of FIG. 11 (1110, 1120, 1130, 1140)) on a display module as the electronic device is sequentially controlled according to the mapped instructions.

[0275] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device may be able to display an area controlled according to instructions mapped on screens using visual objects (e.g., objects of FIG. 10 (1015, 1025, 1035, 1045) or objects of FIG. 11 (1125, 1135, 1145)).

[0276] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device may be able to display a user interface (e.g., user interface (1115) of FIG. 11) requesting user input on a screen corresponding to a target operation for at least one target operation that requires user input.

[0277] According to one embodiment, the electronic device may include a memory for storing instructions. The electronic device may include at least one processor for executing instructions. When the instructions are executed individually or collectively by at least one processor, the electronic device may select at least a portion of the content loaded into the electronic device to be displayed on a display module based on a first user input. When the instructions are executed individually or collectively by at least one processor, the electronic device may determine candidate actions applicable to an external electronic device communicating with the electronic device based on at least a portion of the loaded content. When the instructions are executed individually or collectively by at least one processor, the electronic device may map the candidate actions to instructions that cause the external electronic device to perform the candidate actions. When the instructions are executed individually or collectively by at least one processor, the electronic device may determine at least one target action among the candidate actions based on a second user input. When the instructions are executed individually or collectively by at least one processor, the electronic device may transmit instructions mapped to at least one target action to the external electronic device to cause the external electronic device to perform at least one target action.

[0278] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device may be able to extract identified actions from at least a portion of loaded content. When instructions are executed individually or collectively by at least one processor, the electronic device may be able to determine whether each of the identified actions is a first action that can be performed by an external electronic device based on an artificial intelligence model that has learned actions that an external electronic device can perform.

[0279] According to one embodiment, when instructions are executed individually or collectively by at least one processor, the electronic device may determine, based on a learned artificial intelligence model, whether there exists a third operation that can replace a second operation, if there exists a second operation among the identified operations that cannot be performed by an external electronic device.

[0280] According to one embodiment, a method of operating an electronic device may include an operation of selecting at least a portion of content loaded into the electronic device to be displayed on a display module based on a first user input. A method of operating an electronic device may include an operation of determining candidate operations applicable to the electronic device based on at least a portion of the loaded content. A method of operating an electronic device may include an operation of mapping candidate operations to instructions that cause the electronic device to perform the candidate operations. A method of operating an electronic device may include an operation of determining at least one target operation among the candidate operations based on a second user input. A method of operating an electronic device may include an operation of performing at least one target operation based on instructions mapped to at least one target operation.

[0281] According to one embodiment, the method of operating an electronic device may further include the operation of displaying a list in which candidate operations are sequentially displayed. The operation of determining at least one target operation may determine at least one target operation among the candidate operations based on a second user input to the list.

[0282] According to one embodiment, the operation of selecting at least a portion of the loaded content may select all or part of the loaded content to extract candidate operations based on a first user input.

[0283] According to one embodiment, the operation of extracting candidate operations may include the operation of extracting operations identified from at least a portion of loaded content. The operation of extracting candidate operations may include the operation of determining whether each of the identified operations is a first operation that can be performed on an electronic device. The operation of extracting candidate operations may include the operation of determining whether a third operation capable of replacing a second operation exists among the identified operations, if a second operation that cannot be performed on an electronic device exists.

[0284] According to one embodiment, the operation of determining whether a first operation is a first operation can determine whether each of the identified operations is a first operation that can be performed by the electronic device based on an artificial intelligence model that has learned operations that the electronic device can perform. The operation of determining whether a third operation that can replace a second operation exists can determine whether a third operation that can replace a second operation exists based on an artificial intelligence model.

[0285] According to one embodiment, a non-transient computer-readable recording medium may store one or more computer programs including instructions that execute an operation to select at least a portion of content loaded into an electronic device to be displayed on a display module based on a first user input. A non-transient computer-readable recording medium may store one or more computer programs including instructions that execute an operation to determine candidate operations applicable to the electronic device based on at least a portion of the loaded content. A non-transient computer-readable recording medium may store one or more computer programs including instructions that cause the electronic device to perform candidate operations and instructions that execute an operation to map the candidate operations. A non-transient computer-readable recording medium may store one or more computer programs including instructions that execute an operation to determine at least one target operation among candidate operations based on a second user input. A non-transient computer-readable recording medium may store one or more computer programs including instructions that execute an operation to perform at least one target operation based on instructions mapped to at least one target operation.

[0286]

[0287] Furthermore, the embodiments of the present invention disclosed in this specification and drawings are merely specific examples provided to facilitate the explanation of the technical content according to the embodiments of the present invention and to aid in understanding the embodiments of the present invention, and are not intended to limit the scope of the embodiments of the present invention. Accordingly, the scope of the various embodiments of the present invention should be interpreted to include all modifications or variations derived based on the technical concept of the various embodiments of the present invention, in addition to the embodiments disclosed herein.

Claims

1. In an electronic device (101), At least one processor (120) including a processing circuit; and Memory (130) for storing instructions Includes, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) is enabled, Select at least a portion of the content loaded into the electronic device (101) to be displayed on the display module (160) based on the first user input, and Candidate operations applicable to the electronic device (101) are determined based on at least a portion of the loaded content, and Maps the candidate operations to commands that cause the electronic device (101) to perform the candidate operations, and Based on the second user input, at least one target action is determined among the above candidate actions, and Performing the at least one target operation based on the commands mapped to the at least one target operation, Electronic device (101).

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) is enabled, Displays a list (510, 710) in which the above candidate actions are displayed sequentially, and Determining at least one target action among the candidate actions based on the second user input for the above list (510, 710), Electronic device (101).

3. In either Paragraph 1 or Paragraph 2, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) is enabled, Selecting all or part of the loaded content to extract the candidate actions based on the first user input, Electronic device (101).

4. In any one of paragraphs 1 through 3, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) is enabled, Extracting actions identified from at least a portion of the loaded content, and Determining whether each of the above identified operations is a first operation that can be performed in the electronic device (101), and If there is a second operation among the identified operations that cannot be performed by the electronic device (101), determine whether there is a third operation that can replace the second operation. Electronic device (101).

5. In any one of paragraphs 1 through 4, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) is enabled, Based on an artificial intelligence model that has learned the operations that the electronic device (101) can perform, determining whether each of the identified operations is the first operation that can be performed by the electronic device (101), and based on the artificial intelligence model, determining whether there exists a third operation that can replace the second operation. Electronic device (101).

6. In any one of paragraphs 1 through 5, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) is enabled, If the second action exists and the third action capable of replacing the second action exists, the candidate actions are determined by replacing the second action with the third action among the identified actions. Electronic device (101).

7. In any one of paragraphs 1 through 6, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) is enabled, Displaying a user interface indicating that the above second operation exists and has been replaced by the above third operation, Electronic device (101).

8. In any one of paragraphs 1 through 7, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) is enabled, If the above second operation does not exist, a user interface (810) indicating that the above identified operations cannot be performed is displayed on the loaded content. Electronic device (101).

9. In any one of paragraphs 1 through 8, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) is enabled, As the electronic device (101) is sequentially controlled according to the mapped commands, screens (1010, 1020, 1030, 1040, 1110, 1120, 1130, 1140) corresponding to at least one target operation are sequentially displayed on the display module (160). Electronic device (101).

10. In any one of paragraphs 1 through 9, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) is enabled, Displaying the area controlled according to the mapped commands in the above screens (1010, 1020, 1030, 1040, 1110, 1120, 1130, 1140) using a visual object (1015, 1025, 1035, 1045, 1125, 1135, 1145), Electronic device (101).

11. In any one of paragraphs 1 through 10, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) is enabled, For a target action requiring user input among at least one of the above target actions, a user interface (1115) requesting user input is displayed on a screen corresponding to the target action. Electronic device (101).

12. In the method of operating the electronic device (101), An operation of selecting at least a portion of the content loaded into the electronic device (101) to be displayed on the display module (160) based on the first user input; An operation to determine candidate operations applicable to the electronic device (101) based on at least a portion of the loaded content; Instructions to cause the electronic device (101) to perform the candidate operations and an operation to map the candidate operations; An action of determining at least one target action among the candidate actions based on a second user input; and An operation that performs the at least one target operation based on commands mapped to the at least one target operation. including, Method of operation.

13. In Paragraph 12, An operation to display a list (510, 710) in which the above candidate operations are displayed sequentially. Includes more, The operation determining at least one target operation above is, Determining at least one target action among the candidate actions based on the second user input for the above list (510, 710), Method of operation.

14. In either Paragraph 12 or Paragraph 13, The action of selecting at least a portion of the loaded content is, Selecting all or part of the loaded content to extract the candidate actions based on the first user input Method of operation.

15. In any one of paragraphs 12 through 14, The operation of extracting the above candidate operations is, An action of extracting actions identified from at least a portion of the loaded content; An operation to determine whether each of the above identified operations is a first operation that can be performed in the electronic device (101); and If there is a second operation among the identified operations that cannot be performed by the electronic device (101), an operation to determine whether there is a third operation that can replace the second operation. including, Method of operation.

Images