Electronic device for executing user request for access-restricted content, operation method thereof, and recording medium

Abstract

This electronic device may comprise: at least one processor including processing circuitry; and a memory for storing instructions. The instructions may cause, when executed individually or collectively by the at least one processor, the electronic device to: provide first content through a first application; while providing the first content, identify a user request related to the first content; on the basis of the user request, attempt to acquire a first content element from the first content; on the basis, at least in part, of a determination that the acquisition of the first content element is restricted, acquire at least one second content element corresponding to the first content element; cause a response to the user request to be generated using the at least one second content element instead of the first content element; and provide the response. Various other embodiments are also possible.

Description

Electronic device for performing a user request for access-restricted content, the method of operation thereof, and a recording medium

[0001] The present disclosure relates to an electronic device for performing a user request for access-restricted content according to one embodiment, a method of operation thereof, and a recording medium.

[0002] The electronic device can provide various types of content (e.g., still images, videos, audio, documents, web pages, application screens).

[0003] An electronic device can receive content from another electronic device (e.g., a server) and display it through the electronic device's display. For example, the electronic device can capture and save the screen displaying the content upon a user request.

[0004] An electronic device receives a user request (e.g., text input or user utterance) and can process the received user request. The electronic device can identify the data necessary to process the user request.

[0005] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0006] According to one embodiment, the electronic device may include at least one processor comprising a processing circuit and a memory for storing instructions. When the instructions are executed individually or collectively by at least one processor, they may cause the electronic device to provide a first content through a first application. When the instructions are executed individually or collectively by at least one processor, they may cause the electronic device to check for a user request related to the first content while providing the first content. When the instructions are executed individually or collectively by at least one processor, they may cause the electronic device to attempt to obtain a first content element from the first content based on the user request. When the above instructions are executed individually or collectively by at least one processor, they may cause the electronic device to acquire at least one second content element corresponding to the first content element, based at least in part on a determination that the acquisition of the first content element is restricted. When the above instructions are executed individually or collectively by at least one processor, they may cause the electronic device to generate a response to the user request using the at least one second content element instead of the first content element. When the above instructions are executed individually or collectively by at least one processor, they may cause the electronic device to provide the response.

[0007] According to one embodiment, a method of operating an electronic device may include an operation of providing a first content through a first application. The method may include an operation of checking a user request related to the first content while providing the first content. The method may include an operation of attempting to obtain a first content element from the first content based on the user request. The method may include an operation of obtaining at least one second content element corresponding to the first content element based at least partially on a determination that the acquisition of the first content element is restricted. The method may include an operation of causing a response to the user request to be generated using the at least one second content element instead of the first content element. The method may include an operation of providing the response.

[0008] According to one embodiment, in a non-transitory computer-readable recording medium storing instructions, the instructions may cause the electronic device to perform at least one operation when executed individually or collectively by at least one processor of the electronic device. The at least one operation may include an operation of providing a first content through a first application. The at least one operation may include an operation of checking a user request related to the first content while providing the first content. The at least one operation may include an operation of attempting to obtain a first content element from the first content based on the user request. The at least one operation may include an operation of obtaining at least one second content element corresponding to the first content element based at least partially on a determination that the obtaining of the first content element is restricted. The at least one operation may include an operation that causes a response to the user request to be generated using the at least one second content element instead of the first content element. The at least one operation may include an operation that provides the response.

[0009] According to one embodiment, the electronic device may include a display module including a display, at least one processor including a processing circuit, and a memory for storing instructions. When the instructions are executed individually or collectively by at least one processor, the electronic device may cause the electronic device to display a first screen of a first application. The first screen may include first content. When the instructions are executed individually or collectively by at least one processor, the electronic device may cause the electronic device to check for a user request to create new content using the first content of the first screen. When the instructions are executed individually or collectively by at least one processor, the electronic device may cause the electronic device to check whether the first content is restricted access content based on the fact that data related to the first content is required to perform the user request. Restricted access content may be content in which access to original reference data is restricted. When the above instructions are executed individually or collectively by at least one processor, they may cause the electronic device to acquire replacement data to replace the original reference data based on the fact that the first content is the access-restricted content. When the above instructions are executed individually or collectively by at least one processor, they may cause the electronic device to perform the user request using the replacement data.

[0010] According to one embodiment, a method of operating an electronic device may include an operation of displaying a first screen of a first application. The first screen may include first content. The method may include an operation of confirming a user request to create new content using the first content of the first screen. The method may include an operation of confirming whether the first content is restricted access content based on the fact that data related to the first content is required to perform the user request. Restricted access content may be content in which access to original reference data is restricted. The method may include an operation of obtaining replacement data to replace the original reference data based on the fact that the first content is restricted access content. The method may include an operation of performing the user request using the replacement data.

[0011] According to one embodiment, in a non-transitory computer-readable recording medium for storing instructions, the instructions may cause the electronic device to perform at least one operation when executed individually or collectively by at least one processor of the electronic device. The at least one operation may include an operation of displaying a first screen of a first application. The first screen may include first content. The at least one operation may include an operation of confirming a user request to create new content using the first content of the first screen. The at least one operation may include an operation of confirming whether the first content is restricted access content based on the fact that data related to the first content is required to perform the user request. Restricted access content may be content in which access to original reference data is restricted. The at least one operation may include an operation of obtaining replacement data to replace the original reference data based on the fact that the first content is restricted access content. The above at least one operation may include an operation to perform the user request using the alternative data.

[0012] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment.

[0013] FIG. 2a is a block diagram of a generative artificial intelligence (AI) system according to one embodiment.

[0014] FIG. 2b is a block diagram of an AI Framework according to one embodiment.

[0015] FIG. 3 is a flowchart of a method of operation of an electronic device according to one embodiment.

[0016] FIG. 4a is a flowchart of a method of operation of an electronic device according to one embodiment.

[0017] FIG. 4b is a diagram illustrating the operation of an electronic device according to one embodiment.

[0018] FIG. 4c is a drawing illustrating the operation of an electronic device according to one embodiment.

[0019] FIG. 4d is a drawing illustrating the operation of an electronic device according to one embodiment.

[0020] FIG. 5 is a diagram illustrating the operation of an electronic device according to one embodiment.

[0021] FIG. 6 is a diagram illustrating the operation of an electronic device according to one embodiment.

[0022] FIG. 7 is a diagram illustrating the operation of an electronic device according to one embodiment.

[0023] FIG. 8 is a flowchart of a method of operation of an electronic device according to one embodiment.

[0024] FIG. 9 is a diagram illustrating the operation of an electronic device according to one embodiment.

[0025] FIG. 10 is a drawing illustrating the operation of an electronic device according to one embodiment.

[0026] FIG. 11 is a flowchart of a method of operation of an electronic device according to one embodiment.

[0027] FIG. 12 is a drawing illustrating the operation of an electronic device according to one embodiment.

[0028] FIG. 13 is a flowchart of a method of operation of an electronic device according to one embodiment.

[0029] FIG. 14a is a drawing illustrating the operation of an electronic device according to one embodiment.

[0030] FIG. 14b is a flowchart of a method of operation of an electronic device according to one embodiment.

[0031] FIG. 15 is a flowchart of a method of operation of an electronic device according to one embodiment.

[0032] FIG. 16 is a drawing illustrating the operation of an electronic device according to one embodiment.

[0033] FIG. 17 is a flowchart of a method of operation of an electronic device according to one embodiment.

[0034] FIG. 18 is a flowchart of a method of operation of an electronic device according to one embodiment.

[0035] FIG. 19 is a drawing illustrating the operation of an electronic device according to one embodiment.

[0036] FIG. 20 is a flowchart of a method of operation of an electronic device according to one embodiment.

[0037] FIG. 21 is a flowchart of a method of operation of an electronic device according to one embodiment.

[0038] FIG. 22 is a flowchart of a method of operation of an electronic device according to one embodiment.

[0039] FIG. 23 is a drawing illustrating the operation of an electronic device according to one embodiment.

[0040] FIG. 24 is a flowchart of a method of operation of an electronic device according to one embodiment.

[0041] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to one embodiment.

[0042] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0043] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), a secure processing unit (SPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0044] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network (DQN), or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0045] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0046] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146). One or more related applications may form a service configured to handle a series of user requests.

[0047] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0048] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0049] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0050] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0051] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0052] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0053] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0054] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0055] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0056] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) may be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0057] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0058] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., local area network (LAN) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0059] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. The NR access technology can support enhanced mobile broadband (Embb), massive machine type communications (mMTC), or ultra-reliable and low-latency communications (URLLC). The wireless communication module (192) can support high-frequency bands (e.g., mmWave bands) to achieve high data transmission rates, for example. The wireless communication module (192) can support various technologies for securing performance in high-frequency bands, for example, beamforming, multiple-input and multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or large-scale antennas. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., electronic device (104)), or a network system (e.g., a second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0060] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0061] According to one embodiment, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0062] At least some of the above components can be connected to each other and exchange signals (e.g., commands or data) through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI).

[0063] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least a part of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to the electronic device (101).

[0064] The electronic device (101) may process the above results, either as is or additionally, and provide them as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services, for example, by using distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199). The electronic device (101) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0065] The operations of the electronic device (101) can be described in detail with reference to the embodiments described above (e.g., embodiments of FIG. 1) and embodiments described below (e.g., embodiments of FIG. 2a, FIG. 2b, FIG. 3 to FIG. 22). Each embodiment is disclosed in a separate figure and a separate paragraph, but this is for convenience of explanation only, and at least some of the embodiments described above and at least some of the embodiments described below may be applied together. At least some of the embodiments described above and at least some of the embodiments described below may be omitted.

[0066] In this document, the electronic device (101) performing a specific operation may mean that a processor (120), such as a microcontrolling unit (MCU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a microprocessor, or an application processor (AP), performs a specific operation. According to one embodiment, the processor (120) may include a processing circuit. The electronic device (101) performing a specific operation may mean that the processor (120) controls other hardware to perform a specific operation. The electronic device (101) performing a specific operation may mean that the processor (120) or other hardware is caused to perform a specific operation as at least one instruction for performing a specific operation, which was stored in the storage circuit (e.g., memory (130)) of the electronic device (101), is executed. The at least one instruction stored in the memory (130) of the electronic device (101) may cause the electronic device (101) to perform at least one operation, either individually or collectively, when executed by the processor (120).

[0067] Functions related to artificial intelligence according to the present disclosure may be operated through a processor (120) and a memory (130). The processor (120) may be composed of one or more processors (120). In this case, the one or more processors (120) may be general-purpose processors such as a CPU, AP, DSP (Digital Signal Processor), graphics-dedicated processors such as a GPU, VPU (Vision Processing Unit), or artificial intelligence-dedicated processors such as an NPU. The one or more processors (120) control input data to be processed according to predefined operation rules or artificial intelligence models stored in the memory (130). Alternatively, if the one or more processors (120) are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0068] The predefined operating rules or artificial intelligence model are characterized by being created through learning. Here, being created through learning means that a predefined operating rules or artificial intelligence model is created by a basic artificial intelligence model being trained using multiple learning data by a learning algorithm to perform a desired characteristic (or purpose). The artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of operations of the previous layer and the multiple weights. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is performed, or it may be performed through a separate server (e.g., server (108) of FIG. 1) and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0069] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0070] In the method of operation of an electronic device (101) according to the present disclosure, the electronic device (101) can recognize speech. According to one embodiment, the electronic device (101) can recognize a user's speech by receiving an analog voice signal through an input module (150) (e.g., a microphone) and can interpret the intent of the speech included in the recognized speech. According to one embodiment, the electronic device (101) can recognize audio included in a video and interpret the intent of the speech included in the recognized audio. The electronic device (101) can convert a portion of the speech into computer-readable text using an Automatic Speech Recognition (ASR) model. It can obtain the intent of the speech by interpreting the converted text using a Natural Language Understanding (NLU) model. Here, the ASR model or the NLU model may be an artificial intelligence model. The artificial intelligence model may be processed by a processor (120) (e.g., an artificial intelligence dedicated processor) designed with a hardware structure specialized for processing the artificial intelligence model. The artificial intelligence model may be created through learning. Linguistic understanding is a technology that recognizes, applies, and processes human language and text, and includes Natural Language Processing, Machine Translation, Dialog Systems, Question Answering, and Speech Recognition / Synthesis.

[0071] In the method of operation of an electronic device (101) according to the present disclosure, the electronic device (101) can recognize an image. According to one embodiment, the electronic device (101) can obtain output data that recognizes an image by using image data as input data for an artificial intelligence model. The artificial intelligence model can be created through learning. Visual understanding is a technology that recognizes and processes objects like human vision, and includes object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / localization, image enhancement, etc.

[0072] FIG. 2a is a generative artificial intelligence (AI) system (200) according to one embodiment. Referring to FIG. 2a, the generative AI system (200) may include a user interface (210), an AI framework (220), a generative AI model (230), a knowledge repository (240), and an application / service module (250). These components may be operated on one or more of an electronic device (101), an external electronic device (102 or 104), or a server (108). For example, the user interface (210) and the AI ​​framework (220) may be operated on the electronic device (101), and the knowledge repository (240) and the generative AI model (230) may be operated on the server (108).

[0073] According to one embodiment, the User Interface (210) may receive user input (e.g., user query). User input may be received in the form of text, images, voice (e.g., natural language), video, menu selection, or a combination thereof. The User Interface (210) may include various context information (e.g., running application or user location) related to the generative artificial intelligence system (200) at the time the user input is received, in addition to or instead of the user input. The User Interface (210) may provide the user input or the context information to the AI ​​Framework (220) and provide the result of processing therefrom to the user, for example, through the AI ​​Framework (220). According to one embodiment, in addition to user input, the electronic device may provide context information obtained using information included on the screen to the AI ​​Framework (220). The result may be provided in the form of text, images, voice, video, an action requested by the user (e.g., launching a specified function or app), or a combination thereof.

[0074] According to one embodiment, the AI ​​Framework (220) can identify (e.g., estimate) a user intent based on at least some user input or context information received from the User Interface (210), control each of the relevant modules (e.g., 221, 223, or 225) to perform a function or action corresponding to the identified user intent, and coordinate cooperation between two or more modules. The AI ​​Framework (220) may include a Prompt Design Module (221), an API / Plug-in Management Module (223), and an Output Modification Module (225) (e.g., an Output Adjustment Module), as illustrated in FIG. 2a.

[0075] According to one embodiment, the Prompt Design Module (221) can generate a prompt to be input to the Generative AI Model (230) based at least partially on user input or context information received from the User Interface (210). For example, the Prompt Design Module (221) can generate a prompt using user preferences, a prompt library, or prompt examples stored in the Knowledge Repository (240) based at least partially on user input or context information.

[0076] According to one embodiment, the API / Plug-in Management Module (223) may communicate, for example, via an API, with various resources (e.g., Knowledge Repository (240)) that provide said additional information when there is a request for said additional information in relation to user input. Additionally or generally, when a specified action (e.g., function, app, or service) is performed in response to said user input, the API / Plug-in Management Module (223) may request the Application / Service Module (250) to perform said specified action via a corresponding API. The API / Plug-in Management Module (223) may provide information obtained from the Knowledge Repository (240), the Application / Service Module 250, or another external resource to the Prompt Design Module (221). That obtained information may be used by the Prompt Design Module (221) to generate a prompt along with the user input, or provided to a generative AI model (230).

[0077] According to one embodiment, the Output Modification Module (225) can fine-tune the results obtained through the Generative AI Model (230) as at least part of the response to user input (e.g., user query). For example, the Output Modification Module (225) can determine whether the content of the response obtained through the Generative AI Model (230) is appropriate as a response to a request made by the user input. For example, the Output Modification Module (225) can determine the degree of relevance, degree of bias (e.g., political or social bias), or degree of harmfulness (e.g., sexual or profanity) of the difference between the response obtained through the Generative AI Model (230) and the user input. Additionally or generally, the Output Modification Module (225) can request that additional AI processing be performed on the obtained response, or provide the user with a hint to avoid unwanted output. For example, additional prompts can be generated through the Prompt Design Module to obtain a response again through the Generative AI Model (230).

[0078] According to one embodiment, the Generative AI Model (230) may form at least part of an artificial intelligence neural network and may include a model that generates images or a model that generates language. The image generation model may include, for example, a generative adversarial network (GAN), a variational autoencoder (VAE), or a Diffusion-based model using a VAE and a Transformer. The language generation model may include, for example, a large language model (LLM), a large multimodal model (LMM), a large vision model (LVM), or a large action model (LAM). The LAM may automatically generate actions for an environment (e.g., a robot, a car, an electronic device (101), or a program (140)). Additionally, for at least some AI models (e.g., LLM), there may be a low-rank adaptation (LoRA) adaptor that is fine-tuned for, for example, a specific task or a specific situation.

[0079] FIG. 2b illustrates an AI Framework (220) having on-device AI processing capabilities according to one embodiment. In this case, the AI ​​Framework (220) may generate and learn a response to the user input using resources within the device, instead of sending the user input received through a User Interface (210) operating on the same device (e.g., electronic device (101)) to a Generative AI Model (230) operating on an external device (e.g., server (108)), or additionally. Referring to FIG. 2b, the AI ​​Framework (220) may include a Cross-Application Action Module (270), a Personal Data Managing Module (280), an On-device AI Model (290), and an Orchestration Module (260).

[0080] According to one embodiment, the Cross-Application Action Module (270) determines one or more additional applications required for the operation of an executed application (e.g., an assistant app) and may link or suggest operations between the app and at least one additional application, or between multiple additional applications. For example, the Cross-Application Actions Module (270) may execute one or more additional applications to be used to respond to a user request through the assistant app sequentially or at least partially, simultaneously. Additionally, the Cross-Application Action Module (270) may communicate with the additional applications so that the result of the execution of one additional application (e.g., content) can be shared with other additional applications.

[0081] According to one embodiment, the Personal Data Managing Module (280) may provide personal information (e.g., schedule, contact, or message information) about a user of the application (e.g., assistant app) or the additional application running on the device (e.g., electronic device 101) or other related individuals (e.g., family or friends) to another module of the AI ​​Framework (220) or a related module (e.g., Generative AI Model 230) running on another device.

[0082] According to one embodiment, the on-device AI model (290) may include at least one model among one or more AI models (e.g., GAN, VAE, LLM, LMM, LVM, or LAM) operated on an external device (e.g., server (108)) or a corresponding lightweight AI model. Additionally, for said model or said lightweight model, there may be, for example, a LoRA adaptor.

[0083] According to one embodiment, the Orchestration Module (260) may select one or more AI models to be used to obtain a response to user input (e.g., user query). For example, the Orchestration Module (260) may select one or more AI models from among an On-Device AI Model (290), an AI model operating on an external device (e.g., server (108)) (e.g., Generative AI Model (230)), or a third AI model (not shown) operating on another external device. When multiple AI models are selected, the Orchestration Module (260) may communicate with the selected models or devices so that the operation between the selected AI models and the processing of the results thereof can be coordinated between the relevant models or devices.

[0084] According to one embodiment, two or more modules of a generative AI system (200) (e.g., Cross-Application Action Module (270) and Orchestration Module (260)) may be implemented as a single module to maintain the same functionality. Various variations are possible.

[0085] FIG. 3 is a flowchart of a method of operation of an electronic device according to one embodiment.

[0086] FIG. 3 can be described based on the embodiments of FIG. 1, FIG. 2a, FIG. 2b, and the embodiments described below.

[0087] At least some of the operations of FIG. 3 may be omitted. The order of the operations of FIG. 3 may be changed. Operations other than the operations of FIG. 3 may be performed before, during, or after the operations of FIG. 3.

[0088] With reference to FIG. 3, the operation of the electronic device (101) can be described. The operations of the electronic device (101) of FIG. 3 can be specifically described based on the operations of the electronic device (101) of FIG. 4a and the operations of the drawings described below.

[0089] Referring to FIG. 3, in operation 301, according to one embodiment, an electronic device (101) may provide first content through a first application. For example, the electronic device (101) may display a first screen of the first application, and the first screen may include first content. The content (e.g., first content or second content described below) may include a video, an image, text, a search result, an icon of an executable file, a file execution screen, a web page, a camera preview screen, an augmented reality screen, or an icon of a folder, and there is no limitation on the type of content. The screen may include one piece of content or multiple pieces of content. An application (e.g., the first application) may be configured to display a screen (e.g., the first screen) containing content (e.g., the first content), and there is no limitation on the type of application. Operation 301 may be described in detail with reference to operation 401 of FIG. 4a as an embodiment.

[0090] In operation 303, according to one embodiment, the electronic device (101) may check for a user request related to the first content. The electronic device (101) may check for a user request related to the first content while providing the first content through the first application. The electronic device (101) may receive the user request and may check for the received user request. For example, the electronic device (101) may obtain the user request through an artificial intelligence (AI) assistant (e.g., an AI agent). For example, while playing a video through a screen, the electronic device (101) may check for a user request as the user queries the AI ​​agent to suggest information about the place displayed on the screen and activities related to that place. There is no limitation on the type of user request. Operation 303 may be described in detail with reference to operation 403 of FIG. 4a as an example.

[0091] In operation 305, according to one embodiment, the electronic device (101) may attempt to obtain a first content element from the first content based on a user request. The first content element of FIG. 3 may include the original reference data of FIG. 4a. For example, the operation of obtaining a first content element from the first content in FIG. 3 may include the operation of obtaining the original reference data related to the first content in FIG. 4a. For example, the electronic device (101) may attempt to obtain the original reference data related to the first content based on a user request. According to one embodiment, the first content element (e.g., original reference data) may be a captured screen. For example, the operation of attempting to obtain a first content element (e.g., original reference data) from the first content may include the operation of attempting a capture operation for a screen containing the first content. For example, the electronic device (101) may attempt a capture operation on a screen containing first content based on a user request.

[0092] In operation 307, according to one embodiment, the electronic device (101) may acquire at least one second content element corresponding to the first content element based at least in part on a determination that the acquisition of the first content element is restricted. According to one embodiment, the electronic device (101) may attempt to acquire the first content element from the first content and determine that the acquisition of the first content element is restricted based on failure to acquire the first content element from the first content. According to one embodiment, the electronic device (101) may determine that the acquisition of the first content element from the first content is restricted based on the first content being restricted access content. According to one embodiment, the electronic device (101) may determine that the acquisition of the first content element from the first content is restricted based on identifying that a capture operation for the screen area where the first content is displayed is prohibited. The second content element of the operation 307 may include substitute data of FIG. 4a. For example, the operation of obtaining the second content element in FIG. 3 may include the operation of obtaining substitute data related to the first content in FIG. 4a. For example, the second content element may include a substitute screen obtained as a search result or reference information related to the first content, and there are no limitations on the second content element. For example, the second content element may include text information, image information, and / or audio information. According to one embodiment, the electronic device (101) may receive data related to the first content (e.g., a second content element) from a content provider of the first content based at least in part on a decision that the acquisition of the first content element is restricted, or acquire additional information (e.g., a second content element) based on input from a user of the electronic device (101), or search for information about the first content (e.g., a second content element).According to one embodiment, the electronic device (101) can obtain at least a portion of a second content element by searching for at least one second content element that can replace the first content element outside the electronic device (101) using information related to the first content accessible within the first application. The electronic device (101) can verify information related to the first content accessible within the first application. The electronic device (101) can search for at least one second content element that can replace the first content element outside the electronic device (101) using information related to the first content. The electronic device (101) can obtain the data searched outside the electronic device (101) as at least a portion of the second content element. According to one embodiment, the electronic device (101) can obtain at least a portion of a second content element by searching for at least one second content element capable of replacing the first content element outside the electronic device (101) using additional information describing the first content element input by a user of the electronic device (101) after a decision that the acquisition of the first content element is restricted. The electronic device (101) can verify additional information describing the first content element input by a user of the electronic device (101) after a decision that the acquisition of the first content element from the first content is restricted. The electronic device (101) can search for at least one second content element capable of replacing the first content element outside the electronic device (101) using additional information. The electronic device (101) can obtain the data searched outside the electronic device (101) as at least a portion of the second content element. According to one embodiment, the electronic device (101) may receive at least a portion of a second content element from a provider of the first content after a determination that the acquisition of a first content element from the first content has been restricted.The electronic device (101) may request at least a portion of a second content element from the provider of the first content after a decision that the acquisition of a first content element from the first content is restricted, and may receive at least a portion of the second content element from the provider of the first content. According to one embodiment, the electronic device (101) may receive at least a portion of the second content element from a wearable electronic device (e.g., the electronic device (102 or 104) of FIG. 1) connected to the electronic device (101) and worn on the face of the user of the electronic device (101). The electronic device (101) may receive at least a portion of the second content element from a wearable electronic device (e.g., the electronic device (102 or 104) of FIG. 1) connected to the electronic device (101) and worn on the face of the user of the electronic device (101) after a decision that the acquisition of a first content element from the first content is restricted. For example, the second content element may include at least a portion of an image captured through a camera of a wearable electronic device (e.g., the electronic device (102 or 104) of FIG. 1). The 307 operation may be described in detail with reference to the 409 operation of FIG. 4a as an example.

[0093] In operation 309, according to one embodiment, the electronic device (101) may cause a response to a user request to be generated using at least one second content element instead of a first content element. The response to the user request may be generated at the electronic device (101) or may be generated externally (e.g., another electronic device (102, 104) or server (108) of FIG. 1) based on the request of the electronic device (101). For example, the electronic device (101) may generate at least a portion of the response to the user request using another electronic device (e.g., another electronic device (102, 104) or server (108) of FIG. 1).

[0094] In operation 311, according to one embodiment, the electronic device (101) may provide a generated response. There are no limitations on the method of providing the response. For example, the electronic device (101) may display a screen containing the generated response. For example, the electronic device (101) may output the generated response as sound. According to one embodiment, the electronic device (101) may perform a user request using at least one second content element (e.g., alternative data of the first content). For example, the electronic device (101) may cause a first result data to be generated based on at least one second content element (e.g., alternative data of the first content) and the user request. Based on the first result data, the electronic device (101) may display a second screen containing second content corresponding to the user request. For example, the electronic device (101) may generate second content through a second application using at least one second content element and provide the second content as at least part of a response. For example, the electronic device (101) may identify a second application corresponding to a user request, generate second content through the second application using at least one second content element, and provide the second content as at least part of a response. For example, the electronic device (101) may generate second content by converting at least one second content element to correspond to the user interface of the second application and display a screen containing the second content.

[0095] Operations 309 and 311 can be described in detail with reference to operation 411 of FIG. 4a as an example.

[0096] According to one embodiment, the electronic device (101) may delete at least a portion of the second content element from the electronic device (101) after providing a response.

[0097] With reference to FIG. 4a and the embodiments of the drawings described below, the embodiment of FIG. 3 will be explained in more detail.

[0098] FIG. 4a is a flowchart of a method of operation of an electronic device according to one embodiment. FIG. 4a may be explained based on the embodiments of FIG. 1 and 2, the embodiments of FIG. 4b, 4c, 4d, FIG. 5, FIG. 6, and 7, and the embodiments described below. FIG. 4b is a diagram illustrating the operation of an electronic device according to one embodiment. FIG. 4c is a diagram illustrating the operation of an electronic device according to one embodiment. FIG. 4d is a diagram illustrating the operation of an electronic device according to one embodiment. FIG. 5 is a diagram illustrating the operation of an electronic device according to one embodiment. FIG. 6 is a diagram illustrating the operation of an electronic device according to one embodiment. FIG. 7 is a diagram illustrating the operation of an electronic device according to one embodiment.

[0099] At least some of the operations of FIG. 4a may be omitted. The order of the operations of FIG. 4a may be changed. Operations other than those of FIG. 4a may be performed before, during, or after the operations of FIG. 4a.

[0100] Referring to FIG. 4a, in operation 401, according to one embodiment, an electronic device (101) may display a first screen of a first application. The first screen may include first content. The content (e.g., first content or second content described below) may include video, image, text, search results, an icon of an executable file, a file execution screen, a web page, a camera preview screen, an augmented reality screen, audio, voice, or an icon of a folder, and there is no limitation on the type of content. The screen may include one piece of content or multiple pieces of content. An application (e.g., the first application) may be configured to display a screen (e.g., the first screen) containing content (e.g., the first content), and there is no limitation on the type of application. For example, the electronic device (101) may display a screen of an application configured to display video (e.g., 410 in FIG. 4b), and the screen may include video. For example, the electronic device (101) may display a screen of an application configured to display an image, and the screen may include an image. For example, the electronic device (101) may display a screen of an application configured to display text, and the screen may include text. For example, the electronic device (101) may display a screen of an application configured to display search results, and the screen may include search results. For example, the electronic device (101) may display a screen of an application configured to display an icon of an executable file or an icon of a folder, and the screen may include an icon of an executable file or an icon of a folder. For example, the electronic device (101) may display a screen of an application configured to display a file execution screen, and the screen may include a file execution screen.According to one embodiment, the content displayed on the screen of the electronic device (101) may be content stored in the memory (130) of the electronic device (101). The content stored in the memory (130) of the electronic device (101) may be content generated by the electronic device (101). The content stored in the memory (130) of the electronic device (101) may be content provided from an external source (e.g., the server (108) of FIG. 1). According to one embodiment, the content displayed on the screen of the electronic device (101) may be content provided by an external content provider (e.g., the server of the content provider (e.g., 108 of FIG. 1)). For example, the external content provider may provide video content, but this is merely an example, and there is no limitation on the type of content provided by the content provider. The content provided by the external content provider may be stored in the electronic device (101). Content provided by an external content provider may be displayed on the screen of the electronic device (101) via streaming. According to one embodiment, the content displayed on the screen of the electronic device (101) may be content provided by an external search server (e.g., search results). However, the types of content and applications are merely examples and are not limited thereto.

[0101] In operation 403, according to one embodiment, the electronic device (101) may acknowledge a user request. The electronic device (101) may receive a user request and acknowledge the received user request. There is no limitation on the manner in which the electronic device (101) receives the user request. For example, the electronic device (101) may acknowledge the user request based on user input including user utterance (e.g., 417 in FIG. 4b), text input (e.g., 418 in FIG. 4b), and / or drawing object input (e.g., 419 in FIG. 4b). For example, the electronic device (101) may acknowledge a drawing object input (e.g., 419 in FIG. 4b) that corresponds to a user command or depicts a result desired by the user, and may acknowledge the user request based on the drawing object input.

[0102] According to one embodiment, the electronic device (101) may obtain a user request through an artificial intelligence (AI) assistant (e.g., an AI agent). The AI ​​assistant (e.g., an AI agent) may be an AI function provided by the electronic device (101). For example, the AI ​​assistant may correspond to an intelligent assistant or a conversational agent. For example, the AI ​​assistant (e.g., an AI agent) may be configured to engage in a conversation with the user and process the user's request. For example, the AI ​​assistant may understand the user's natural language input (e.g., utterance), perform one or more tasks based on the natural language input, and provide the result of the performance using a linguistic response that follows grammatical rules. For example, the electronic device (101) can display the user interface (UI) of the AI ​​assistant (e.g., AI agent) based on a request to call the AI ​​assistant (e.g., AI agent), and can obtain a user request based on user input while the UI of the AI ​​assistant (e.g., AI agent) is displayed.

[0103] According to one embodiment, the electronic device (101) may check for a user request while displaying a first screen of the 401 operation. For example, while displaying the first screen of the 401 operation (e.g., 420 in FIG. 4c), the electronic device (101) may display the UI of the AI ​​assistant (e.g., AI agent) (e.g., 421 in FIG. 4c) based on a request to call the AI ​​assistant (e.g., AI agent), and while the UI of the AI ​​assistant (e.g., AI agent) is displayed, obtain a user request based on user input. There is no limitation on the type of user request. For example, the user request may be related to the performance of a task to obtain data (e.g., still image, video, audio, or text) from at least a part of the content (e.g., first content).

[0104] According to one embodiment, the electronic device (101) may receive a user request to create new content using content (e.g., first content) displayed on a screen (e.g., first screen). For example, in FIG. 4c, the electronic device (101) may display the UI (421) of an AI assistant (e.g., AI agent) based on a request to call an AI assistant (e.g., AI agent) while displaying a screen (420) containing video content, and may check a user request (e.g., "Find this cafe and where the cafe is located, and plan a travel itinerary for that place and save it in a note") based on user input (e.g., utterance, text input, drawing object input). For example, the request "Find this cafe and where the cafe is located" may correspond to a command to search for the name of a specific cafe displayed on the screen and the location where that cafe is located. For example, the request "Plan a travel itinerary for that place and save it to a note" can be interpreted as a command to create a plan for traveling to the area where the cafe is located and save it to a note (or memo) application.

[0105] According to one embodiment, "this cafe" mentioned in the user request of FIG. 4c may be a cafe included in the video content currently included in the screen (420) of the electronic device (101). Furthermore, the cafe may not be limited to the screen at the time the user request is entered, but may also include objects appearing in screens that are adjacent to that time, either before or after. When interpreting the user request, the electronic device (101) may consider screens within a certain range before and after the time the user request is confirmed as reference targets. For example, the electronic device (101) may dynamically determine the range of video referenced based on the length of the user input, the number of frames containing similar scenes, and / or the similarity between scenes. According to one embodiment, the electronic device (101) may determine which part of the screen the content corresponding to the user request is displayed on, and process the user request based on the data of the determined content. According to one embodiment, with reference to FIG. 4d, an electronic device (101) can identify an input (432) from a user (431) specifying a target area (e.g., a circular gesture input received via a touchscreen (e.g., touch and drag input)) while displaying a screen (430) containing video content, identify a part area (433) of the screen (435) as the target area based on the input (432), and obtain a user request related to the target area (e.g., “Find this cafe and the place where the cafe is located, plan a travel itinerary for that place, and save it in a note”) while the UI (434) of an AI assistant (e.g., AI agent) is displayed. For example, in the user request of FIG. 4d, “note” may be a note application installed on the electronic device (101). The electronic device (101) can process the final result to be suitable for the format of a note application in response to a user request to "save it to a note," and this will be described later.The note application is merely an example, and there are no restrictions on the types of applications that can be used to output results based on user requests.

[0106] In operation 405, according to one embodiment, the electronic device (101) can determine that data (e.g., original reference data or alternative data) related to the first content (e.g., the first content displayed on the first screen) is required to perform a user request. Data related to the content (e.g., original reference data or alternative data) will be described later. For example, in FIG. 4d, the electronic device (101) can determine that data regarding the cafe in the video content currently displayed on the screen is required, as information about "this cafe" and "the place where the cafe is located" is required to perform the user request, and that data regarding the cafe in the video content currently displayed on the screen is required. For example, the electronic device (101) can analyze the content displayed on the current screen or the content displayed in the target area of ​​the current screen, and by analyzing the user request, determine that data (e.g., original reference data or alternative data) related to the first content (e.g., the first content displayed on the first screen) is required to perform the user request.

[0107] In operation 407, according to one embodiment, the electronic device (101) can determine whether the first content displayed on the first screen is access-restricted content. For example, the electronic device (101) can determine whether the first content is access-restricted content based on the fact that data related to the first content is required to perform a user request. Access-restricted content may be content for which access to original reference data is restricted. The original reference data may be original data for the content currently displayed on the screen. For example, the original reference data may include data obtained through a capture of the screen corresponding to the content. For example, if the first content is content to which restrictions on usage or access methods are imposed for the protection of rights (e.g., copyright) (e.g., 510 in FIG. 5(a)) or is secure content stored in the electronic device (101) (e.g., 520 or 530 in FIG. 5(a)), the operation of capturing the screen on which the first content is displayed may be restricted while the first content is displayed on the electronic device (101). In such a case, the first content may be referred to as access-restricted content.

[0108] According to one embodiment, restricted access content may include content that is allowed access only to authorized devices, content that is allowed access only to authorized users, content that is allowed to be displayed (e.g., played) only in an authorized manner, or content for which reproduction of at least a portion of the content is prohibited. An example of restricted access content is illustrated in FIG. 5(a). For example, digital right management (DRM) may be a technology or system that imposes specific restrictions on the method of use or access of said content to prevent illegal copying, distribution, or unauthorized use of said digital content, and 510 in FIG. 5(a) may represent content (e.g., video content provided by a content provider) for which restrictions on the method of use or access are imposed for copyright protection. For example, an electronic device (101) may confirm that the first content is restricted access content based on confirming that the first content is content for which restrictions on the method of use or access are imposed for copyright protection. For example, the electronic device (101) can identify what constraints are applied to the first content. The number or type of constraints that can be applied to the content may be one or multiple. For example, 510 in FIG. 5 (a) may include the image content of FIG. 4b, but there are no limitations.

[0109] For example, 520 in FIG. 5(a) may represent an icon of a file that requires authentication to access data (e.g., a secure file stored in the electronic device (101)). For example, 530 in FIG. 5(a) may represent files of secure applications among the applications stored in the electronic device (101) that are restricted from access by security settings. Different types of access restrictions may be applied to the files in correspondence with the type of secure application used to execute each file. For example, 520 and 530 in FIG. 5(a) may correspond to secure content (e.g., the secure content (1610) of FIG. 16). For example, to access the original reference data of the secure content (e.g., restricted access content), authentication may be required, access rights to the secure area may be required, login may be required, or rights granted by the content provider may be required. According to one embodiment, the electronic device (101) can identify the first content as secure content (e.g., restricted access content) based on the fact that the first content is content stored in a secure area within the memory (130). According to one embodiment, the electronic device (101) can identify the first content as secure content (e.g., restricted access content) based on the fact that a first application corresponding to a first screen containing the first content is classified as a secure application. According to one embodiment, the electronic device (101) can identify the first content as secure content (e.g., restricted access content) based on the fact that a password is set on the first content. According to one embodiment, the electronic device (101) can identify the first content as secure content (e.g., restricted access content) based on the fact that authentication is required to verify the entire content of the first content. According to one embodiment, the electronic device (101) can determine that the first content is access restricted content based on the fact that the first content is security content.

[0110] For example, in FIG. 5 (b) and (c), the electronic device (101) may display a screen (540) in which video content (541), which is access restricted content, is being played. At this time, when a capture operation is performed on the screen of the electronic device (101), the area (551) corresponding to the video content in the captured screen (550) may be blanked out based on the fact that the video content is access restricted content. For example, the electronic device (101) may obtain a screenshot in which the part corresponding to the video content (541) among the screen elements included in the screen (540) is excluded (e.g., blanked out) based on the fact that the video content (541) is access restricted content. In this case, the electronic device (101) may not be able to obtain the original reference data of the video content through the operation of capturing the screen.

[0111] According to one embodiment, the electronic device (101) may inform the user that the first content is restricted access content based on confirming that the first content is restricted access content. For example, while displaying a screen (610) containing the first content (e.g., video content) in FIG. 6 (a), the electronic device (101) may check a user request (e.g., 621 in FIG. 6 (b-1) or 622 in FIG. 6 (b-2)) and inform the user that the first content is restricted access content (630 in FIG. 6 (c)) based on confirming that the first content corresponding to the user request is restricted access content. A method of informing a user that the first content is restricted access content (630 of (c) in FIG. 6) may include a method of outputting a screen to a display (e.g., display module (160) of FIG. 1), a method of outputting voice to a speaker (e.g., sound output module (155) of FIG. 1), and / or a method of outputting vibration through a haptic module (e.g., haptic module (179) of FIG. 1), and there are no limitations on the method of informing a user that the first content is restricted access content.

[0112] In FIG. 6, according to one embodiment, an electronic device (101) can identify a user request (e.g., 621 in (b-1) of FIG. 6 or 622 in (b-2) of FIG. 6). In FIG. 6 (b-1), the electronic device (101) can identify a user request (621) that says, “Find this cafe and the place where the cafe is located, plan a travel itinerary for that place, and save it in a note.” In this case, the electronic device (101) can identify that the user request (621) is a request for the creation of content (e.g., note content) corresponding to a note application by analyzing the part that says “save it in a note.” Accordingly, the electronic device (101) can determine the application corresponding to the user request (621) as a note application and obtain content (e.g., note content) corresponding to the user request by processing the result data according to the user request into a format corresponding to the note application. For example, the electronic device (101) can determine the application corresponding to the user request (621) as a note application, identify the input fields of the note application necessary to generate content (e.g., note content) corresponding to the note application, classify and rewrite the result data according to the user request into information for inputting into the input fields of the note application, and input the information into the note application to generate note content. In (b-2) of FIG. 6, the electronic device (101) can confirm a user request (622) that says, “Find this cafe and where the cafe is located, and make a travel itinerary for that place.” In this case, the electronic device (101) can determine to output a response corresponding to the user request (621) through the UI of an AI assistant (e.g., AI agent) by analyzing the part that says “make a travel itinerary.”Accordingly, the electronic device (101) can obtain content corresponding to a user request by processing result data corresponding to a user request into a format corresponding to the UI of an AI assistant (e.g., AI agent). For example, the electronic device (101) can determine an application corresponding to a user request (622) as an AI assistant (e.g., AI agent), identify input fields necessary to generate content corresponding to the AI ​​assistant (e.g., AI agent), classify and rewrite result data corresponding to the user request into information to be entered into the input fields of the AI ​​assistant (e.g., AI agent), and input the information into the AI ​​assistant (e.g., AI agent) to display the content.

[0113] In operation 409, according to one embodiment, the electronic device (101) may obtain substitute data related to the first content. The electronic device (101) may obtain substitute data to replace the original reference data based on the fact that the first content is restricted access content. The substitute data may be data that replaces the original reference data of the first content in order to process a user request for the first content. For example, when a captured screen of video content (e.g., original reference data) cannot be obtained, the electronic device (101) may receive data related to the first content (e.g., substitute data) from the content provider to replace the original reference data, obtain additional information (e.g., substitute data) based on user input of the electronic device (101), or search for information about the first content (e.g., substitute data). There are no limitations on the type of substitute data, and the types of substitute data will be described later. The operation of determining the acquisition method for the electronic device (101) to acquire alternative data will be described later.

[0114] In operation 411, according to one embodiment, the electronic device (101) may perform a user request using alternative data. For example, the electronic device (101) may cause first result data to be generated based on the alternative data of the first content and the user request. The electronic device (101) may display a second screen containing second content corresponding to the user request based on the first result data. According to one embodiment, the second screen containing the second content may be a screen of the UI of an AI assistant (e.g., an AI agent). For example, the electronic device (101) may obtain the second content and display a second screen containing the second content by processing the first result data generated based on the alternative data of the first content and the user request into a format corresponding to the UI of the AI ​​assistant (e.g., an AI agent). According to one embodiment, the second screen containing the second content may be an execution screen of an application corresponding to the user request. For example, the electronic device (101) can identify a second application corresponding to a user request, and by processing replacement data of the first content and first result data generated based on the user request into a format corresponding to the second application, acquire the second content and display a second screen containing the second content.

[0115] For example, with reference to FIG. 7 regarding operations 409 and 411, the electronic device (101) can check a user request related to the first content (e.g., "Find this cafe and where the cafe is located, and plan a travel itinerary for that place") while displaying a screen containing the first content (e.g., video content corresponding to an "AA movie"), and can confirm that access to the data of the first content is required to process the user request, but that the first content is restricted access content. In FIG. 7 (a), the electronic device (101) can select a search method through a search application (e.g., a browser) as a method for obtaining alternative data of the first content and provide a guidance message (710) to the user. According to one embodiment, the electronic device (101) can check the public information of the first content (e.g., video content corresponding to an "AA movie") in order to obtain alternative data. The public information may be information accessible through the electronic device (101) in relation to the first content. For example, while a screen containing the first content is displayed, the electronic device (101) may obtain information about the first content provided by the content provider of the first content (e.g., title of the video, time of playback of the video, description of the scene displayed on the screen, description of the place or object displayed in the scene) as public information. There is no limitation on the type of public information. For example, the public information may be data that the electronic device (101) can access among the metadata of the content (e.g., the first content). For example, the public information may be data that can be obtained from a screen area other than the area where access is restricted (e.g., the area corresponding to the video content (541)) among the screen (540) containing the video content (541) which is restricted access content.For example, the public information may include at least one of audio information (e.g., sound output from the electronic device (101) immediately before and after the time when the user request is received) for video content (541) which is restricted access content, subtitle information, comment information, address information on the web corresponding to the content (e.g., uniform resource locator), location information of a pointer on a playback progress bar, or playback time information related to the user request. The electronic device (101) can obtain the searched data as replacement data for the first content by performing a search using at least one of the public information and the user request.

[0116] According to one embodiment, the electronic device (101) may obtain additional information from a user to obtain alternative data. The additional information may be information obtained based on user input (e.g., speech input, text input, and / or drawing input) through a UI for obtaining additional information. The acquisition of additional information will be described later. The electronic device (101) may obtain the retrieved data as alternative data for the first content by performing a search using the additional information (or data generated using the additional information). According to one embodiment, the electronic device (101) may obtain the retrieved data as alternative data for the first content by performing a search using a user request, public information, and / or additional information.

[0117] According to one embodiment, the electronic device (101) may acquire one alternative data or acquire a plurality of alternative data. In FIG. 7(b), the electronic device (101) acquires a plurality of alternative data related to the first content, displays a screen (720) for selecting one of the plurality of alternative data, and can perform a user request using the alternative data selected based on user input. In FIG. 7(c), the electronic device (101) can display a screen (730) (e.g., a second screen) containing content (e.g., a second content) corresponding to the user request by performing the user request using the alternative data. The screen (730) of FIG. 7(c) may be a screen of the UI of an AI assistant (e.g., an AI agent). As the "Save to Note App" button is selected on the screen (730) of FIG. 7 (c), the electronic device (101) in FIG. 7 (d) can save the content generated by the user request (e.g., second content) to the note application (e.g., second application) (e.g., 741, 742 of FIG. 7 (d). The display of the screen (730) of FIG. 7 (c) can be omitted, and the screen of FIG. 7 (d) (e.g., second screen) can be displayed directly. For example, the electronic device (101) can determine an application corresponding to a user request as a note application (e.g., a second application), and by processing the replacement data of the first content and the result data generated based on the user request into the format of the note application (e.g., a second application), obtain content corresponding to the user request (e.g., a second content), and display a screen (742) containing the obtained content (e.g., a second content).

[0118] 413 In operation, according to one embodiment, the electronic device (101) may obtain original reference data of the first content. The electronic device (101) may obtain original reference data based on the fact that the first content is not restricted access content. According to one embodiment, based on obtaining original reference data of the first content, an anonymization operation may be performed on the original reference data so that the original reference data is not used as training data for an artificial intelligence model.

[0119] According to one embodiment, the original reference data (or alternative data) obtained to perform a user request may be data configured to be available under specific conditions. The specified conditions (e.g., specific conditions) may be set by the provider of the content (e.g., the first content). For example, the original reference data (or alternative data) may be data configured to be available only within a specific period. For example, the original reference data (or alternative data) may be data configured to be available only for a specific purpose (e.g., performing a specified user request). For example, the original reference data (or alternative data) may be data configured so that it is not allowed to be copied or transmitted externally after being stored in the electronic device (101). For example, the original reference data (or alternative data) may be data configured to be deleted after being used to perform a user request in the background, and not to be provided to the user or stored in the electronic device (101). According to one embodiment, if a function to be performed using said content is required while providing content to which digital right management (DRM) related settings or security related settings are assigned, data that can be used only for the performance of a specific function related to said content is acquired, and when the performance of said function is completed, the acquired data may be deleted.

[0120] In operation 415, according to one embodiment, the electronic device (101) may perform a user request using original reference data. For example, the electronic device (101) may cause second result data to be generated based on the original reference data of the first content and the user request. For example, the electronic device (101) may perform a search based on the original reference data of the first content and the user request, and cause second result data to be generated based on the searched data. The electronic device (101) may display a third screen containing third content corresponding to the user request based on the second result data. According to one embodiment, the third screen containing the third content may be a screen of the UI of an AI assistant (e.g., an AI agent). For example, the electronic device (101) may obtain the third content by processing the second result data into a format corresponding to the UI of the AI ​​assistant (e.g., an AI agent) and display a third screen containing the third content. The operation of processing data into a format corresponding to the UI of an AI assistant (e.g., an AI agent) will be described later. According to one embodiment, a third screen containing third content may be an execution screen of an application corresponding to a user request. For example, the electronic device (101) may identify a second application corresponding to a user request, obtain third content by processing second result data into a format corresponding to the second application, and display a third screen containing third content. The operation of processing data into a format corresponding to the second application will be described later.

[0121] According to one embodiment, an electronic device (101) that performs the operation of FIG. 4a may be described. The electronic device (101) may display a first screen of a first application containing first content. The electronic device (101) may receive user input. The electronic device (101) may analyze user input (e.g., voice, text, or drawing object based on speech). The electronic device (101) may verify a user request based on user input. The electronic device (101) may verify an application to be analyzed (e.g., the first application) and content displayed on the screen for processing the user request. The electronic device (101) may verify that data of the content displayed on the screen is required to process the user request. The electronic device (101) may verify whether the content displayed on the screen is restricted access content. The electronic device (101) may determine a method of obtaining data of the content displayed on the screen (e.g., original reference data or alternative data). The electronic device (101) can obtain data of content displayed on a screen (e.g., original reference data or alternative data) through communication with a content provider. The electronic device (101) can obtain alternative data of the content through a search for the content. The electronic device (101) can determine an application for searching for alternative data. The electronic device (101) can obtain alternative data by performing a search for the content using the determined application. The electronic device (101) can determine whether to perform a search using public information of the content. The electronic device (101) can obtain alternative data of the content by searching for the content using public information of the content. The electronic device (101) can determine whether to perform a search using additional information obtained from a user. The electronic device (101) can obtain alternative data of the content by searching for the content using additional information obtained from a user.The electronic device (101) can determine and manage acquired alternative data. The electronic device (101) can manage actions to perform user requests using the acquired alternative data. The electronic device (101) can plan a process to perform user requests using the acquired alternative data and process user requests according to the planned process. The electronic device (101) can perform user authentication to extract original reference data of content displayed on the screen. The electronic device (101) can apply security to result content containing original reference data. In addition, the electronic device (101) performing the operation of FIG. 4a can perform the embodiments described below.

[0122] FIG. 8 is a flowchart of a method of operation of an electronic device according to one embodiment. FIG. 8 can be explained based on the embodiments described above, the embodiments of FIG. 9, and the embodiments described below. FIG. 9 is a diagram illustrating the operation of an electronic device according to one embodiment.

[0123] With reference to the operations of FIGS. 8 and 9, operations 409 and 411 of FIG. 4a can be explained. The operations of FIGS. 8 and 9 can be performed after operation 407 of FIG. 4a has been performed. With reference to FIGS. 8 and 9, operations of obtaining additional information describing content (e.g., text, drawing object) from a user and searching for alternative data using the additional information can be explained.

[0124] At least some of the operations of FIG. 8 may be omitted. The order of the operations of FIG. 8 may be changed. Operations other than those of FIG. 8 may be performed before, during, or after the operations of FIG. 8.

[0125] Referring to FIG. 8, in operation 801, according to one embodiment, the electronic device (101) may display a UI for obtaining additional information from a user based on the fact that the first content displayed on the screen (e.g., a scene from a movie) is restricted access content. For example, in FIG. 9, the electronic device (101) may analyze a user request (e.g., “Find this cafe and where the cafe is located, and plan a travel itinerary for that place”), determine a phrase for obtaining additional information related to the user request (e.g., “Please describe the appearance of the cafe in detail”), and display a UI (e.g., 911, 912, 913) containing the determined phrase.

[0126] In operation 803, according to one embodiment, the electronic device (101) may obtain additional information related to the first content based on user input through a UI for obtaining additional information. For example, in (a-1) of FIG. 9, the electronic device (101) may obtain additional information describing the content displayed on the screen based on text input (or voice input). For example, the electronic device (101) may request the user to provide a more detailed description, and the user may input a detailed description of the image shown in the access-restricted content intended by the user, such as "green plants are growing on the brick building, and there are many tables with pink parasols on the terrace," and the electronic device (101) may utilize the additional information to search for alternative data. For example, in (a-2) of FIG. 9, the electronic device (101) may obtain additional information describing the content displayed on the screen based on drawing input. The electronic device (101) may request the user to draw a more detailed view and provide a UI that can receive sketch drawing input from the user. The user may input a sketch drawing containing strokes and colors that represent the features of the original content using a sketch drawing tool via a finger or a stylus pen (900). The sketch may include not only simple shape drawings but also handwritten text inputs or drawings representing control actions. The electronic device (101) may utilize sketch image information based on the sketch drawing input for searching alternative data (e.g., images) through a search application (e.g., a browser). For example, in (a-3) of FIG. 9, the electronic device (101) may obtain additional information describing the content displayed on the screen based on text input (or voice input) and drawing input.When user speech sounds are input during the sketch drawing process, the electronic device (101) can acquire time stamp information for the corresponding point in time when the sketch drawing and sound information are input. The electronic device (101) can reflect sketch information and sound input information and / or text input information together, and can acquire alternative data by using additional information input by the user (e.g., text, speech sound, sketch image information) in combination. For example, the electronic device (101) can utilize additional information input by the user (e.g., text, speech sound, and / or sketch image information) in combination for searching alternative data (e.g., images) through a search application (e.g., browser). According to one embodiment, the electronic device (101) can generate data to be used for searching by using additional information (e.g., text, speech sound, or sketch image information at least one). For example, the electronic device (101) can generate an image using descriptive information about content entered by a user by transmitting additional information (e.g., at least one of text, speech sound, or sketch image information) as input to a generative AI model. For example, in FIG. 9 (b), the electronic device (101) can generate multiple data to be used for searching using additional information (e.g., text, speech sound, or at least one of sketch image information) and display a screen (920) for selecting one of the multiple data. For example, the electronic device (101) can generate images to be used for searching using additional information (e.g., text, speech sound, or at least one of sketch image information) and determine the image to be used for searching among the generated images according to the user's selection.The electronic device (101) may generate data to be used for a search using additional information (e.g., text, speech sound, or at least one of sketch image information) and may perform the search described below with the generated data without a selection procedure for the data. The electronic device (101) may perform a search using the data generated based on the additional information (e.g., text, speech sound, or at least one of sketch image information), or may use the data generated based on the additional information (e.g., text, speech sound, or at least one of sketch image information) itself as replacement data.

[0127] In operation 805, according to one embodiment, the electronic device (101) may perform a search through an external search server (e.g., server (108) of FIG. 1) using additional information. The electronic device (101) may perform a search using additional information of operation 803, or may perform a search using data generated using additional information. Referring to 920 of FIG. 9 (c), according to one embodiment, the electronic device (101) may determine an application to perform the search and perform a search using additional information (or data generated using additional information) through the determined application. According to one embodiment, the electronic device (101) may perform a search by generating a query based on additional information and / or public information regarding the first content. According to one embodiment, the electronic device (101) may perform a search by generating a query based on data generated in the electronic device (101) using additional information and / or public information regarding the first content. The electronic device (101) may perform a retrieval-augmented generation (RAG) method during the retrieval of alternative data.

[0128] In operation 807, according to one embodiment, the electronic device (101) may obtain search data provided by a search server as replacement data for the first content. Subsequently, the electronic device (101) may perform a user request using the replacement data.

[0129] According to one embodiment, the electronic device (101) can perform a search (e.g., the search of operation 805 of FIG. 8) using an AI agent dedicated to a search application. For example, the AI ​​agent dedicated to a search application can plan a workflow through a planning process using an AI model including a Large Language Model (LM) based on functions that can be performed within the search application based on a received prompt, and generate a workflow list to proceed with and manage the execution of operations necessary to process user requests (e.g., search and filtering, creation of final search result content). The AI ​​agent dedicated to a search application performs the planned operations using an AI model including a Large Action Model (LAM), and a process to supplement the executed workflow can be performed through a process including reflection on the results of the operation execution. Regarding the results of the operation of an artificial intelligence agent dedicated to a search application, for example, the final data obtained by performing filtering on multiple candidate data obtained after the search is completed can be generated and provided as response content for a reply based on the context of the user request. According to one embodiment, the electronic device (101) can check additional information based on the user request and user input, and / or public information regarding the content, and determine whether a search for alternative data is necessary.

[0130] According to one embodiment, the electronic device (101) may determine that it can perform a user request using only the user request. In this case, the electronic device (101) may generate a prompt for searching for data necessary to create content corresponding to the user request using the user request, and transmit the generated prompt to an artificial intelligence agent dedicated to the search application. Subsequently, the electronic device (101) may create content corresponding to the user request using the searched data by utilizing the artificial intelligence agent dedicated to the search application.

[0131] According to one embodiment, the electronic device (101) may determine that it can perform a user request using public information regarding the user request and content. In this case, the electronic device (101) may generate a prompt for searching for data necessary to create content corresponding to the user request using the user request and public information, and transmit the generated prompt to an artificial intelligence agent dedicated to the search application. Subsequently, the electronic device (101) may create content corresponding to the user request using the searched data by utilizing the artificial intelligence agent dedicated to the search application.

[0132] According to one embodiment, the electronic device (101) may determine that it can perform a user request using additional information based on a user request and user input. In this case, the electronic device (101) may generate a prompt for searching for data necessary to create content corresponding to the user request using the user request and additional information, and transmit the generated prompt to an artificial intelligence agent dedicated to a search application. Subsequently, the electronic device (101) may create content corresponding to the user request using the searched data by utilizing the artificial intelligence agent dedicated to the search application.

[0133] According to one embodiment, the electronic device (101) may determine that it can perform a user request by utilizing a user request, public information regarding content, and additional information based on user input. In this case, the electronic device (101) may generate a prompt for searching for data necessary to create content corresponding to the user request using the user request, public information, and additional information, and transmit the generated prompt to an artificial intelligence agent dedicated to the search application. Subsequently, the electronic device (101) may create content corresponding to the user request by utilizing the searched data through the artificial intelligence agent dedicated to the search application.

[0134] According to one embodiment, the electronic device (101) may determine that it can perform a user request by using additional search data through a search application, in addition to user requests, public information regarding content, or additional information based on user input. In this case, the electronic device (101) may generate a prompt for searching for data to replace the original reference data of the content by using the user request, public information regarding content, or additional information based on user input, and transmit the generated prompt to an AI agent dedicated to the search application. Subsequently, the electronic device (101) may acquire the searched data as replacement data by utilizing the AI ​​agent dedicated to the search application. At this time, based on the fact that multiple replacement data are acquired, the electronic device (101) may perform an operation to select one replacement data according to the user's selection. The electronic device (101) may generate a prompt for searching for data necessary to create content corresponding to the user request by using the acquired (or selected) replacement data, and transmit the generated prompt to an AI agent dedicated to the search application. Afterward, the electronic device (101) can generate content that responds to user requests by utilizing the searched data through an artificial intelligence agent dedicated to the search application.

[0135] FIG. 10 is a drawing illustrating the operation of an electronic device according to one embodiment.

[0136] According to one embodiment, a UI for obtaining additional information from a user can be described with reference to FIG. 10. In FIG. 10 (a), an electronic device (101) may display a screen (1010) containing a first content (e.g., video content). While displaying the screen (1010) containing the first content (e.g., video content), the electronic device (101) may receive a user request for the first content (e.g., video content). The electronic device (101) may determine that additional information from the user is required to process the user request. For example, the electronic device (101) may determine that additional information from the user is required to process the user request based on the fact that the first content (e.g., video content) is access-restricted content. In FIG. 10(b), the electronic device (101) may display a UI (1021) for obtaining additional information on a screen (1020, 1030) containing first content (e.g., video content). The UI (1021, 1031) for obtaining additional information may be provided in a transparent or translucent overlay form. The electronic device (101) may adjust the transparency of the UI (1021) for obtaining additional information. Depending on the transparency of the UI (1021) for obtaining additional information, a portion of the screen (1020) displayed below the UI (1021) for obtaining additional information may be visible. The user may input sketch additional information that describes the original by sketching along the shape of an object in a desired area within the screen. The electronic device (101) may utilize the obtained sketch additional information for searching alternative data (e.g., images) through a search application (e.g., a browser). The electronic device (101) can obtain the retrieved data as replacement data by utilizing additional sketch information.The electronic device (101) can perform a user request using the acquired alternative data (e.g., causing result data to be generated using the alternative data and displaying a screen containing second content based on the result data).

[0137] FIG. 11 is a flowchart of a method of operation of an electronic device according to one embodiment. FIG. 11 can be explained based on the embodiments described above, the embodiments of FIG. 12, and the embodiments described below. FIG. 12 is a diagram illustrating the operation of an electronic device according to one embodiment.

[0138] Operations 409 and 411 of FIG. 4a can be explained with reference to the operations of FIG. 11 and FIG. 12. The operations of FIG. 11 and FIG. 12 can be performed after operation 407 of FIG. 4a has been performed.

[0139] At least some of the operations of FIG. 11 may be omitted. The order of the operations of FIG. 11 may be changed. Operations other than those of FIG. 11 may be performed before, during, or after the operations of FIG. 11.

[0140] Referring to FIG. 11, in operation 1101, according to one embodiment, the electronic device (101) can determine whether there is an artificial intelligence agent provided by a content provider (e.g., a content provider of the first application) based on the fact that the first content is content provided by an external content provider. The content provider may provide its own artificial intelligence agent, and in this case, the electronic device (101) may decide to perform a user request using the artificial intelligence agent of the content provider.

[0141] In operation 1103, according to one embodiment, the electronic device (101) may transmit a first request to the content provider (e.g., to the content provider's server (e.g., the server (108) of FIG. 1)) to request data related to the first content necessary to perform a user request, based on the existence of the content provider's artificial intelligence agent. For example, in the case of video content, the electronic device (101) may transmit the user request and accessible playback information of the current content to the content provider's artificial intelligence agent (e.g., 1200 of FIG. 12 (a)).

[0142] In operation 1105, according to one embodiment, the electronic device (101) may perform a user request based on data provided by a content provider based on a first request. For example, the data provided by the content provider based on the first request may include original reference data corresponding to a screen containing the first content (e.g., the original image of (b-1) in FIG. 12), replacement data replacing the original reference data (e.g., replacement images (1221, 1222) of (b-2) in FIG. 12), reference information related to the first content provided by the content provider (e.g., reference information of (b-3) in FIG. 12), result data generated by the content provider's content server (e.g., the server (108) in FIG. 1) based on the user request, or content provided by the content provider's content server (e.g., the server (108) in FIG. 1) in response to the user request (e.g., content about a travel itinerary of (b-4) in FIG. 12). For example, the electronic device (101) may receive an original image corresponding to the content from a content provider based on a first request and provide guidance (1210) to the user. For example, the electronic device (101) may receive an alternative image corresponding to the content from a content provider based on a first request and provide guidance (1220) to the user. For example, the electronic device (101) may receive reference information related to the first content (e.g., the name of the cafe and information about the region where the cafe is located) from a content provider based on a first request and provide guidance (1230) to the user. For example, the electronic device (101) may receive result data generated from the content provider's content server (e.g., server (108) of FIG. 1) based on a user request, or content (e.g., travel itinerary content) provided from the content provider's content server (e.g., server (108) of FIG. 1) in response to a user request, and provide guidance (1240) to the user.According to one embodiment, the electronic device (101) determines the characteristics of data provided by a content provider based on a first request, defines a task for performing a user request based on the characteristics of the data, and can perform the user request based on the received data.

[0143] FIG. 13 is a flowchart of a method of operation of an electronic device according to one embodiment. FIG. 13 can be explained based on the embodiments described above and the embodiments described below.

[0144] With reference to the operations of FIG. 13, operations 409 and 411 of FIG. 4a can be explained. The operations of FIG. 13 can be performed after operation 407 of FIG. 4a has been performed.

[0145] At least some of the operations of FIG. 13 may be omitted. The order of the operations of FIG. 13 may be changed. Operations other than those of FIG. 13 may be performed before, during, or after the operations of FIG. 13.

[0146] Referring to FIG. 13, in operation 1301, according to one embodiment, an electronic device (101) may acquire a plurality of alternative data related to a first content. While displaying a first screen of a first application containing the first content, the electronic device (101) may acquire a user request related to the first content and acquire alternative data in place of the original reference data of the first content based on the fact that the first content is restricted access content. At this time, there may be a plurality of acquired alternative data.

[0147] 1303 In operation, according to one embodiment, the electronic device (101) may display a screen for selecting one of a plurality of alternative data. For example, in FIG. 7(b), the electronic device (101) may display a screen (720) for selecting one of a plurality of alternative data. The electronic device (101) may obtain a plurality of searched data as alternative data for the first content by performing a search using a user request, public information, and / or additional information, and may display a screen for selecting one of the plurality of alternative data.

[0148] In operation 1305, according to one embodiment, the electronic device (101) may perform a user request using alternative data selected based on user input. The performance of the user request can be understood by referring to operation 411 of FIG. 4a. For example, the electronic device (101) may cause result data corresponding to the user request to be generated using alternative data selected based on user input, and may display a screen including a second content based on the result data.

[0149] FIG. 14a is a drawing illustrating the operation of an electronic device according to one embodiment.

[0150] FIG. 14a(a) shows a state (1410) in which a user (1412) is wearing an electronic device (1413). In FIG. 14a(a), the electronic device (101) may be provided in a form (1413) that the user (1412) can wear (e.g., a VST (video see-through) device). The user can view a screen (1411) in three-dimensional space provided by the electronic device (1413). FIG. 14a(b) illustrates an electronic device (101) provided in the form of a smartphone or tablet. The user (1421) can view a screen (1420) provided by the electronic device (101). It is obvious that even when the electronic device (101) is provided in a form (1413) that the user (1412) can wear, operations of the electronic device (101) can be performed. Figure 14b can be explained with reference to Figure 14a.

[0151] FIG. 14b is a flowchart of a method of operation of an electronic device according to one embodiment. FIG. 14b can be described based on the embodiments described above, the embodiments of FIG. 14a, and the embodiments described below.

[0152] At least some of the operations of FIG. 14b may be omitted. The order of the operations of FIG. 14b may be changed. Operations other than those of FIG. 14b may be performed before, during, or after the operations of FIG. 14b.

[0153] Referring to FIG. 14b, in operation 1431, according to one embodiment, the electronic device (101, 1413) of FIG. 14a (e.g., the electronic device (101) of FIG. 1) can play a first content (e.g., video content). The operation of playing the first content of operation 1431 may include an operation of providing the first content through the first application of FIG. 3 (e.g., an operation of displaying a first screen of the first application containing the first content of FIG. 4a).

[0154] In operation 1433, according to one embodiment, the electronic device (101, 1413) may detect a user request (1414, 1422) for a first content being played on the electronic device (101, 1413). The user request of operation 1431 may include a user request related to the first content of FIG. 3 (e.g., a user request to create new content using the first content of FIG. 4a).

[0155] In operation 1435, according to one embodiment, the electronic device (101, 1413) may extract accessible first information of the first content based on a user request. For example, the accessible first information of the first content may include public information including an identified content ID, content name, episode number, content playback time information, and / or performer information.

[0156] In operation 1437, according to one embodiment, the electronic device (101, 1413) may determine second information that is the subject of analysis of the first content based on a user request. For example, the electronic device (101) may define second information of the first content that is necessary for generating a result based on a user request. According to operation 1439 described below, the second information of the first content that is necessary for generating a result based on a user request may be obtained from the electronic device (101, 1413) or provided by the content provider of the first content.

[0157] In operation 1439, according to one embodiment, the electronic device (101, 1413) can determine whether the first content is restricted access content. For example, the electronic device (101, 1413) can determine that the first content is restricted access content based on confirming that the first content is content subject to restrictions on usage or access methods for protection protected by a right (e.g., copyright) or is secure content stored in the electronic device (101). For example, the electronic device (101, 1413) can determine that the first content is not restricted access content based on confirming that the first content is not content subject to restrictions on usage or access methods for protection protected by a right (e.g., copyright) and is not secure content stored in the electronic device (101).

[0158] In operation 1441, according to one embodiment, the electronic device (101, 1413) may obtain second information of the first content necessary for generating a result in response to a user request in the electronic device (101), based on the fact that the first content is not restricted access content. For example, the second information may include original reference data obtained in the electronic device (101) (e.g., the first content element of FIG. 3 (e.g., the original reference data of FIG. 4a)).

[0159] In operation 1443, according to one embodiment, the electronic device (101, 1413) may search for data corresponding to a user request outside of the electronic device (101, 1413) through a search application (e.g., a browser). For example, when performing operation 1443 after operation 1441, the electronic device (101, 1413) may search for data corresponding to the user request based on the user request and second information obtained from the electronic device (101).

[0160] In operation 1445, according to one embodiment, the electronic device (101, 1413) may generate a prompt for generating result data corresponding to the user request based on search data retrieved based on the user request and second information obtained from the electronic device (101). In operation 1445, according to one embodiment, the electronic device (101, 1413) may omit operation 1443 and generate a prompt for generating result data corresponding to the user request based on the user request and second information obtained from the electronic device (101).

[0161] In operation 1447, according to one embodiment, the electronic device (101, 1413) may cause result data corresponding to a user request to be generated based on the prompt of operation 1445. As previously mentioned, the result data may be generated in the electronic device (101, 1413) or may be generated in an external server.

[0162] In operation 1449, according to one embodiment, the electronic device (101, 1413) may provide second content based on result data. For example, the electronic device (101, 1413) may generate second content corresponding to a user request by processing the result data into a format corresponding to the UI of an AI assistant (e.g., an AI agent), or may generate second content by processing the result data into a format corresponding to a second application according to a user request. The electronic device (101, 1413) may provide a screen including the generated second content.

[0163] The operations 1443, 1445, 1447, and 1449 following the operation 1459 will be described later.

[0164] Based on the confirmation in the aforementioned operation 1439 that the first content is not restricted access content, operation 1451 may be performed.

[0165] In operation 1451, according to one embodiment, the electronic device (101, 1413) may identify an AI agent provided by the content provider of the first content based on the fact that the first content is not restricted access content. As previously described, the content provider may provide its own AI agent, and the electronic device (101, 1413) may perform operation 1453 as the content provider's AI agent is identified.

[0166] In operation 1453, according to one embodiment, an electronic device (101, 1413) may request a content provider to provide information related to a user request and / or second information based on a user account. For example, the electronic device (101, 1413) may request the provision of information related to a user request and / or second information while providing the following information(s). For example, the electronic device (101, 1413) may provide the content provider with second information regarding a first content for performing a user request and / or the user request. For example, the electronic device (101, 1413) may provide the content provider with information regarding a user account. For example, the electronic device (101, 1413) may provide the content provider with information regarding the characteristics of the first content, which is access restricted content.

[0167] In operation 1455, according to one embodiment, a content provider (e.g., a server of a content provider (e.g., 108 of FIG. 1)) may determine information to be transmitted to an electronic device (101, 1413) based on information provided from an electronic device (101, 1413) (e.g., a user request, second information about a first content for performing a user request, information about a user account, and / or information about characteristics of the first content), and may generate or extract the determined information. For example, a content provider (e.g., a server of the content provider (e.g., 108 of FIG. 1)) may determine as information to be transmitted to the electronic device (101, 1413) information regarding the right to temporarily access the first content within the electronic device (101, 1413), information regarding the original reference data of the second information of the first content (e.g., a capture screen of the scene corresponding to the user request), information regarding the replacement data of the second information of the first content (e.g., data in which specific information within the video content (e.g., person information unrelated to the user request) is deleted and descriptive information regarding the scene corresponding to the user request is included, or data similar to the scene corresponding to the user request), or information regarding the content corresponding to the user request and the second information (e.g., content that the content provider has already obtained from another business operator or content generated using the content provider's AI model). Even if the content requiring analysis for performing a user request is the same content (e.g., the first content), since the user may generate unexpected requests with various characteristics that suit their situation each time, the content provider (e.g., the content provider's AI agent) may grasp the context of the request made by the user each time, extract and analyze information including video or sound within the segment of the content the user wants, and transmit different data to the electronic device (101, 1413) according to the user request.For example, when content corresponding to a user request and second information is transmitted to an electronic device (101, 1413), the content transmitted to the electronic device (101, 1413) may be content that has already been secured and held by the content provider in relation to a specific section of the first content (for example, if the user requests fashion information of the content, the corresponding fashion information content advertisement secured from another business operator), or it may be content that corresponds to a newly generated result created by using a generative AI model managed by the content provider's AI agent, taking the original data of the first content as input and reflecting the user request in the prompt.

[0168] According to one embodiment, in operation 1457, a content provider (e.g., a server of the content provider (e.g., 108 in FIG. 1)) transmits information of operation 1455 to an electronic device (101, 1413), and in operation 1459, the electronic device (101, 1413) can analyze information received from the content provider (e.g., a server of the content provider (e.g., 108 in FIG. 1)). For example, the electronic device (101, 1413) can identify the type of received information. The electronic device (101, 1413) can analyze and determine the type of information received from the content provider and define a workflow for subsequent operations to perform a user request according to the type of information in an AI agent within the electronic device (101, 1413). An example of a workflow is as follows.

[0169] According to one embodiment, the electronic device (101, 1413) may obtain second information about the first content (e.g., the first content element of FIG. 3 (e.g., the original reference data of FIG. 4a)) in the electronic device (101, 1413) based on receiving information about a right to temporarily access the first content within the electronic device (101, 1413) from a content provider (e.g., the server of the content provider (e.g., 108 of FIG. 1)) according to the said right.

[0170] According to one embodiment, an electronic device (101, 1413) may perform an operation 1443 based on receiving information about original reference data of the second information of the first content (e.g., a capture screen of a scene corresponding to a user request) from a content provider (e.g., a server of the content provider (e.g., 108 of FIG. 1)), or information about alternative data of the second information of the first content (e.g., data in which specific information within the video content (e.g., person information unrelated to the user request) is deleted and descriptive information about the scene corresponding to the user request is included, or data similar to the scene corresponding to the user request).

[0171] According to one embodiment, in operation 1443 following operation 1459, the electronic device (101, 1413) can search for data corresponding to a user request outside the electronic device (101, 1413) through a search application (e.g., a browser) based on original reference data (e.g., a capture screen of a scene corresponding to a user request) or alternative data (e.g., data in which specific information within the video content (e.g., person information unrelated to the user request) is deleted and descriptive information about a scene corresponding to a user request is included, or data similar to a scene corresponding to a user request) received from a content provider (e.g., a server of a content provider (e.g., 108 of FIG. 1)). According to one embodiment, in operation 1445, the electronic device (101, 1413) generates a prompt for generating result data corresponding to a user request based on the retrieved data, and in operation 1447, the electronic device (101, 1413) may cause result data corresponding to a user request to be generated based on the prompt. In operation 1449, the electronic device (101, 1413) may provide second content through the UI of an AI assistant (e.g., AI agent) by processing the result data into a format corresponding to an AI assistant (e.g., AI agent) based on the user request, or provide second content by processing the result data into a format corresponding to a second application corresponding to the user request.

[0172] According to one embodiment, when an electronic device (101, 1413) receives original reference data (e.g., a captured screen of a scene corresponding to a user request) from a content provider, it may perform an anonymization operation on the original reference data so that it is not used for training the AI ​​model of the electronic device (101, 1413), or delete the original reference data after use. According to one embodiment, the electronic device (101, 1413) may classify a second content generated based on the original reference data (e.g., a captured screen of a scene corresponding to a user request) from a content provider as secure content or as conditional content available only for a specific period.

[0173] According to one embodiment, the electronic device (101, 1413) may perform the 1449 operation based on receiving content corresponding to a user request and second information from a content provider (e.g., a server of a content provider (e.g., 108 in FIG. 1)).

[0174] According to one embodiment, in operation 1449 following operation 1459, the electronic device (101, 1413) may provide a second content through the UI of an AI assistant (e.g., AI agent) by processing content received from a content provider (e.g., server of a content provider (e.g., 108 in FIG. 1)) into a format corresponding to an AI assistant (e.g., AI agent) of the electronic device (101, 1413), or provide a second content by processing content received from a content provider (e.g., server of a content provider (e.g., 108 in FIG. 1)) into a format corresponding to a second application that responds to a user request.

[0175] FIG. 15 is a flowchart of a method of operation of an electronic device according to one embodiment. FIG. 15 can be explained based on the embodiments described above, the embodiments of FIG. 16, and the embodiments described below. FIG. 16 is a diagram illustrating the operation of an electronic device according to one embodiment.

[0176] With reference to the operations of FIGS. 15 and 16, operations 407 through 415 of FIG. 4a can be explained. The operations of FIGS. 15 and 16 can be performed after operation 405 of FIG. 4a is performed. With reference to FIGS. 15 and 16, operations for using data of security content stored in the electronic device (101) can be explained. For example, the electronic device (101) may provide a function that generates content that requires user authentication and provides it as security content by temporarily using the data of security content, or extracts the data of security content and, when input into a generative AI model within the electronic device (101), does not store the data of the security content in the knowledge database of the generative AI model so that it is used only once, so that the data of the security content is not used for learning.

[0177] At least some of the operations of FIG. 15 may be omitted. The order of the operations of FIG. 15 may be changed. Operations other than those of FIG. 15 may be performed before, during, or after the operations of FIG. 15.

[0178] Referring to FIG. 15, in operation 1501, according to one embodiment, an electronic device (101) can access secure content (e.g., first content) through a first application (e.g., a document editor, a notepad). For example, the electronic device (101) can confirm a request for execution of secure content (e.g., first content) through the first application (e.g., a document editor, a notepad). In operation 1503, according to one embodiment, the electronic device (101) can confirm user access to secure content. The electronic device (101) can confirm user access to secure content based on the fact that the first content is secure content, in response to a request for execution of the first content through the first application (e.g., a document editor, a notepad). For example, the electronic device (101) can identify the first content as secure content based on the fact that the first content is content stored in a secure area within the memory (130). For example, the electronic device (101) can identify the first content as secure content based on the fact that the first application on the first screen containing the first content is classified as a secure application. For example, the electronic device (101) can identify the first content as secure content based on the fact that a password is set on the first content. For example, the electronic device (101) can identify the first content as secure content based on the fact that authentication is required to verify the entire content of the first content.

[0179] In operation 1505, according to one embodiment, the electronic device (101) may provide a list of accessible items (e.g., conditional content or restricted content) provided by the security content. The security content may include accessible items (e.g., conditional content or restricted content) as user authentication is performed. The electronic device (101) may provide a list of accessible items (e.g., conditional content or restricted content) provided by the security content and request authentication to verify information about said items.

[0180] In operation 1507, according to one embodiment, the electronic device (101) may perform authentication based on user input and, upon completion of authentication, provide the user with an accessible item of secure content (e.g., conditional content or restricted content).

[0181] 1509 In operation, according to one embodiment, the electronic device (101) may select at least some of the accessible items of security content (e.g., conditional content or restricted content) based on the user's choice.

[0182] In operation 1511, according to one embodiment, the electronic device (101) may reprocess content based on information regarding a selected item (e.g., conditional content or restricted content). For example, the electronic device (101) may pass information regarding an item selected by a user (e.g., conditional content or restricted content) as input to a generative AI model and output content reprocessed to a level that can be disclosed to the user using the generative AI model. According to one embodiment, the electronic device (101) may display a notification indicating that the content selected by the user is being reprocessed.

[0183] In operation 1513, according to one embodiment, the electronic device (101) can detect a user request to use reprocessed content. For example, the electronic device (101) can detect an operation to use an AI model within the electronic device (101) using reprocessed content provided by a generative AI model through user input (e.g., text, voice, drawing).

[0184] In operation 1515, according to one embodiment, the electronic device (101) can determine whether a user request affects the AI ​​model of the electronic device (101). For example, the electronic device (101) can transmit the content of the operation corresponding to the user request as an input to an LLM to analyze, and determine whether the operation corresponding to the user request affects the parameters of the AI ​​model of the electronic device (101) based on the analysis result of the LLM.

[0185] In operation 1517, according to one embodiment, the electronic device (101) can perform security processing on reprocessed content based on determining that a task responding to a user request may change parameters by involving the learning process of the AI ​​model of the electronic device (101).

[0186] 1519 In operation, according to one embodiment, the electronic device (101) can perform a user request based on securely processed reprocessed content.

[0187] For example, in FIG. 16 (a), the electronic device (101) can confirm a user access request for security content (1610) within the electronic device (101). In FIG. 16 (b), the electronic device (101) can provide (1620) a list of items (e.g., conditional content or restricted content) that are partially accessible during user authentication (e.g., account login, biometric authentication, password input) of the security content. In FIG. 16 (c), once authentication is complete, the electronic device (101) can determine, based on user selection, which content to view among the conditionally accessible content items (1630) provided in the security content. The electronic device (101) can transmit information about the partially accessible content selected by the user as input to a generative AI model in accordance with the user request (1631). The generative AI model of the electronic device (101) can receive information on restricted content and output data that has been reprocessed to a level that can be disclosed to the user. The electronic device (101) can display a notification indicating that the content selected by the user is being reprocessed. The electronic device (101) transmits the reprocessed data as input to an LLM to analyze the task requested by the user, and can determine whether the user's requested task affects the parameters of the artificial intelligence model within the device based on the LLM analysis results. If the electronic device (101) determines that the user's requested task interferes with the learning process of the artificial intelligence model within the device and may change the parameters, it can perform security processing (e.g., anonymization operation) on the reprocessed content. The electronic device (101) can perform the task requested by the user based on the security-applied reprocessed content and provide the result (1640).

[0188] For example, when a user wants to perform a specific task using content stored within the electronic device (101) through an application (e.g., notepad, diary, video editor), the electronic device (101) may receive a user access request signal for such content if the content is pre-classified as security content. In order to grant partial access rights to only authorized users among the security content, the electronic device (101) may allow only authorized users to access the security content through an authentication procedure that includes user account authentication and biometric authentication (e.g., facial recognition, fingerprint recognition, iris recognition). Once the user authentication procedure is completed, the electronic device (101) may display some publicly available details regarding the security content and allow the user to select one or more of these items. For example, when a user wants to access security content regarding 'XX program data,' which is statistical data containing user personal information about a specific program, the electronic device (101) may display items included in this data, such as user gender, user age, and user country, as partially publicable items to the user, and prevent the user from viewing items containing sensitive personal information, such as user resident registration number, user contact information, and user address. If the user selects one or more of the partially publicable data items, the electronic device (101) may transmit those items as input to a generative AI model within the electronic device (101) or to a generative AI model in a server (e.g., server (108) of FIG. 1). If the type of data transmitted as input to the generative AI model is text, the electronic device (101) may perform only embedding to binarize the text. The electronic device (101) can perform resizing to transmit the image at the size requested by the AI ​​model when the data type is an image, and then divide the image into patches (e.g., 15×15, 21×21, etc.) to perform embedding.In this process, the electronic device (101) may also transmit positional encoding information as input to the generative AI model to remember the location information of the patch image for the entire image. The generative AI model outputs reprocessed content so that the user can recognize the item selected from the security content stored in a database. For example, the security content is quantized and binarized for each item, and the generative AI model can output this by converting it into a table or text type that the user can recognize. Using the reprocessed content output by the generative AI model, the user can receive a task using the AI ​​model within the electronic device (101). The user can input a command to modify a document already contained in the electronic device (101) using the reprocessed content. At this time, the command input may include text, voice, or gestures. If the type of command entered by the user is speech, a speech-to-text (STT) AI model is used to convert it into a text type, and if it is a gesture, a pose estimation AI model is used to map a predefined gesture into a text type to convert the type. The electronic device (101) can transmit the user request as input to the LLM to analyze the task content requested by the user. The LLM can perform string analysis from the user request text to identify the core action requested by the user and the corresponding subject. For example, in the user request, it can be seen that 'update content of XX program usage analysis report.xyz' is the core action and the subject is 'regenerated data'. It is determined whether performing the core action in the analyzed sentence affects the learning of the AI ​​model of the electronic device (101).As such, since the core operation is to save the modified document to the electronic device (101) rather than verifying and discarding the modified document once, there is a possibility that it may be used for training the AI ​​model of the electronic device (101) in the future. If it is determined that it will be used for training the AI ​​model of the electronic device (101), security may be applied to the electronic device (101) on the reprocessed content. The security application method may include local type and online type. For example, in the case of the local type using the AI ​​model of the electronic device (101), the electronic device (101) may apply a method including masking, hashing, and / or homomorphic encryption. For example, in the case of using the AI ​​model of a server (e.g., server (108) of FIG. 1), the electronic device (101) may apply security to the reprocessed content using a method that ensures the data used as input to the model does not leak outside the electronic device (101) even when using the AI ​​model on the server using federated learning techniques. Secured content is transmitted as input to the AI ​​model of the electronic device (101) along with a user request. According to the user request (e.g., text generation, voice editing, etc.), the AI ​​model of the electronic device (101) that has been pre-trained operates and a corresponding result can be output.

[0189] FIG. 17 is a flowchart of a method of operation of an electronic device according to one embodiment. FIG. 17 can be explained based on the embodiments described above and the embodiments described below.

[0190] At least some of the operations of FIG. 17 may be omitted. The order of the operations of FIG. 17 may be changed. Operations other than those of FIG. 17 may be performed before, during, or after the operations of FIG. 17.

[0191] Referring to FIG. 17, in operation 1701, according to one embodiment, an electronic device (101) can obtain security data. For example, the security data may include original reference data of access-restricted content.

[0192] In operation 1703, according to one embodiment, the electronic device (101) may perform an anonymization operation on the security data so that the security data is not used as training data for an artificial intelligence model. For example, operations 1701 and 1703 may be performed when original data related to content is received from a content provider, or when original data regarding security content within the electronic device (101) is obtained. The electronic device (101) may perform an anonymization process so that the security content is not used for training an artificial intelligence model, or perform a process of deleting it after one-time use so that it is not referenced as knowledge by the artificial intelligence model, and may create the final content created using the security data as security content requiring user authentication or as conditional content that can be used only for a specific period.

[0193] FIG. 18 is a flowchart of a method of operation of an electronic device according to one embodiment. FIG. 18 can be explained based on the embodiments described above, the embodiments of FIG. 19, and the embodiments described below. FIG. 19 is a diagram illustrating the operation of an electronic device according to one embodiment.

[0194] Referring to FIG. 18, in a search result searched through a search application (e.g., a browser), the full text of the first content included in the searched link (e.g., a site) is not displayed, and only some data of the first content is displayed, the operation of the electronic device (101) processing a user request for the first content can be described.

[0195] At least some of the operations of FIG. 18 may be omitted. The order of the operations of FIG. 18 may be changed. Operations other than those of FIG. 18 may be performed before, during, or after the operations of FIG. 18.

[0196] Referring to FIG. 18, in operation 1801, according to one embodiment, an electronic device (101) may display a first screen (e.g., 1900 of FIG. 19) containing some data of first content included in a searched link (e.g., a site provided by an external server) based on a search result searched through a search application (e.g., a browser). For example, in FIG. 19, the electronic device (101) may display some data (1930) of the first content included in a searched link (e.g., a site provided by an external server) based on the search result on a screen (1900) of a search application (1910) including a search window (1920). For example, some data (1930) of the data of the first content included in the searched link (e.g., a site provided by an external server) may include the address (e.g., uniform resource locator) of the searched link (e.g., a site provided by an external server) and some of the content of the first content. For example, the electronic device (101) may provide a screen that includes some of the content of a specific paper provided by a specific site based on a search based on user input.

[0197] In operation 1803, according to one embodiment, the electronic device (101) can check a user request for the first content. For example, the user request may be "summarize the searched papers."

[0198] In operation 1805, according to one embodiment, the electronic device (101) may determine that additional data related to the first content is required to perform a user request. For example, the search result of FIG. 19 includes some data (1930) among the data included in the corresponding link, so additional data may be required to process the user request.

[0199] In operation 1807, according to one embodiment, the electronic device (101) may determine that authentication is required to obtain first additional data (e.g., the full text of the first content) related to the first content displayed on the first screen (e.g., 1900 in FIG. 19). For example, to access a link included in the search results of FIG. 19 and obtain additional data (e.g., the full text of the first content), it may be necessary to log in or subscribe to the site. For example, an image or video displayed on the first screen (e.g., 1900 in FIG. 19) may not be saved or captured. For example, to obtain first additional data (e.g., the full text of the first content) related to the first content displayed on the first screen (e.g., 1900 in FIG. 19), it may be necessary to watch an advertisement. In this case, the electronic device (101) may perform operation 1809 to obtain additional data without authentication (e.g., logging in, subscribing, or watching an advertisement). According to one embodiment, the electronic device (101) can access the site and obtain first additional data (e.g., full text of the first content) for the first content based on the fact that it is not necessary to log in (or subscribe or watch an advertisement) to the site in order to access the link included in the search results of FIG. 19 and obtain additional data (e.g., full text of the first content).

[0200] In operation 1809, according to one embodiment, the electronic device (101) can identify keywords related to the first content (e.g., a search result containing some data of the first content). For example, the electronic device (101) can extract keywords related to the first content (e.g., a search result containing some data of the first content) based on the search result of FIG. 19 and a user request.

[0201] In operation 1811, according to one embodiment, the electronic device (101) may obtain second additional data related to the first content through another path without authentication based on the extracted keywords. For example, to obtain the first additional data by accessing the link of the search result of FIG. 19, it is necessary to log in to the site (or subscribe or watch an advertisement), but to obtain the second additional data through another path by performing a search again based on the extracted keywords, separate authentication may not be required. The electronic device (101) may obtain data related to the first content (e.g., search result) through operations 1801 to 1811. For example, to obtain information about the first content, an analysis of the webpage displaying the first content (e.g., text analysis, webpage code analysis, or webpage display screen analysis) is performed, and based on the determination that the first content is restricted access content, accessible data within the webpage displaying the restricted access content (e.g., content title, publicly available parts of specific content, comments regarding the content) is obtained, and then the acquired accessible data is used to additionally search for the webpage most similar to the restricted access content to obtain data related to the first content (e.g., search results).

[0202] FIG. 20 is a flowchart of a method of operation of an electronic device according to one embodiment. FIG. 20 can be explained based on the embodiments described above, the embodiments of FIG. 21, and embodiments described below. FIG. 21 is a diagram illustrating the operation of an electronic device according to one embodiment.

[0203] The operations of FIGS. 20 and 21 can be performed after the operation 407 of FIG. 4a is performed. With reference to FIGS. 20 and 21, an operation to determine a method of acquiring alternative data according to the type of content can be described.

[0204] At least some of the operations of FIG. 20 may be omitted. The order of the operations of FIG. 20 may be changed. Operations other than those of FIG. 20 may be performed before, during, or after the operations of FIG. 20.

[0205] Referring to FIG. 20, in operation 2001, according to one embodiment, the electronic device (101) may determine that alternative data related to the first content is required based on confirming that the first content corresponding to the user request is access restricted content. In operation 2003, according to one embodiment, the electronic device (101) may determine a method for obtaining alternative data based on the type of the first content. The electronic device (101) may determine the type of the first content and determine a method for obtaining alternative data for the first content. In operation 2005, according to one embodiment, the electronic device (101) may obtain alternative data related to the first content based on the determined method of acquisition.

[0206] For example, in the operation of 2101 of FIG. 21, according to one embodiment, the electronic device (101) can identify the type of the first content.

[0207] According to one embodiment, the content may be local security content. The local security content may be security content stored in the memory (130) of the electronic device (101). In operation 2103, the electronic device (101) may determine to acquire alternative data in a first manner based on the fact that the content is local security content. The first manner may include a method of acquiring data by the operations described in FIGS. 15 and 16. However, the method of acquiring data by the operations described in FIGS. 15 and 16 is not applicable only when acquiring data of local security content. The electronic device (101) may acquire data of other content other than the data of local security content through the method of acquiring data by the operations described in FIGS. 15 and 16.

[0208] According to one embodiment, the content may be external security content. The external security content may be the content of FIG. 18. For example, in an embodiment where only some data of the search results in FIG. 18 is displayed on the screen and authentication is required to check additional data, the search results displayed on the screen or the content confirmed by accessing the corresponding link may be external security content. In operation 2105, the electronic device (101) may decide to acquire alternative data in a second manner based on the fact that the content is external security content. The second manner may include a method of acquiring data by the operations described in FIG. 18 and FIG. 19. However, the method of acquiring data by the operations described in FIG. 18 and FIG. 19 is not applied only when acquiring data of external security content. The electronic device (101) may acquire data of other content other than the data of external security content through the method of acquiring data by the operations described in FIG. 18 and FIG. 19.

[0209] According to one embodiment, the content may be content of a content provider. In operation 2107, the electronic device (101) may determine to acquire replacement data in a third method or a fourth method based on the fact that the content is content of a content provider. The third method may include a method of acquiring data by the operations described in FIG. 11. For example, the electronic device (101) may acquire data regarding the first content by utilizing an artificial intelligence agent provided by the content provider based on the fact that the first content is content of a content provider. The fourth method may include a method of acquiring data by the operations described in FIG. 8. For example, the electronic device (101) may display a UI for acquiring additional information from a user based on the fact that the first content is content of a content provider, acquire additional information related to the first content based on user input through the UI, and acquire replacement data using the additional information. For example, the electronic device (101) may display a UI for obtaining additional information from the user based on the fact that the content provider of the first content does not provide its own artificial intelligence agent. Operations after obtaining alternative data according to the operations of FIG. 20 and FIG. 21 can be understood by referring to the embodiments of the previously described drawings.

[0210] FIG. 22 is a flowchart of a method of operation of an electronic device according to one embodiment. FIG. 22 can be explained based on the embodiments described above and the embodiments described below.

[0211] The operations of FIG. 22 can be performed after operation 409 or operation 413 of FIG. 4a has been performed. With reference to FIG. 22, an operation to determine the method of processing data based on the type of acquired data can be described.

[0212] At least some of the operations of FIG. 22 may be omitted. The order of the operations of FIG. 22 may be changed. Operations other than those of FIG. 22 may be performed before, during, or after the operations of FIG. 22.

[0213] Referring to FIG. 22, in operation 2201, according to one embodiment, the electronic device (101) can determine the type of acquired data. In operation 2203, according to one embodiment, the electronic device (101) can determine the method of processing data according to a user request based on the type of acquired data.

[0214] For example, the electronic device (101) may, in FIGS. 11 and 12, transmit a first request to a content provider to request data related to a first content necessary to perform a user request, and receive data related to the first content from the content provider. Based on receiving an original image corresponding to the content from the content provider, the electronic device (101) may generate content corresponding to the user request using the original image, and manage the original image so that it is not used for training an artificial intelligence model. Based on receiving a replacement image corresponding to the content from the content provider, the electronic device (101) may generate content corresponding to the user request using the replacement image. Based on receiving reference information related to the first content from the content provider, the electronic device (101) may perform a search using the reference information and generate content corresponding to the user request using the search data as replacement data. The electronic device (101) can display the received content on a screen based on receiving the content itself provided by the content provider's content server (e.g., server (108) of FIG. 1) in response to a user request.

[0215] FIG. 23 is a drawing illustrating the operation of an electronic device according to one embodiment.

[0216] Referring to FIG. 23, according to one embodiment, the electronic device (101) may include a wearable device in the form of glasses (e.g., 101 of FIG. 23). The electronic device (101) may provide an XR (e.g., AR, VR, MR) interface. In FIG. 23, 2330 may be an area that displays the physical environment outside the electronic device (101) as is, or an area that displays the external environment acquired by a camera (e.g., camera module (180)). In FIG. 23, 2310 and 2360 may be areas that display a digitized virtual UI.

[0217] According to one embodiment, an electronic device (101) can receive a user request to be performed using information from an external environment (e.g., 2330) while being worn by a user (2300). The electronic device (101) can check whether there is a screen (e.g., 2330) of another device (e.g., 102 or 104 of FIG. 1) (e.g., TV) in the external environment. Based on checking the screen (e.g., 2330) of another device (e.g., 102 or 104 of FIG. 1) (e.g., TV), the electronic device (101) can request information about the screen from the other device (e.g., 102 or 104 of FIG. 1) (e.g., TV). The electronic device (101) and the other device (e.g., 102 or 104 of FIG. 1) (e.g., TV) can be connected via wireless or wired communication to exchange information. According to one embodiment, based on receiving information about a screen from another device (e.g., 102 or 104 of FIG. 1) (e.g., TV), the electronic device (101) can perform a user request based on the received information.

[0218] According to one embodiment, based on not receiving information about a screen from another device (e.g., 102 or 104 of FIG. 1) (e.g., TV) (e.g., based on the content displayed on the screen of another device (e.g., 102 or 104 of FIG. 1) (e.g., TV) being restricted access content), an electronic device (101) can perform a user request using information (e.g., 2330) identified by a camera (e.g., camera module (180)). For example, the electronic device (101) can obtain at least one of text information, image information, or context information from at least one screen (e.g., 2330) present on a camera preview. For example, the electronic device (101) may capture video or images of an external environment including a screen (e.g., 2330) of another device (e.g., 102 or 104 of FIG. 1) (e.g., TV) through a camera (e.g., camera module (180)) operatively connected to the electronic device (101). For example, the electronic device (101) may delete the information obtained through the camera (e.g., camera module (180)) after use or store it in a secure area of ​​memory (130). In the embodiment of FIG. 23, input of a user request may be received through another device (e.g., 102 or 104 of FIG. 1) (e.g., TV) that displays the screen (e.g., 2330). If it is identified that an access restriction setting is applied to a part of another device (e.g., 102 or 104 in FIG. 1) (e.g., TV), information acquired (e.g., captured) through the electronic device (101) may be transmitted to another device (e.g., 102 or 104 in FIG. 1) (e.g., TV) to perform a user request on the other device (e.g., 102 or 104 in FIG. 1) (e.g., TV).

[0219] FIG. 24 is a flowchart of a method of operation of an electronic device according to one embodiment. FIG. 24 can be explained based on the embodiments described above and the embodiments described below.

[0220] At least some of the operations of FIG. 24 may be omitted. The order of the operations of FIG. 24 may be changed. Operations other than those of FIG. 24 may be performed before, during, or after the operations of FIG. 24.

[0221] Referring to FIG. 24, in operation 2401, according to one embodiment, an electronic device (101) may display a first screen of a first application. The first screen may include first content. The operation of displaying the first screen of the first application in operation 2401 may include an operation of providing first content through the first application of FIG. 3 (e.g., an operation of displaying the first screen of the first application including the first content of FIG. 4a).

[0222] In operation 2403, according to one embodiment, the electronic device (101) may receive a request to share the first screen with an external device (e.g., 102 or 104 of FIG. 1). For example, the operation of sharing the first screen with an external device (e.g., 102 or 104 of FIG. 1) may be an operation of providing data related to the first screen to the external device (e.g., 102 or 104 of FIG. 1) so that the first screen can be screen-mirrored on the external device (e.g., 102 or 104 of FIG. 1).

[0223] In operation 2405, according to one embodiment, the electronic device (101) can determine that the first content of the first screen is restricted access content. For example, the electronic device (101) can determine that the first content is restricted access content based on determining that the first content is content to which restrictions regarding the method of use or the method of access are imposed for copyright protection. For example, the electronic device (101) can determine that the first content is restricted access content based on determining that the first content is secure content stored in the electronic device (101). The operation of determining that the first content is restricted access content has been described in detail with reference to the above-described embodiments.

[0224] In operation 2407, according to one embodiment, an electronic device (101) can obtain alternative data for a first content that is restricted access content. For example, the electronic device (101) can obtain alternative data for the first content by using an API (application programming interface) provided by an application providing the first content that performs the function of providing alternative data for the first content. For example, the electronic device (101) can request and receive alternative data for the first content from an AI agent dedicated to the application providing the first content. For example, the electronic device (101) can obtain alternative data for the first content by using standard technology (e.g., MCP (model context protocol), A2A (Agent to Agent)) in an AI agent that receives and processes user requests, performing a process of analyzing the context of a user request to search for and connect to the service of the most appropriate external AI agent to process the user request, and obtaining alternative data for the first content through the service of the external AI agent. In this case, the external AI agent may be an AI agent of a content provider providing the first content, or the content provider providing the first content may be an AI agent of another business operator. The electronic device (101) may additionally perform operations including verification or access control of the received alternative data to check the reliability of the alternative data.

[0225] In operation 2409, according to one embodiment, the electronic device (101) may share alternative data related to the first content with an external device (e.g., 102 or 104 of FIG. 1). The external device (e.g., 102 or 104 of FIG. 1) may perform screen mirroring of the first screen of the electronic device (101) based on the received data.

[0226] Unlike the conventional technology in which, when the first content displayed on the screen of an electronic device (101) is an access-restricted application (e.g., a video application or a messenger mail application), the area where the application is displayed is blanked out by screen mirroring on an external device (e.g., 102 or 104 of FIG. 1), for example, the electronic device (101) can obtain alternative data for the access-restricted first content, such as providing only a summary or publicly available information based on the original content including a message or mail received from the application, based on the embodiment of FIG. 24, and display it on an external device (e.g., 102 or 104 of FIG. 1).

[0227] Those skilled in the art will understand that the embodiments described herein may be applied interchangeably to the extent applicable. For example, those skilled in the art will understand that at least some operations of an embodiment described herein may be omitted, and at least some operations of the embodiments may be applied interchangeably.

[0228] The present disclosure is not limited to the foregoing, and other unmentioned variations will be apparent to those skilled in the art from the present disclosure.

[0229] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0230] According to one embodiment, the electronic device (101) may include at least one processor (120) comprising a processing circuit and a memory (130) for storing instructions. When the instructions are executed individually or collectively by at least one processor (120), they may cause the electronic device (101) to provide a first content through a first application. When the instructions are executed individually or collectively by at least one processor (120), they may cause the electronic device (101) to check for a user request related to the first content while providing the first content. When the instructions are executed individually or collectively by at least one processor (120), they may cause the electronic device (101) to attempt to obtain a first content element from the first content based on the user request. When the above instructions are executed individually or collectively by at least one processor (120), they may cause the electronic device (101) to acquire at least one second content element corresponding to the first content element, based at least in part on a determination that the acquisition of the first content element is restricted. When the above instructions are executed individually or collectively by at least one processor (120), they may cause the electronic device (101) to generate a response to the user request using the at least one second content element instead of the first content element. When the above instructions are executed individually or collectively by at least one processor (120), they may cause the electronic device (101) to provide the response.

[0231] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the second content to be generated through a second application using the at least one second content element. When the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the second content to be provided as at least part of the response.

[0232] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the second content to be generated by converting the at least one second content element to correspond to the user interface of the second application.

[0233] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to determine that the acquisition of the first content element is restricted based on identifying that a capture operation for the screen area where the first content is displayed is prohibited.

[0234] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to obtain at least a portion of the second content element by searching outside the electronic device (101) for the at least one second content element that can replace the first content element by using information related to the first content accessible within the first application.

[0235] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to obtain at least a portion of the second content element by searching outside the electronic device (101) for the at least one second content element that can replace the first content element, using additional information describing the first content element input by a user of the electronic device (101) after the decision.

[0236] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause to receive at least a portion of the second content element from the provider of the first content after the decision.

[0237] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause at least a portion of the second content element to be deleted from the electronic device (101) after providing the response.

[0238] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to receive at least a portion of the second content element from a wearable electronic device connected to the electronic device (101) and worn on the face of the user of the electronic device (101).

[0239] According to one embodiment, the second content element may include at least a portion of an image captured through a camera of the wearable electronic device.

[0240] According to one embodiment, a method of operation of an electronic device (101) may include an operation of providing a first content through a first application. The method may include an operation of checking a user request related to the first content while providing the first content. The method may include an operation of attempting to obtain a first content element from the first content based on the user request. The method may include an operation of obtaining at least one second content element corresponding to the first content element based at least partially on a determination that the acquisition of the first content element is restricted. The method may include an operation of causing a response to the user request to be generated using the at least one second content element instead of the first content element. The method may include an operation of providing the response.

[0241] According to one embodiment, the method may include an operation of generating second content through a second application using at least one second content element. The method may include an operation of providing the second content as at least part of the response.

[0242] According to one embodiment, the method may include the operation of generating the second content by converting the at least one second content element to correspond to the user interface of the second application.

[0243] According to one embodiment, the method may include an operation of determining that the acquisition of the first content element is restricted based on identifying that a capture operation for a screen area where the first content is displayed is prohibited.

[0244] According to one embodiment, the method may include the operation of obtaining at least a portion of the second content element by searching for at least one second content element capable of replacing the first content element outside the electronic device (101) using information related to the first content accessible within the first application.

[0245] According to one embodiment, the method may include the operation of obtaining at least a portion of the second content element by searching for at least one second content element capable of replacing the first content element outside the electronic device (101) using additional information describing the first content element input by a user of the electronic device (101) after the determination.

[0246] According to one embodiment, the method may include the operation of receiving at least a portion of the second content element from the provider of the first content after the determination.

[0247] According to one embodiment, the method may include the operation of deleting at least a portion of the second content element from the electronic device (101) after providing the response.

[0248] According to one embodiment, the method may include receiving at least a portion of the second content element from a wearable electronic device connected to the electronic device (101) and worn on the face of a user of the electronic device (101).

[0249] According to one embodiment, in the method, the second content element may include at least a portion of an image captured through a camera of the wearable electronic device.

[0250] According to one embodiment, in a non-transitory computer-readable recording medium storing instructions, the instructions may cause the electronic device (101) to perform at least one operation when executed individually or collectively by at least one processor (120) of the electronic device (101). The at least one operation may include an operation of providing a first content through a first application. The at least one operation may include an operation of checking a user request related to the first content while providing the first content. The at least one operation may include an operation of attempting to obtain a first content element from the first content based on the user request. The at least one operation may include an operation of obtaining at least one second content element corresponding to the first content element based at least partially on a determination that the acquisition of the first content element is restricted. The at least one operation may include an operation that causes a response to the user request to be generated using the at least one second content element instead of the first content element. The at least one operation may include an operation that provides the response.

[0251] According to one embodiment, the at least one operation may include an operation of generating second content through a second application using the at least one second content element. The at least one operation may include an operation of providing the second content as at least part of the response.

[0252] According to one embodiment, the at least one operation may include an operation of generating the second content by converting the at least one second content element to correspond to the user interface of the second application.

[0253] According to one embodiment, the at least one operation may include an operation of determining that the acquisition of the first content element is restricted based on identifying that a capture operation for the screen area where the first content is displayed is prohibited.

[0254] According to one embodiment, the at least one operation may include an operation of obtaining at least a portion of the second content element by searching for the at least one second content element capable of replacing the first content element outside the electronic device (101) using information related to the first content accessible within the first application.

[0255] According to one embodiment, the at least one operation may include an operation of obtaining at least a portion of the second content element by searching for the at least one second content element capable of replacing the first content element outside the electronic device (101) using additional information describing the first content element input by a user of the electronic device (101) after the determination.

[0256] According to one embodiment, the at least one operation may include receiving at least a portion of the second content element from the provider of the first content after the determination.

[0257] According to one embodiment, the at least one operation may include deleting at least a portion of the second content element from the electronic device (101) after providing the response.

[0258] According to one embodiment, the at least one operation may include receiving at least a portion of the second content element from a wearable electronic device connected to the electronic device (101) and worn on the face of a user of the electronic device (101).

[0259] According to one embodiment, in the recording medium, the second content element may include at least a portion of an image captured through a camera of the wearable electronic device.

[0260] According to one embodiment, the electronic device (101) may include a display module (160) including a display, at least one processor (120) including a processing circuit, and a memory (130) for storing instructions. When the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the electronic device (101) to display a first screen of a first application. The first screen may include first content. When the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the electronic device (101) to confirm a user request to create new content using the first content of the first screen. When the above instructions are executed individually or collectively by at least one processor (120), they may cause the electronic device (101) to determine whether the first content is access-restricted content based on the fact that data related to the first content is required to perform the user request. Access-restricted content may be content in which access to original reference data is restricted. When the above instructions are executed individually or collectively by at least one processor (120), they may cause the electronic device (101) to obtain substitute data to replace the original reference data based on the fact that the first content is access-restricted content. When the above instructions are executed individually or collectively by at least one processor (120), they may cause the electronic device (101) to perform the user request using the substitute data.

[0261] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the first result data to be generated based on the alternative data of the first content and the user request. When the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the second screen containing second content corresponding to the user request to be displayed based on the first result data.

[0262] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to identify a second application corresponding to the user request. By processing the first result data into a format corresponding to the second application, the second content may be obtained.

[0263] According to one embodiment, the electronic device (101) may include a communication module (190) including a communication circuit. When the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the content provider of the first application to check whether there is an artificial intelligence agent provided by the content provider, based on the fact that the first content is content provided by an external content provider. When the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the content provider to send a first request to the content provider to request data related to the first content necessary to perform the user request, based on the fact that the artificial intelligence agent of the content provider exists. When the above instructions are executed individually or collectively by at least one processor (120), they may cause the electronic device (101) to perform the user request based on data provided by the content provider based on the first request.

[0264] According to one embodiment, the data provided by the content provider based on the first request may include original reference data corresponding to a screen containing the first content, replacement data replacing the original reference data, reference information related to the first content provided by the content provider, result data generated by the content server of the content provider based on the user request, and / or content provided by the content server of the content provider in response to the user request.

[0265] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause a user interface (UI) to display on the first screen to obtain additional information from a user of the electronic device (101) based on the fact that the first content is the access restricted content. When the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the additional information related to the first content to be obtained based on user input through the UI. When the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the additional information to be used to perform a search through an external search server. When the above instructions are executed individually or collectively by at least one processor (120), they may cause the electronic device (101) to acquire search data provided by the search server as the alternative data of the first content.

[0266] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to obtain, through the UI for obtaining the additional information, text describing the first content or a drawing object entered by the user to represent the first content as the additional information based on the user input. When the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to perform the search using the additional information or to perform the search using data generated in the electronic device (101) using the additional information.

[0267] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to perform the search by generating a query based on the additional information and the public information regarding the first content. When the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to perform the search by generating a query based on the data generated in the electronic device (101) using the additional information and the public information regarding the first content. The public information may include information regarding the first content provided by the content provider of the first application while the first screen containing the first content is displayed.

[0268] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause to acquire a plurality of alternative data related to the first content. When the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause to display a screen for selecting one of the plurality of alternative data. When the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause to perform the user request using the alternative data selected based on user input.

[0269] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the first content to confirm that the first content is access restricted content based on the fact that the first content is security content. When the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the first content to obtain the original reference data of the first content or the replacement data that replaces the original reference data by performing authentication for the security content. When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the first content to be identified as the security content based on whether the first content is content stored in a security area within the memory (130), whether the first application is classified as a security application, whether a password is set on the first content, or whether authentication is required to verify the entire content of the first content.

[0270] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may cause the user to be informed that the first content is the access-restricted content based on confirming that the first content is the access-restricted content.

[0271] According to one embodiment, when the instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to perform an anonymization operation on the original reference data so that the original reference data is not used as training data for an artificial intelligence model, based on acquiring the original reference data of the first content.

[0272] According to one embodiment, a method of operation of an electronic device (101) may include an operation of displaying a first screen of a first application. The first screen may include first content. The method may include an operation of confirming a user request to create new content using the first content of the first screen. The method may include an operation of confirming whether the first content is access-restricted content based on the fact that data related to the first content is required to perform the user request. Access-restricted content may be content in which access to original reference data is restricted. The method may include an operation of obtaining substitute data to replace the original reference data based on the fact that the first content is access-restricted content. The method may include an operation of performing the user request using the substitute data.

[0273] According to one embodiment, the method may include an operation that causes a first result data to be generated based on the replacement data of the first content and the user request. The method may include an operation that displays a second screen containing second content corresponding to the user request based on the first result data.

[0274] According to one embodiment, the method may include an operation of identifying a second application corresponding to the user request. It may include an operation of obtaining the second content by processing the first result data into a format corresponding to the second application.

[0275] According to one embodiment, the method may include an operation of checking whether an artificial intelligence agent provided by the content provider of the first application exists, based on the fact that the first content is content provided by an external content provider. The method may include an operation of transmitting a first request to the content provider to request data related to the first content necessary to perform the user request, based on the existence of the artificial intelligence agent of the content provider. The method may include an operation of performing the user request based on the data provided by the content provider based on the first request.

[0276] According to one embodiment, in the method, the data provided by the content provider based on the first request may include original reference data corresponding to a screen containing the first content, replacement data replacing the original reference data, reference information related to the first content provided by the content provider, result data generated by the content server of the content provider based on the user request, and / or content provided by the content server of the content provider in response to the user request.

[0277] According to one embodiment, the method may include an operation of displaying a user interface (UI) on the first screen to obtain additional information from a user of the electronic device (101) based on the fact that the first content is the access-restricted content. The method may include an operation of obtaining the additional information related to the first content based on user input through the UI. The method may include an operation of performing a search through an external search server using the additional information. The method may include an operation of obtaining search data provided by the search server as the alternative data of the first content.

[0278] According to one embodiment, the method may include an operation of acquiring, through the UI for acquiring the additional information, text describing the first content or a drawing object input by the user to represent the first content as the additional information based on the user input. The method may include an operation of performing the search using the additional information or performing the search using data generated in the electronic device (101) using the additional information.

[0279] According to one embodiment, the method may include an operation of performing the search by generating a query based on the additional information and the public information regarding the first content. The method may include an operation of performing the search by generating a query based on the data generated in the electronic device (101) using the additional information and the public information regarding the first content. The public information may include information regarding the first content provided by the content provider of the first application while the first screen containing the first content is displayed.

[0280] According to one embodiment, the method may include an operation of acquiring a plurality of alternative data related to the first content. The method may include an operation of displaying a screen for selecting one of the plurality of alternative data. The method may include an operation of performing the user request using the alternative data selected based on user input.

[0281] According to one embodiment, the method may include an operation of confirming that the first content is access-restricted content based on the fact that the first content is security content. The method may include an operation of obtaining the original reference data of the first content or the replacement data that replaces the original reference data by performing authentication for the security content. The method may include an operation of confirming the first content as security content based on the fact that the first content is content stored in a security area within the memory (130), that the first application is classified as a security application, that a password is set on the first content, or that authentication is required to check the entire content of the first content.

[0282] According to one embodiment, the method may include an operation of informing a user that the first content is the access-restricted content based on confirming that the first content is the access-restricted content.

[0283] According to one embodiment, the method may include an operation to perform an anonymization operation on the original reference data so that the original reference data is not used as training data for an artificial intelligence model, based on obtaining the original reference data of the first content.

[0284] According to one embodiment, in a non-transitory computer-readable recording medium for storing instructions, the instructions may cause the electronic device (101) to perform at least one operation when executed individually or collectively by at least one processor (120) of the electronic device (101). The at least one operation may include an operation of displaying a first screen of a first application. The first screen may include first content. The at least one operation may include an operation of confirming a user request to create new content using the first content of the first screen. The at least one operation may include an operation of confirming whether the first content is access-restricted content based on the fact that data related to the first content is required to perform the user request. Access-restricted content may be content in which access to original reference data is restricted. The at least one operation may include an operation of obtaining substitute data to replace the original reference data based on the fact that the first content is the access-restricted content. The at least one operation may include an operation of performing the user request using the substitute data.

[0285] According to one embodiment, in the recording medium, the at least one operation may include an operation that causes a first result data to be generated based on the replacement data of the first content and the user request. The at least one operation may include an operation that displays a second screen containing second content corresponding to the user request based on the first result data.

[0286] According to one embodiment, in the recording medium, the at least one operation may include an operation of identifying a second application corresponding to the user request. The operation may include an operation of obtaining the second content by processing the first result data into a format corresponding to the second application.

[0287] According to one embodiment, in the recording medium, the at least one operation may include an operation of checking whether an artificial intelligence agent provided by the content provider of the first application exists based on the fact that the first content is content provided by an external content provider. The at least one operation may include an operation of transmitting a first request to the content provider to request data related to the first content necessary to perform the user request based on the existence of the artificial intelligence agent of the content provider. The at least one operation may include an operation of performing the user request based on the data provided by the content provider based on the first request.

[0288] According to one embodiment, in the recording medium, the data provided by the content provider based on the first request may include original reference data corresponding to a screen containing the first content, replacement data replacing the original reference data, reference information related to the first content provided by the content provider, result data generated by the content server of the content provider based on the user request, and / or content provided by the content server of the content provider in response to the user request.

[0289] According to one embodiment, in the recording medium, the at least one operation may include an operation of displaying a user interface (UI) on the first screen to obtain additional information from a user of the electronic device (101) based on the fact that the first content is the access restricted content. The at least one operation may include an operation of obtaining the additional information related to the first content based on user input through the UI. The at least one operation may include an operation of performing a search through an external search server using the additional information. The at least one operation may include an operation of obtaining search data provided by the search server as the alternative data of the first content.

[0290] According to one embodiment, in the recording medium, the at least one operation may include, through the UI for obtaining the additional information, an operation of obtaining text describing the first content or a drawing object entered by the user to represent the first content as the additional information based on the user input. The at least one operation may include an operation of performing the search using the additional information or performing the search using data generated in the electronic device (101) using the additional information.

[0291] According to one embodiment, in the recording medium, the at least one operation may include an operation of performing the search by generating a query based on the additional information and the public information regarding the first content. The at least one operation may include an operation of performing the search by generating a query based on the data generated in the electronic device (101) using the additional information and the public information regarding the first content. The public information may include information regarding the first content provided by the content provider of the first application while the first screen containing the first content is displayed.

[0292] According to one embodiment, in the recording medium, the at least one operation may include an operation of acquiring a plurality of alternative data related to the first content. The at least one operation may include an operation of displaying a screen for selecting one of the plurality of alternative data. The at least one operation may include an operation of performing the user request using the alternative data selected based on user input.

[0293] According to one embodiment, in the recording medium, the at least one operation may include an operation of confirming that the first content is access restricted content based on the fact that the first content is security content. The at least one operation may include an operation of obtaining the original reference data of the first content or the replacement data that replaces the original reference data by performing authentication for the security content. The at least one operation may include an operation of confirming the first content as security content based on the fact that the first content is content stored in a security area within the memory (130), the first application is classified as a security application, a password is set on the first content, or authentication is required to check the entire content of the first content.

[0294] According to one embodiment, in the recording medium, the at least one operation may include an operation of guiding a user that the first content is the access-restricted content based on confirming that the first content is the access-restricted content.

[0295] According to one embodiment, in the recording medium, the at least one operation may include an operation of performing an anonymization operation on the original reference data so that the original reference data is not used as training data for an artificial intelligence model, based on obtaining the original reference data of the first content.

[0296] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0297] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0298] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0299] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored on a storage medium readable by a machine (e.g., an electronic device). For example, a processor (e.g., a controller) of the machine may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0300] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0301] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101), At least one processor (120) including a processing circuit; and It includes a memory (130) for storing instructions, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is enabled, Provides the first content through the first application, and While providing the first content, check user requests related to the first content, and Based on the above user request, an attempt is made to obtain a first content element from the first content, and Based at least in part on the decision that the acquisition of the first content element is restricted, at least one second content element corresponding to the first content element is acquired, and Causing a response to the user request to be generated by using at least one second content element instead of the first content element, and Causing to provide the above response, Electronic device (101).

2. In Paragraph 1, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is enabled, Using the above at least one second content element, a second content is generated through a second application, and Causing to provide the above second content as at least part of the above response, Electronic device (101).

3. In Paragraph 1 or 2, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is enabled, By converting the above at least one second content element to correspond to the user interface of the second application, thereby causing the second content to be generated, Electronic device (101).

4. In any one of paragraphs 1 to 3, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is enabled, Based on identifying that a capture operation for the screen area where the first content is displayed is prohibited, causing to determine that the acquisition of the first content element is restricted, Electronic device (101).

5. In any one of paragraphs 1 to 4, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is enabled, By using information related to the first content accessible within the first application, searching for at least one second content element capable of replacing the first content element outside the electronic device (101), thereby causing to obtain at least a portion of the second content element. Electronic device (101).

6. In any one of paragraphs 1 through 5, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is enabled, By using additional information describing the first content element input by the user of the electronic device (101) after the above decision, searching for at least one second content element capable of replacing the first content element outside the electronic device (101), thereby causing to obtain at least a portion of the second content element. Electronic device (101).

7. In any one of paragraphs 1 through 6, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is enabled, Causing to receive at least a portion of the second content element from the provider of the first content after the above decision, Electronic device (101).

8. In any one of paragraphs 1 through 7, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is enabled, After providing the above response, causing at least a portion of the second content element to be deleted from the electronic device (101), Electronic device (101).

9. In any one of paragraphs 1 through 8, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is enabled, Causing to receive at least a portion of the second content element from a wearable electronic device connected to the electronic device (101) and worn on the face of the user of the electronic device (101), Electronic device (101).

10. In any one of paragraphs 1 through 9, The second content element comprises at least a portion of an image captured through the camera of the wearable electronic device. Electronic device (101).

11. In the method of operating the electronic device (101), The operation of providing the first content through the first application, and While providing the first content, an operation to check a user request related to the first content, and Based on the above user request, an operation to attempt to obtain a first content element from the first content, and Based at least in part on the decision that the acquisition of the first content element is restricted, the operation of acquiring at least one second content element corresponding to the first content element, and An operation that causes a response to the user request to be generated by using at least one second content element instead of the first content element, and including the operation of providing the above response, method.

12. In Paragraph 11, The operation of generating second content through a second application using at least one second content element, and The operation of providing the second content as at least part of the response, method.

13. In Paragraph 11 or 12, The method includes the operation of generating the second content by converting the at least one second content element to correspond to the user interface of the second application. method.

14. In any one of paragraphs 11 through 13, A method including an operation to determine that the acquisition of the first content element is restricted based on identifying that a capture operation for the screen area where the first content is displayed is prohibited. method.

15. In a non-transitory computer-readable recording medium for storing instructions, the instructions cause the electronic device (101) to perform at least one operation when executed individually or collectively by at least one processor (120) of the electronic device (101), and The above at least one operation is, The operation of providing the first content through the first application, and While providing the first content, an operation to check a user request related to the first content, and Based on the above user request, an operation to attempt to obtain a first content element from the first content, and Based at least in part on the decision that the acquisition of the first content element is restricted, the operation of acquiring at least one second content element corresponding to the first content element, and An operation that causes a response to the user request to be generated by using at least one second content element instead of the first content element, and including the operation of providing the above response, Recording media.

Images