Electronic device for extracting and utilizing context by using artificial intelligence model, computer-readable storage medium, and operating method thereof

By determining the importance of user interface data and applying automated weights, the electronic device optimizes context delivery to LLMs, addressing the limitations of fixed context size and enhancing user intent understanding and response quality.

WO2026240361A1PCT designated stage Publication Date: 2026-11-19SAMSUNG ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2026/005153
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-15
Filing Date
2026-04-02
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Large Language Models (LLMs) are limited by a fixed context size, leading to incomplete data transmission and reduced accuracy in understanding user intents due to the inability to handle large volumes of information effectively.

Method used

An electronic device and method that determines the importance of user interface data using automated weights, selectively providing context to LLMs based on user interaction and experience to optimize the quality of information within the limited context.

Benefits of technology

Improves the accuracy and efficiency of LLMs by maximizing the use of limited context, ensuring relevant information is transmitted, thereby enhancing user utterance understanding and response quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2026005153_19112026_PF_FP_ABST
    Figure KR2026005153_19112026_PF_FP_ABST
Patent Text Reader

Abstract

This electronic device comprises a memory storing instructions and including one or more storage media, and at least one processor including processing circuitry, wherein the instructions, when individually or collectively executed by the at least one processor, may control the electronic device to: receive a user input; display, in response to the user input, a user interface (UI) including a plurality of information items; identify a user input (interaction) or a viewing state for the plurality of information items displayed on the user interface (UI); determine importance of each of the plurality of information items on the basis of the identified user input or viewing state; determine, on the basis of the determined importance, a data level of context data to be provided to a large language model (LLM) from each information item; extract context data from the plurality of information items on the basis of the determined data level; generate a prompt including the extracted context data and the user input, and transmit the generated prompt to the LLM; receive, from the LLM, a response to the prompt; and perform an operation on the basis of the received response.
Need to check novelty before this filing date? Find Prior Art

Description

Electronic device for extracting and utilizing context using an artificial intelligence model, computer-readable storage medium, and method of operation thereof

[0001] This document concerns context extraction and management technologies utilizing Large Language Models (LLMs) in the field of voice assistants. In particular, it relates to a method for selecting and delivering optimal information in conjunction with user experience within a limited context window to effectively utilize LLMs in voice assistant systems.

[0002] Due to technical constraints, LLM is designed to receive only context of a limited size. When a user speaks a specific command via a voice assistant, the LLM generates and transmits context based on the information the user is currently seeing or knows. In this process, the type of information and the amount of information that can be transmitted as context can be important factors.

[0003] Conventional technology utilizes methods that either display information directly to the user or construct the entire result as a context to form an LLM prompt. Through this, the LLM can provide results by utilizing information from the context that matches or is similar to the user's intent. For example, when a user makes a utterance such as "Tell me my schedule for this month," a voice assistant can locate the schedules registered with the user and display the information for each schedule on the screen. While viewing this screen, the user can make various consecutive utterances, such as "Delete the first schedule" or "Delete schedules after Tuesday." The voice assistant can then understand the intent of the utterance and perform the next action based not only on the user's additional utterances but also on previously delivered screens, responses, and device status.

[0004] There is a growing trend of integrating voice assistants with LLM to provide natural conversational experiences and understand user intent based on various contexts. When utilizing LLM, a method can be employed that generates prompts based on diverse data—including conversational records containing user utterances and context—and executes the response received in order to fully grasp the user's intent. However, LLM is limited by the fact that each model can only use a specific size of context. In other words, to improve response quality, it is necessary to deliver as much useful information as possible within that limited context.

[0005] In conventional technology, conversation history or results are stored for each user utterance, and can be provided as context to an LLM to interpret subsequent utterances by utilizing the previously stored history and results. The LLM can generate commands by utilizing the utterance and the transmitted context. This method allows for the utilization of not only the user's utterance or provided text or voice responses, but also information regarding the results of view components provided through the UI, when generating additional responses.

[0006] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0007] Fundamentally, LLM has a limitation in that excess data cannot be used once the context size limit is reached. Additionally, if only a certain number of items are selected by a specific policy, it has the disadvantage of not being able to display the entire data to the user. Furthermore, although data is provided to the user, it is not used as an LLM context, which may lead to limitations in understanding additional user utterances.

[0008] For example, assuming a user has 100 monthly schedules, it may be difficult to include all schedule information within the LLM context. Consequently, only a portion is included, and even if the user scrolls to view the relevant information, it may be omitted from the context. Additionally, to accommodate all information within a limited context due to the large total volume, each domain may use specific filtering logic to include only titles or manually use only partial information for each list item within the context. Even in these cases, while the goal is to continue the conversation with detailed information about the screen the user is currently viewing, limitations may arise.

[0009] In addition, there is the inconvenience of having to define the amount of information for each domain, and even in this case, if there is too much of a data list, the aforementioned problems may occur.

[0010] An electronic device and its method of operation that extract and utilize context using an artificial intelligence model according to this document can resolve the aforementioned problems. Specifically, when all information exposed to the user is transmitted, the vast amount of data may lead to a problem where the accuracy of extracting necessary data is lowered; conversely, when only partial data is used, a lack of the information desired by the user may lead to a problem where accuracy is lowered.

[0011] An electronic device and its method of operation that extract and utilize context using an artificial intelligence model according to this document can provide a method to improve overall accuracy by applying weights to parts relatively needed by the user. In particular, when providing specific results as the context of LLM, it can provide a method to assign weights to data currently provided through the user interface (UI) and adjust the level of detail by determining the importance of the data step by step. Through this, the comprehension and response quality of LLM can be improved by selectively providing the most useful information within a limited context.

[0012] An electronic device and its method of operation that extract and utilize context using an artificial intelligence model according to this document can determine the priority of the context by utilizing the user's direct experience and interaction information, and based on this, optimize the quality of information transmitted to the LLM.

[0013] The electronic device stores instructions and includes at least one processor including memory comprising one or more storage media and processing circuitry, wherein the instructions are executed individually or collectively by the at least one processor, the electronic device receives user input, displays a user interface (UI) comprising a plurality of information items in response to the user input, checks user input (interaction) or viewing state for the plurality of information items displayed on the user interface (UI), determines the importance of each of the plurality of information items based on the checked user input or viewing state, determines the data level of context data to be provided to a Large Language Model (LLM) from each information item based on the determined importance, extracts context data from the plurality of information items based on the determined data level, generates a prompt including the extracted context data and the user input and transmits it to the LLM, and receives a response to the prompt from the LLM It can receive and control the operation to be performed based on the received response.

[0014] An electronic device and its method of operation that extract and utilize context using an artificial intelligence model according to this document can improve the accuracy of utilizing large-scale language models (LLM) for user utterance understanding and operation by proposing a method that can maximize the use of limited context size constraints.

[0015] An electronic device and its method of operation that extract and utilize context using an artificial intelligence model according to this document can provide a relatively easy application method by applying automated weights based on user interface (UI) layouts, so that developers do not have to directly specify the data to be delivered as context for each domain.

[0016] In addition, the electronic device and its method of operation that extract and utilize context using the artificial intelligence model according to this document can construct a context closely linked to the user experience by applying automated context weights based on UI records shown to the user, and can increase efficiency by directly adjusting the resolution of each piece of information in the system to match the size of the context.

[0017] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0018] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0019] FIG. 2 is a generative artificial intelligence (AI) system (200) according to one embodiment.

[0020] FIG. 3 illustrates an AI Framework (220) having on-device AI processing capabilities according to one embodiment.

[0021] FIG. 4 is a block diagram illustrating an ASR module (400) according to one embodiment.

[0022] FIG. 5 is a block diagram of a prompt generation system according to one embodiment.

[0023] FIG. 6 illustrates an interface in which an electronic device according to one embodiment finds and provides a schedule registered with a user.

[0024] FIG. 7 is a block diagram showing the components of an electronic device and an LLM according to one embodiment.

[0025] FIG. 8 is a flowchart illustrating the operation method of an electronic device that extracts and utilizes context using an artificial intelligence model according to one embodiment.

[0026] FIG. 9 is a flowchart illustrating the process of an electronic device according to one embodiment processing a user's utterance and responding using an LLM.

[0027] FIG. 10 is a flowchart illustrating a method for creating a context of an electronic device according to one embodiment.

[0028] FIG. 11a illustrates an interface and context data extraction process provided by an electronic device according to one embodiment in response to a user's utterance.

[0029] FIG. 11b illustrates the process of an electronic device according to one embodiment classifying the importance of information items according to the user's display state and generating context data of different levels.

[0030] FIG. 11c illustrates the structure of a prompt that an electronic device according to one embodiment transmits to a large-scale language model (LLM).

[0031] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0032] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0033] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0034] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0035] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0036] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0037] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0038] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0039] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0040] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0041] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0042] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0043] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0044] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0045] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0046] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0047] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0048] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0049] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0050] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0051] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0052] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0053] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0054] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0055] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as 'A or B', 'at least one of A and B', 'at least one of A or B', 'A, B or C', 'at least one of A, B and C', and 'at least one of A, B, or C' each may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as 'first', 'second', or 'first' or 'second' may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as ‘coupled’ or ‘connected’ to another (e.g., 2nd) component, with or without the terms ‘functionally’ or ‘communicationally,’ it means that the said component may be connected to the said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0056] The term 'module' as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0057] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0058] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0059] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0060] According to one embodiment, the number of processors (120) may be one or more. For example, the processor (120) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core. The processor (120) can control the operations of the electronic device (101) by executing instructions stored in memory (130). For example, the processor (120) may correspond to a plurality of processors that collectively perform a plurality of operations by dividing them among the processors.

[0061] FIG. 2 is a generative artificial intelligence (AI) system (200) according to one embodiment. Referring to FIG. 2, the generative AI system (200) may include a User Interface (210), an AI Framework (220), a Generative AI Model (230), a Knowledge Repository (240), and an Application / Service Module (250). These components may be operated on one or more of an electronic device (101), an external electronic device (102 or 104), or a server (108). For example, the User Interface (210) and the AI ​​Framework (220) may be operated on the electronic device (101), and the Knowledge Repository (240) and the Generative AI Model (230) may be operated on the server (108).

[0062] According to one embodiment, the User Interface (210) may receive user input (e.g., user query). User input may be received in the form of text, images, voice (e.g., natural language), video, menu selection, or a combination thereof. The User Interface (210) may include various context information (e.g., running application or user location) related to the generative artificial intelligence system (200) at the time the user input is received, in addition to or instead of the user input. The User Interface (210) may provide the user input or the context information to the AI ​​Framework (220) and provide the result of processing therefrom to the user, for example, through the AI ​​Framework (220). According to one embodiment, in addition to user input, the electronic device may provide context information obtained using information included on the screen to the AI ​​Framework (220). The result may be provided in the form of text, images, voice, video, an action requested by the user (e.g., launching a specified function or app), or a combination thereof.

[0063] According to one embodiment, the AI ​​Framework (220) can identify (e.g., estimate) a user intent based on at least some user input or context information received from the User Interface (210), control each of the relevant modules (e.g., 221, 223, or 225) to perform a function or action corresponding to the identified user intent, and coordinate collaboration between two or more modules. The AI ​​Framework (220) may include a Prompt Design Module (221), an API / Plug-in Management Module (223), and an Output Modification Module (225), as illustrated in FIG. 2.

[0064] According to one embodiment, the Prompt Design Module (221) can generate a prompt to be input to the Generative AI Model (230) based at least partially on user input or context information received from the User Interface (210). For example, the Prompt Design Module (221) can generate a prompt using user preferences, a prompt library, or prompt examples stored in the Knowledge Repository (240) based at least partially on user input or context information.

[0065] According to one embodiment, the API / Plug-in Management Module (223) may communicate, for example, via an API, with various resources (e.g., Knowledge Repository (240)) that provide said additional information when there is a request for said additional information in relation to user input. Additionally or generally, when a specified action (e.g., function, app, or service) is performed in response to said user input, the API / Plug-in Management Module (223) may request the Application / Service Module (250) to perform said specified action via a corresponding API. The API / Plug-in Management Module (223) may provide information obtained from the Knowledge Repository (240), the Application / Service Module 250, or another external resource to the Prompt Design Module (221). That obtained information may be used by the Prompt Design Module (221) to generate a prompt along with the user input, or provided to a generative AI model (230).

[0066] According to one embodiment, the Output Modification Module (225) can fine-tune the results obtained through the Generative AI Model (230) as at least part of the response to user input (e.g., user query). For example, the Output Modification Module (225) can determine whether the content of the response obtained through the Generative AI Model (230) is appropriate as a response to a request made by the user input. For example, the Output Modification Module (225) can determine the degree of relevance, degree of bias (e.g., political or social bias), or degree of harmfulness (e.g., sexual or profanity) of the difference between the response obtained through the Generative AI Model (230) and the user input. Additionally or generally, the Output Modification Module (225) can request that additional AI processing be performed on the obtained response, or provide the user with a hint to avoid unwanted output. For example, additional prompts can be generated through the Prompt Design Module to obtain a response again through the Generative AI Model (230).

[0067] According to one embodiment, the Generative AI Model (230) may form at least part of an artificial intelligence neural network and may include a model that generates images or a model that generates language. The image generation model may include, for example, a generative adversarial network (GAN), a variational autoencoder (VAE), or a Diffusion-based model using a VAE and a Transformer. The language generation model may include, for example, a large language model (LLM), a large multimodal model (LMM), a large vision model (LVM), or a large action model (LAM). The LAM may automatically generate actions for an environment (e.g., a robot, a car, an electronic device (101), or a program (140)). Additionally, for at least some AI models (e.g., LLM), there may be a low-rank adaptation (LoRA) adaptor that is fine-tuned for, for example, a specific task or a specific situation.

[0068] FIG. 3 illustrates an AI Framework (220) having on-device AI processing capabilities according to one embodiment. In this case, the AI ​​Framework (220) may generate and learn a response to the user input using resources within the device, instead of sending the user input received through a User Interface (210) operating on the same device (e.g., electronic device (101)) to a Generative AI Model (230) operating on an external device (e.g., server (108)), or additionally. Referring to FIG. 3, the AI ​​Framework (220) may include a Cross-Application Action Module (310), a Personal Data Managing Module (330), an On-device AI Model (350), and an Orchestration Module (370).

[0069] According to one embodiment, the Cross-Application Action Module (310) determines one or more additional applications required for the operation of an executed application (e.g., an assistant app) and may link or suggest operations between the app and at least one additional application, or between multiple additional applications. For example, the Cross-Application Actions Module (310) may execute one or more additional applications to be used to respond to a user request through the assistant app sequentially or at least partially, simultaneously. Additionally, the Cross-Application Action Module (310) may communicate with the additional applications so that the result of the execution of one additional application (e.g., content) can be shared with other additional applications.

[0070] According to one embodiment, the Personal Data Managing Module (330) may provide personal information (e.g., schedule, contact, or message information) about a user of the application (e.g., assistant app) or the additional application running on the device (e.g., electronic device 101) or other related individuals (e.g., family or friends) to another module of the AI ​​Framework (220) or a related module (e.g., Generative AI Model 230) running on another device.

[0071] According to one embodiment, the on-device AI model (350) may include at least one model among one or more AI models (e.g., GAN, VAE, LLM, LMM, LVM, or LAM) operated on an external device (e.g., server (108)) or a corresponding lightweight AI model. Additionally, for said model or said lightweight model, there may be, for example, a LoRA adaptor.

[0072] According to one embodiment, the Orchestration Module (370) may select one or more AI models to be used to obtain a response to user input (e.g., user query). For example, the Orchestration Module (370) may select one or more AI models from among an On-Device AI Model (350), an AI model operating on an external device (e.g., server (108)) (e.g., Generative AI Model (230)), or a third AI model (not shown) operating on another external device. When multiple AI models are selected, the Orchestration Module (370) may communicate with the selected models or devices so that the operation between the selected AI models and the processing of the results thereof can be coordinated between the relevant models or devices.

[0073] According to one embodiment, two or more modules of a generative AI system (200) (e.g., Cross-Application Action Module (310) and Orchestration Module (370)) may be implemented as a single module to maintain the same functionality. Various variations are possible.

[0074] FIG. 4 is a block diagram illustrating an ASR module (400) according to one embodiment.

[0075] According to one embodiment, the ASR module (400) may include a front end (410), an end-point detector (EPD) (420), a wake-up module (430), and / or an automatic speech recognition model (440). The ASR module (400) may generate text data using voice input. For example, the ASR module (400) may acquire voice input through an I / O interface and / or an external device, process the acquired voice input, and output generated text data. For example, the ASR module (400) may be a software module implemented by executing instructions by a processor. Hereinafter, operations performed by the ASR module (400) and / or its components may be referred to as operations of the processor of the device implementing the ASR module (400).

[0076] The front end (410) can perform preprocessing operations on the voice input. For example, the front end (410) can remove echoes from the voice input using an echo cancellation module (e.g., an acoustic echo canceller (AEC) and / or a residual echo suppressor (RES)). For example, the front end (410) can remove noise from the voice input using a noise removal module (e.g., a noise suppressor, NS).

[0077] The EPD (420) can detect the end point of the voice input. For example, the EPD (420) can detect the end point of the voice input and, based on the detection result, identify the end point of the user utterance corresponding to the voice input.

[0078] The wake-up module (430) can selectively transmit the voice input to an automatic speech recognition model (440) based on whether the voice input contains a specified wake word (or, wake-up signal). For example, the wake-up module (430) can identify whether a specified wake word exists in the voice input preprocessed from the front end (410) through keyword spotting.

[0079] The automatic speech recognition model (440) may include any model for obtaining text data from a speech input. For example, the automatic speech recognition model (440) may extract text data from a speech input or / or convert the speech input into text data based on a STT algorithm. For example, the STT (speech to text) algorithm may include a hidden Markov model, a Gaussian mixture model, a deep neural network model, an N-gram language model, other statistical models, and / or a combination thereof. For example, the automatic speech recognition model (440) may include one or more models. One or more models may be, for example, models for recognizing different languages. For example, the ASR module (400) can be implemented as an E2E (end-to-end) model (e.g., CTC (connectionist temporal classification), RNN-T (recurrent neural network transducer), LAS (listen, attend, and spell), or Hybrid CTC / LAS).

[0080] For example, the ASR module (400) may include other components. For example, the ASR module (400) may further include a feature extraction module (not shown) and an encoder (not shown). The ASR module (400) may extract features from a voice input (e.g., using the feature extraction module) to obtain a feature vector, and encode the feature vector using the encoder.

[0081] FIG. 5 is a block diagram of a prompt generation system according to one embodiment.

[0082] According to one embodiment, the prompt generation system may include a voice assistant (590), a prompt manager (500), and an AI model (550).

[0083] According to one embodiment, a voice assistant (590) may include an AI (artificial intelligence)-based hardware and / or software module that understands the content of voice input by a user and processes actions according to the user's request. For example, the voice assistant (590) may use automatic speck recognition (ASR) technology to analyze a voice signal and convert it into text, interpret the content of the converted text, and process various actions such as executing and controlling an application requested by the user, controlling device settings, and searching for and providing information based on the interpretation of the text.

[0084] According to one embodiment, user input to a voice assistant (590) may include a query requesting the execution and response of a task to an AI model (550), and the voice assistant (590) may transmit the user input containing the query to the AI ​​model (550) to obtain a response to the query from the AI ​​model (550) and provide it to the user.

[0085] According to one embodiment, the AI ​​model (550) may include a generative artificial intelligence model (550) or a large language model (LLM), which is an artificial intelligence model that learns various data to generate new information and sentences. For example, the AI ​​model (550) may include an on-device AI model implemented on an electronic device (e.g., the electronic device (101) of FIG. 1) and a server AI model implemented by an external server, and the server AI model may be implemented and operated on the manufacturer's server of the electronic device or implemented and operated by a third party. The electronic device may select an AI model (550) set by the user among the available AI models (550) to transmit user input, and / or select an AI model (550) to request a response based on the type or content of the query of the user input to transmit user input.

[0086] According to one embodiment, a prompt manager (500) can perform various operations to generate a prompt (or a secondary prompt) from a user input (or a primary prompt). For example, the prompt manager (500) receives user input including a query to an AI model (550), validates the user input based on the number of parameters included in the user input, and if the validation is not passed, generates a prompt based on a prompt template and pre-stored user data. The prompt generated by the prompt manager (500) includes various information compared to the initial user input, and since the included information is based on user data, when input to the AI ​​model (550), the AI ​​model (550) can provide more accurate and detailed information desired by the user as a response.

[0087] According to one embodiment, the prompt manager (500) may be implemented on an electronic device to generate a prompt in response to user input and then provide it to an AI model (550), or implemented on a server-type AI model to generate a prompt in response to user input entered into the AI ​​model (550), and / or implemented on a separate server device to receive user input through a network, generate a prompt, and then transmit it to the AI ​​model (550).

[0088] In the following description, a prompt manager (500) that evaluates user input and generates a new prompt is described as being implemented on an electronic device that acquires user input, but the various embodiments of this document are not limited thereto, and the embodiments described below may be provided by various devices such as a server device rather than an electronic device.

[0089] FIG. 6 illustrates an interface in which an electronic device according to one embodiment finds and provides a schedule registered with a user.

[0090] According to one embodiment, the electronic device (101) may receive a voice command from a user, such as "Tell me what the schedule is for this month," and in response, may display a plurality of interfaces with a response such as "There are 20 schedules for this month," as illustrated in FIG. 6. In FIG. 6, the electronic device (101) may display a plurality of interfaces, including a first interface (610) and a second interface (620), on a display. The electronic device (101) may extract schedule information related to the user from a database storing the user's schedule information and visually represent it, and each interface may include information regarding the title, date, or time of the schedule. The types and number of interfaces are merely examples and are not limited thereto, and may vary depending on the settings.

[0091] According to one embodiment, the first interface (610) may display the user's first schedule, 'New Year's Day,' and indicate that the schedule is set to 'January 1 (Mon), all day.' The electronic device (101) may display different colors or icons on the first interface (610) depending on the importance or attributes of the schedule. The second interface (620) may display the user's second schedule, 'attendance at the New Year's ceremony,' and indicate that the schedule is scheduled for 'January 2 (Tue), 8:40–9:40.' The electronic device (101) may also display icons indicating the attributes of the schedule on the second interface (620) so that the user can intuitively grasp the characteristics of the schedule.

[0092] According to one embodiment, the electronic device (101) can convert the information, along with the interface display, into a context for transmitting to a large-scale language model (LLM). The electronic device (101) may not simply include all schedule information in the context, but may give higher weight to the information regarding the interfaces (610, 620) currently being displayed to the user. For example, the information of the first interface (610) and the second interface (620) that the user is currently viewing via scrolling may be included in detail in the context, while the remaining schedules not displayed on the screen may only briefly include the title and date.

[0093] According to one embodiment, when a user makes a series of utterances such as "delete the first schedule" or "tell me only the Tuesday schedule" while looking at this screen, the electronic device (101) can accurately determine the user's intention by utilizing previously displayed interface information and the screen information currently being viewed by the user together. In a situation where the size of the context transmitted to the LLM is limited, the electronic device (101) can increase the efficiency of context utilization by prioritizing the inclusion of information regarding the interface area where the user has focus.

[0094] According to one embodiment, the electronic device (101) may apply various criteria to determine the importance of each interface element. For example, the importance may be determined by analyzing visual characteristics including the size, boldness, color, or position of text displayed on the interface. The electronic device (101) may also reflect in the determination of importance an interaction pattern in which a user touches a specific interface area or looks at it for more than a specified amount of time. In FIG. 6, the electronic device (101) displays schedule titles such as "First Day of the New Year" and "Attendance at the Opening Ceremony" in a relatively larger font than other information, so it may determine this as more important information and include it preferentially in the context.

[0095] Although FIG. 6 describes the process based on two interfaces, the number of interfaces that the electronic device (101) can display is not limited to this. In fact, if there are 20 or more monthly schedules for the user, the electronic device (101) can display all schedules in a scrollable form, and the context can be configured by applying different levels of importance to schedules that are displayed directly on the screen and schedules that can only be viewed by scrolling. Through this, the electronic device (101) can improve the response quality of the LLM by prioritizing the inclusion of information that is highly relevant to the user's current interests, even within a limited context size.

[0096] FIG. 7 is a block diagram showing the components of an electronic device and an LLM according to one embodiment.

[0097] According to one embodiment, the electronic device (101) can interact with a client (710), a processor (120), and an LLM service (740) and perform context-based voice assistant functions.

[0098] According to one embodiment, the client (710) may serve as an interface for a user to input commands into the electronic device (101). The user may use the client (710) to transmit an utterance, such as "When is your busiest day?", to the electronic device (101). The user's utterance may be transmitted to the processor (120) of the electronic device (101).

[0099] According to one embodiment, a processor (120) inside an electronic device (101) may perform the functions of several modules for processing utterances and generating appropriate responses. Inside the processor (120), functional modules such as a context collector (720), an optimizer (722), a function execution (724), and a view generator (726) may be included. These modules may be physically separated hardware components or may be implemented as software modules executed on the processor (120). The types of modules are merely examples and are not limited thereto.

[0100] According to one embodiment, the context collector (720) collects natural language generation (NLG), views, and user utterances that were generated based on existing conversation history related to the user's utterance, and can also collect previously generated context information. The optimizer (722) inside the context collector (720) can extract previous results, analyze the results of view execution, and evaluate the information of each element. The optimizer (722) can calculate the weight of each element according to a priority policy defined in the database and optimize the context by selecting elements to include in the context. In this process, the optimizer (722) can determine importance by analyzing the visual characteristics of UI elements (e.g., font size, boldness) and the user's interaction pattern.

[0101] According to one embodiment, a function execution (724) can execute a function that matches the user's intent and, depending on the result of the execution, generate a view element of an interface to be provided to the user through a view generator (726). The view generator (726) can play a role in generating a response and a view (e.g., a layout) to be provided to the user based on the result of the executed function. The generated view can not only be provided to the user but can also be stored by the context collector (720) as data for future context configuration.

[0102] According to one embodiment, the processor (120) may receive an optimized context from a context collector (720). The processor (120) may request processing by transmitting the collected optimized context and user utterance to an LLM service (740). The processor (120) may request the execution of an LLM transformation. The processor (120) may transmit the user utterance and the optimized context (e.g., optimized last results) together to the LLM service (740).

[0103] According to one embodiment, the LLM service (740) may be performed inside the electronic device (101) or on an external server. The LLM service (740) may include a prompt generator (742) and an LLM model (744). The prompt generator (742) may generate a prompt that the LLM model (744) can understand by combining a received utterance and an optimized context. In the prompt generation process, the prompt generator (742) may form a structured prompt that includes at least one of the agent's role, a response generation method, a previous conversation history, or an optimized context. The LLM model (744) may receive the generated prompt as input, perform inference, and generate an appropriate response to a user question.

[0104] According to one embodiment, the processor (120) may receive a response generated by the LLM service (740) (e.g., 'The busiest day is January 10') in text or voice form. The electronic device (101) may process the response received from the LLM service (740) and deliver it to the client (710).

[0105] Through this structure, the electronic device (101) can determine which information among the result values ​​to include in the context based on UI elements that the user directly experiences and sees, and can adjust the level of detail of the data to be included in the context by evaluating the importance of each piece of information in stages. In addition, the electronic device (101) can configure the final context to be used according to the determined important information and the level of detail, and generate an optimal response within a limited context size by utilizing this for interpreting user utterances.

[0106] FIG. 8 is a flowchart illustrating the operation method of an electronic device that extracts and utilizes context using an artificial intelligence model according to one embodiment.

[0107] The operations described through FIG. 8 may be implemented based on instructions that can be stored in a computer recording medium or memory (e.g., memory (130) of FIG. 1). The illustrated method (800) may be executed by an electronic device (e.g., electronic device (101) of FIG. 1) described above through FIG. 1 to 7, and the technical features described above will be omitted below. The order of each operation in FIG. 8 may be changed, some operations may be omitted, and some operations may be performed simultaneously.

[0108] According to one embodiment, in operation 810, the electronic device (101) may display a user interface (UI) containing a plurality of information items in response to user input under the control of a processor (e.g., processor (120) of FIG. 1). The user input may include voice utterances such as, for example, 'Tell me this month's schedule', 'Show me nearby restaurants', or 'Show me unread messages'.

[0109] According to one embodiment, the electronic device (101) can identify the intent of the utterance, search for related information, and visually represent it. Multiple information items may include a large amount of information that is difficult to display on a single screen, such as schedule information, a list of restaurants, and message content. This information may be displayed in a user interface in the form of a scrollable list. The electronic device (101) may configure the user interface by utilizing various visual elements such as text, images, and icons according to the characteristics of each information item, and may visually distinguish the importance of the information through at least one of the elements of font size, boldness, or color.

[0110] According to one embodiment, in operation 820, the electronic device (101) can check user input (interaction) or viewing state for a plurality of information items displayed on a user interface (UI) and determine importance. User input may include all actions by which a user interacts with the interface, such as touch, scroll, or click, and the viewing state may refer to a state such as whether a specific information item is currently visible on the screen, whether it was previously visible but is not currently visible, or whether it has not yet been shown to the user.

[0111] According to one embodiment, the electronic device (101) may continuously monitor these states and classify the viewing status of each information item into 'what the user is currently viewing,' 'what the user has passed by,' and 'what the user has not yet viewed.' The classification status is merely an example and is not limited thereto, and may vary depending on the settings. In the process of determining importance, the electronic device (101) may assign the highest priority to the information item currently being viewed by the user, a medium priority to items already viewed, and a low priority to items not yet viewed. Additionally, the electronic device (101) may calculate importance by analyzing the visual characteristics (e.g., font size, boldness, color, position) and frequency of occurrence of the information item. Depending on the characteristics of the information item, the electronic device (101) may set the weight relatively low for forms such as URLs, because the internal characters are not used as a direct context.

[0112] According to one embodiment, in operation 830, the electronic device (101) can determine the data level of the context data based on the determined importance and extract the context data. The data level refers to the degree of detail of the data to be included in the context for each information item, and the electronic device (101) can apply the data level differently according to the importance. For example, for information items with high priority, all property elements (e.g., restaurant name, representative photo, phone number, price, menu, reviews) can be stored as context, for items with middle priority, only major property elements (e.g., restaurant name, phone number, menu, price) can be stored as context, and for items with low priority, only minimal representative property elements (e.g., restaurant name, phone number) can be stored as context. Such policies can be stored in an optimization policy database and can be dynamically changed according to the context size.

[0113] According to one embodiment, the electronic device (101) may apply priority to a defined importance policy for a specific function, for example, if the font size is displayed small according to a UI configuration policy but is an essential item when conveying to the LLM, the weight or priority of that item may be increased.

[0114] According to one embodiment, in operation 840, the electronic device (101) may generate a prompt including extracted context data and user input, transmit it to the LLM, and receive a response. The prompt may include the agent role as a voice assistant and a method for generating a response, a previous conversation history, optimized context data, and the user's current command.

[0115] According to one embodiment, an electronic device (101) transmits a generated prompt to a large-scale language model (LLM) service, and the LLM service converts the information received through a prompt generator into a form that the LLM model can understand, and then performs inference through the LLM model to generate a response. The response thus generated is transmitted to the electronic device (101) in the form of text or voice, and the electronic device (101) can provide it to the user. Through this process, the electronic device (101) can improve the response quality of the LLM by prioritizing the inclusion of information highly relevant to the user's current interests within a limited context size, and can provide a natural conversational experience by configuring an optimal context closely linked to the user experience among a vast amount of information.

[0116] According to one embodiment, the electronic device (101) classifies the display state into either a state currently viewed by the user or a state not yet viewed by the user, and determines that the information item in the state currently viewed by the user includes all attribute information of the information item, and determines that the information item in the state not yet viewed by the user includes only basic attribute information of the information item. Basic attribute information may mean minimal information for identifying the information item.

[0117] According to one embodiment, an electronic device (101) can monitor and classify the viewing state of a plurality of information items displayed on a user interface (UI). This viewing state may be state information indicating whether the user is currently viewing the information item, has passed by it, or has not yet viewed it. The electronic device (101) can classify this viewing state into either a 'currently viewing state' or a 'not yet viewed state'. This classification may be performed by distinguishing between an area currently displayed on the screen and within the user's field of vision and an area that can only be viewed through scrolling or page movement.

[0118] According to one embodiment, the electronic device (101) may differentiate and apply the level of detail of information to be included in the context according to the display state of the information item. Specifically, for the information item in the state currently being viewed by the user, it may be determined to include all property information of the information item in the context. All property information may refer to all data fields possessed by the information item. For example, in the case of restaurant information, all attributes such as the business name, representative photo, phone number, price, menu, and reviews may be included. For example, in the case of schedule information, all detailed information such as the schedule title, date, time, location, attendees, and related memos may be included. The types of all property information are merely examples and are not limited thereto, and may vary depending on the settings.

[0119] According to one embodiment, the electronic device (101) may determine to include only basic property information of an information item in the context for information items that the user has not yet seen. Basic property information may refer to minimal information for identifying the information item, and this may vary depending on the type of information. For example, in restaurant information, only the business name and phone number may be classified as basic property information, and in schedule information, only the schedule title and date may be classified as basic property information. The types of basic property information are merely examples and are not limited thereto, and may vary depending on the settings.

[0120] According to one embodiment, the electronic device (101) can provide sufficient detailed information regarding the information currently of interest to the user even in an environment with context size limitations through such differentiation of information levels (data levels). The electronic device (101) can efficiently utilize the limited context space by including only the minimum necessary information regarding other information. The electronic device (101) can increase consistency between the information the user is visually viewing and the context information provided to the LLM, and can support the user in naturally asking follow-up questions about the content they are viewing on the screen.

[0121] According to one embodiment, the electronic device (101) can re-evaluate importance and update context configuration in real time whenever the display state changes due to a user's scroll action or page movement. To this end, the electronic device (101) can continuously monitor status information, such as items the user has viewed and items the user is currently viewing, through user interface (UI) control of the client application, and transmit this information to a voice assistant platform. In this way, the electronic device (101) can improve conversation quality by configuring an optimal context tailored to the user's current interests.

[0122] According to one embodiment, the electronic device (101) classifies the display state into one of the state currently being viewed by the user, the state that the user has passed by, and the state that the user has not yet seen, and determines the importance as one of the first importance, the second importance, and the third importance based on the classified display state, and assigns the first importance to the information item in the state currently being viewed by the user, assigns the second importance to the information item in the state that the user has passed by, and assigns the third importance to the information item in the state that the user has not yet seen.

[0123] According to one embodiment, the electronic device (101) may determine to include all attribute information of the information item of the first importance, determine to include intermediate level attribute information of the information item of the second importance, and determine to include only basic attribute information of the information item of the third importance. Intermediate level attribute information includes major attributes related to identification, content, and characteristics as part of all attribute information, and basic attribute information may mean minimal information for identifying the information item.

[0124] According to one embodiment, the electronic device (101) can further subdivide and classify the viewing state of information items displayed on a user interface (UI). The electronic device (101) can classify the viewing state of each information item into one of the following: 'currently viewing state', 'already viewed state', and 'not yet viewed state'. Here, the 'currently viewing state' may refer to an information item currently displayed on the screen and within the user's field of vision; the 'already viewed state' may refer to an information item that was previously displayed on the screen but has now disappeared from the screen due to scrolling or page movement; and the 'not yet viewed state' may refer to an information item that has not yet been displayed on the screen.

[0125] According to one embodiment, the electronic device (101) can determine the importance of each information item based on these classified display states. Specifically, the electronic device (101) can determine the importance of the information item as one of a first importance, a second importance, and a third importance. The types and number of importances are merely examples and are not limited thereto, and may vary depending on the settings.

[0126] According to one embodiment, the electronic device (101) may assign a first importance, which is the highest priority, to information items in the 'state the user is currently viewing.' This may be because the information currently being visually encountered by the user is most likely to be closely related to the user's interests. Additionally, the electronic device (101) may assign a second importance, which is an intermediate level of priority, to information items in the 'state the user has passed by.' This may be based on the assumption that the information is within the user's range of perception to some extent, as it has already been exposed to the user. Finally, the electronic device (101) may assign a third importance, which is the lowest priority, to information items in the 'state the user has not yet seen.'

[0127] According to one embodiment, the electronic device (101) may differentiate and apply the level of detail of information to be included in the context according to the importance determined in this way. For an information item of the first importance, it may be determined to include all property information of the information item in the context. All property information includes all data fields and details possessed by the information item, and for example, in the case of restaurant information, it may refer to all attributes such as the business name, representative photo, phone number, address, business hours, price range, menu, reviews, and rating. Through this, the LLM can be supported in generating an accurate and detailed response by utilizing as much context as possible for the information currently being viewed by the user.

[0128] According to one embodiment, for an information item of second importance, the electronic device (101) may determine to include intermediate-level property information of the information item. Intermediate-level property information may include key attributes related to identification, content, and characteristics as part of all property information. For example, in restaurant information, information such as the business name, phone number, main menu, and price range may correspond to intermediate-level property information, and in schedule information, key information such as the schedule title, date, time, and location may be included. Such intermediate-level property information may provide a level of context that allows for basic understanding and reference regarding information that the user has already viewed, but does not include all details.

[0129] According to one embodiment, for information items of third importance, the electronic device (101) may determine to include only basic property information of the information item. Basic property information may refer to minimal information for identifying the information item; for example, in restaurant information, only the business name and phone number, and in schedule information, only the schedule title and date may be considered as basic property information. By including only such minimal information, the electronic device (101) can include the existence of information items in the context even for information items that the user has not yet seen, while efficiently utilizing the limited context space.

[0130] According to one embodiment, the electronic device (101) can configure an optimal context tailored to the user's current experience and interests within a limited context size through the application of such granular importance classification and differentiated data levels. The electronic device (101) can intelligently manage the context by reflecting the user's interface usage pattern and information consumption flow, rather than simply processing all information at the same level.

[0131] FIG. 9 is a flowchart illustrating the process of an electronic device according to one embodiment processing a user's utterance and responding using an LLM.

[0132] According to one embodiment, in operation 910, an electronic device (e.g., the electronic device (101) of FIG. 1) may receive utterances from a user. Here, utterances refer to commands or questions conveyed by the user in voice, and may include inputs in the form of natural language, such as, for example, 'Tell me the schedule for this month,' 'Delete the first schedule,' or 'When is the busiest day.' The electronic device (101) may receive these utterances through a voice input device such as a microphone and convert them into text using speech recognition technology. The user's utterances may be independent questions, continuous questions that continue the context of a previous conversation, or content that refers to information displayed on a screen.

[0133] According to one embodiment, in operation 920, the electronic device (101) can extract elements including transmitted utterances, previous conversation results, and UI focused items. The electronic device (101) can collect various information elements related to the current conversation by utilizing a context collector module. Transmitted utterances refer to questions or commands just entered by the user, and previous conversation results may include the response or execution result of the electronic device (101) to the previous utterance. UI focused items may refer to elements currently receiving the user's attention in the user interface. UI focused items may refer to the area currently viewed by the user among the information displayed on the screen or elements selected by the user. The electronic device (101) can determine the UI focus state using at least one of the user's scroll position, touch, click position, or eye tracking method.

[0134] According to one embodiment, in operation 930, the electronic device (101) can generate a context by optimizing each element. Optimization may mean the process of selecting the most useful information within a limited context size and adjusting the amount of such information. The electronic device (101) can evaluate the importance of the previously extracted elements through a context optimizer module and determine the level of detail of the information accordingly. For example, a first importance may be assigned to UI-focused items to include all property information, a second importance may be assigned to items viewed by the user to include only intermediate-level property information, and a third importance may be assigned to items not yet viewed to include only basic property information.

[0135] According to one embodiment, the electronic device (101) can calculate a weight and construct a final context by considering the visual characteristics of an information item (e.g., font size, boldness, color) and a priority policy defined in an optimization policy database. The types of visual characteristics are merely examples and are not limited thereto.

[0136] According to one embodiment, in operation 940, the electronic device (101) can generate a prompt using user utterance and context and transmit it to the LLM. The prompt is input text required for the LLM to generate an appropriate response, and the electronic device (101) can configure it through a Prompt Generator module. The generated prompt may include elements such as the agent's role definition, conversation rules, previous conversation history, optimized context, and current user utterance. The context may include, for example, content regarding the agent's role definition, such as, 'You are a voice assistant. Interpret the user's request and provide an answer so that the most appropriate response can be given. If necessary, interpret by referring to the conversation history or the last execution result.' The electronic device (101) can configure the prompt in a structure that includes the previous conversation history, optimized context data, and the current user's command. The electronic device (101) can transmit the completed prompt to the LLM service. The LLM service may be executed internally within the electronic device (101) or provided through an external server.

[0137] According to one embodiment, in operation 950, the electronic device (101) may receive a response from the LLM, execute a command, and transmit the execution result to a user terminal. The LLM may analyze the received prompt to identify the user's intent and generate an appropriate response or a command to be executed. If the response generated by the LLM is in the form of text or is a command to execute a specific function, the electronic device (101) may interpret it and execute the corresponding function. For example, if the LLM generates a response saying "Delete the first schedule," the electronic device (101) may interpret this as a command to execute the schedule deletion function and perform the corresponding function through a Function Execution module. After executing the function, the electronic device (101) may visualize the execution result through a View Generator module and display it on a user interface, or use Text-to-Speech technology to announce the result as voice. The electronic device (101) may transmit the processed result to a user terminal. The electronic device (101) can store the processed result as part of the context for the next conversation.

[0138] According to one embodiment, the electronic device (101) can provide accurate and highly relevant responses by utilizing optimal information within a limited context size while maintaining a natural conversational flow with the user. The electronic device (101) can enhance the user experience of the conversational interface by reflecting the user's current interests and UI interaction patterns in the context generation.

[0139] According to one embodiment, the electronic device (101) determines the importance based on at least one of the font size, font thickness, or display position of each information item, and can assign a relatively higher importance as the font size increases, the font thickness increases, or the display position is located at the top.

[0140] According to one embodiment, the electronic device (101) can determine the importance based on the visual characteristics of each information item. Specifically, the electronic device (101) can calculate the importance by analyzing at least one element among the font size, font boldness, or display position of each information item displayed on a user interface (UI).

[0141] According to one embodiment, the electronic device (101) can assign a relatively higher importance as the font size increases. For example, in restaurant information displayed in a list format, if the business name is displayed in a size of 24 points, the address in 16 points, and the phone number in 12 points, the electronic device (101) can assign the highest importance to the business name, medium importance to the address, and low importance to the phone number. This allows visually emphasized information to be reflected more importantly in the context.

[0142] According to one embodiment, the electronic device (101) may also assign a relatively higher importance as the font thickness increases. Font thickness can generally be divided into levels such as regular, semi-bold, bold, and extra-bold. The electronic device (101) may apply weights differentially according to these thickness levels. For example, if important events in a schedule app are displayed in bold text, the electronic device (101) may recognize this visual cue and assign a higher importance to the corresponding event information. The levels of font thickness are merely examples and are not limited thereto, and may vary depending on the settings.

[0143] According to one embodiment, the electronic device (101) may assign a relatively higher importance to a display position as it is located at the top. This is based on the user experience (UX) principle that users generally read from the top when viewing a screen, and thus can determine that information placed at the top is more important or has a higher priority. For example, since news placed at the top of a list of news articles is generally more likely to be more important, the electronic device (101) can utilize this location information to determine importance.

[0144] According to one embodiment, the electronic device (101) may evaluate these visual characteristics individually, or determine the importance by comprehensively considering several characteristics.

[0145] According to one embodiment, the electronic device (101) checks for a predefined weight associated with an information item or a function that generated the information item, and if the predefined weight exists, controls the application of the predefined weight preferentially when determining importance.

[0146] According to one embodiment, the electronic device (101) can identify a predefined weight associated with the information item itself or the function that generated the information item. The predefined weight may refer to an importance value that is pre-set by a developer or user for a specific information item or function. Such weights may be stored as information weights in a view resource or defined in the form of custom priority policies.

[0147] According to one embodiment, the electronic device (101) can check whether there is a predefined weight for an information item. The electronic device (101) can check whether an information weight or priority policy is defined by examining the view resource of the function when a specific function is executed. For example, if it is defined to assign a particularly high weight to a schedule marked as 'important' in a schedule management function, the electronic device (101) can check this defined weight information.

[0148] According to one embodiment, the electronic device (101) can be controlled to apply the predefined weight preferentially when determining importance if the predefined weight exists. This is because the explicitly defined weight can reflect more accurate importance than the importance judgment based on visual characteristics (e.g., font size, boldness, position). For example, the electronic device (101) can compensate for the fact that, even if specific information regarding the UI design is displayed in a small font, the information is important in the context configuration through a predefined high weight.

[0149] According to one embodiment, the electronic device (101) can reflect the importance of information that is difficult to grasp based on visual characteristics alone. For example, in a specific domain, date information may be a very important element in the context even if it is displayed in small font. In this case, the electronic device (101) can increase the importance of the information through a predefined weight. The electronic device (101) can perform more sophisticated context optimization by combining automatic importance judgment based on visual characteristics with explicitly defined weights.

[0150] According to one embodiment, the electronic device (101) can apply different priority policies for each function to configure a context suitable for domain characteristics. For example, menu and price information may be important in a restaurant search function, and date and attendee information may be important in a schedule management function. The electronic device (101) can reflect domain characteristics in context creation through weights defined for each function.

[0151] According to one embodiment, the electronic device (101) can check the context size limit for the LLM and control the amount of information to be dynamically adjusted based on the context size limit. The display state may be determined based on the screen visibility of the information items at the time of user input or the scroll state of the user interface. If a plurality of information items include images, the importance may be determined based on at least one of whether text is included in the image, the video playback time, or the user's video viewing history.

[0152] According to one embodiment, the electronic device (101) can check the context size limit for the LLM and control the dynamic adjustment of the data level based on this limit. The context size limit may refer to the maximum number of tokens or text length that the LLM can process at once. The context size limit may vary depending on the architecture and model scale of the LLM. The electronic device (101) can check the model information of the LLM in use to identify the context window size of the corresponding model and determine the total amount of context to be transmitted based on this.

[0153] According to one embodiment, when the context size approaches a limit, the electronic device (101) can differentially adjust the amount of information according to the importance of the information items. For example, the electronic device (101) may initially assign all attribute information to a first priority item, middle priority item, and basic attribute information to a second priority item. Subsequently, when the context size approaches a limit, the electronic device (101) may maintain the first priority item as is and reduce the second priority item to include only basic attribute information, or exclude the third priority item entirely.

[0154] The electronic device (101) can differentially adjust the amount of information according to rules stored in an optimization policy database. The rules may include a step-by-step method of adjusting the amount of information according to the context size. The electronic device (101) can measure the current context size in real time during the context creation process and perform adjustments before reaching the maximum allowable size.

[0155] According to one embodiment, an electronic device (101) can determine a viewing state based on the visibility of information items on the screen at the time of user input or the scroll state of a user interface (UI). The electronic device (101) can collect information such as the current screen scroll position, page number, and viewport size through a client application. Based on the collected information, the electronic device (101) can determine whether an information item is fully displayed on the current screen, partially displayed, or not displayed at all.

[0156] According to one embodiment, the scroll state may refer to the extent to which a user moves the screen up and down or left and right using a scrollbar. The electronic device (101) can track which parts the user has viewed and which parts they have not yet viewed through the scroll state. For example, if information is displayed in a list form, the electronic device (101) may determine that the items at the top are in a 'state the user has viewed and passed by' if the user has scrolled down from top to bottom. The electronic device (101) may define the items currently visible on the screen as the 'state the user is currently viewing' and classify the items at the bottom that have not yet been scrolled down as the 'state the user has not yet viewed'.

[0157] According to one embodiment, on-screen visibility at the time of user input may mean what information item was being displayed on the screen at the moment the user makes a voice command or text input. The electronic device (101) can capture state information of the screen at the time of the user's utterance and use it to determine the display state.

[0158] According to one embodiment, when a plurality of information items include a video, the importance may be determined based on at least one of whether text is included in the video, the video playback time, or the user's video viewing history. The electronic device (101) can effectively reflect information in the context in the form of a video by applying a differentiated importance judgment criterion to the video item as well.

[0159] According to one embodiment, whether text is included in an image may mean whether the image contains text subtitles or important messages. Since an image containing text is generally more likely to contain more information, the electronic device (101) may assign relatively higher importance to such an image.

[0160] According to one embodiment, the video playback time may refer to the length of the video or the time the user has watched the video. The electronic device (101) may assign a relatively higher importance to the video as the video is longer or as the user watches it for a longer period. The electronic device (101) may determine that the video watched by the user for a longer period is likely to contain more important or interesting content. For example, among several restaurant introduction videos, information related to the video watched by the user for the longest period may be addressed in more detail within the context.

[0161] According to one embodiment, a user's video viewing history may refer to a record of how the user has watched a specific video. The video viewing history may refer to operation patterns including playback, pause, rewind, and fast-forward. The electronic device (101) may assign higher importance to a video that the user has repeatedly watched a specific part of or frequently paused. Such viewing behavior may be interpreted as a signal indicating that the user has a special interest in the content.

[0162] According to one embodiment, the electronic device (101) can determine the importance of a video item by comprehensively considering these various factors, and can adjust the level of inclusion of a description or metadata of the video into the context based on the determined importance. For example, the electronic device (101) can include information regarding the title, content summary, and main scene description into the context for a video judged to have relatively high importance, and only include the title and brief topic for a video of low importance.

[0163] FIG. 10 is a flowchart illustrating a method for creating a context of an electronic device according to one embodiment.

[0164] According to one embodiment, in operation 1010, an electronic device (e.g., the electronic device (101) of FIG. 1) may receive a speech execution request from a user. A speech execution request may mean a command or question conveyed by the user in voice, which may be an input expressed in the form of natural language. For example, an initial utterance such as "Tell me this month's schedule" or a subsequent utterance such as "Delete the first schedule" may be included. The electronic device (101) may convert the user's voice utterance into text using speech recognition technology. The speech execution request may be received directly through a microphone within the terminal or may be transmitted through a connected external device (e.g., a Bluetooth headset, a smart speaker).

[0165] According to one embodiment, in operation 1020, the electronic device (101) can retrieve a previous execution record. The previous execution record may include a history of past conversations with the user, results of previously executed commands, or information on previously displayed view components. The electronic device (101) can retrieve and collect previous conversation records, Natural Language Generation (NLG), and view results stored in the dialog state manager through a context collector module.

[0166] According to one embodiment, the electronic device (101) may apply criteria including temporal proximity, topic relevance, and reference pattern to select previous records that are highly relevant to the current utterance. For example, when the utterance "delete the first schedule" is input, the command executed immediately before and its result may be determined as a previous record with high relevance.

[0167] According to one embodiment, in operation 1030, the electronic device (101) may determine priority based on the state of a user UI component received for context creation. A UI component may refer to individual elements that constitute a user interface. A UI component may include, for example, the form of a button, a list, a card, or a text field. The types of UI components are merely examples and may vary depending on the settings. The electronic device (101) may receive and analyze state information of the current UI component from a client application. The state information may include the viewing state of each UI component, user interaction information, or visual characteristics (e.g., font size, boldness, color).

[0168] According to one embodiment, the electronic device (101) can classify the view state of each UI component through a view component evaluator into one of the following: 'a state currently being viewed by the user', 'a state that the user has passed by', or 'a state that the user has not yet seen'. Based on this classification, the electronic device (101) can assign a first importance, a second importance, and a third importance to each UI component. Additionally, the electronic device (101) can calculate a weight by analyzing the visual characteristics of the UI component. For example, the electronic device (101) may assign higher importance as the font size or boldness increases, and in the case of video, may assign high importance to whether text is included in the video or items that are displayed for a long time. The factors determining the weight are merely examples and are not limited to this, and may vary depending on the settings.

[0169] According to one embodiment, the electronic device (101) may prioritize the use of a view resource of a function executed as needed if there are separately defined weights and priority policies for the view resource. For example, if a specific function displays a small font size according to a UI configuration policy but is an important item when transmitted to the LLM, the weight or priority of that item may be set high. Such policies may be stored and managed in an optimization policy database.

[0170] According to one embodiment, in operation 1040, the electronic device (101) can store information required for the context based on the priority of user UI components and configure the context. The electronic device (101) can differentiate and apply the level of detail of information to be included in the context according to the importance of each UI component through a context optimizer. For UI components assigned a first importance (or high priority), the electronic device (101) can include all property information of the corresponding information item in the context. For example, in the case of restaurant information, data fields such as business name, representative photo, phone number, address, business hours, price range, menu, reviews, and ratings may be included. Restaurant information is merely an example and may vary depending on the settings.

[0171] According to one embodiment, the electronic device (101) may include only intermediate-level property information in the context for a UI component assigned a second importance (or intermediate priority). Intermediate-level property information is part of all property information, and in the case of restaurant information, key information such as business name, phone number, main menu, and price range may correspond to this.

[0172] According to one embodiment, the electronic device (101) may include only basic property information in the context for a UI component assigned a third importance (or low priority), which may mean minimal information (e.g., restaurant name, phone number) for identifying the information item.

[0173] According to one embodiment, the electronic device (101) can optimize the overall context quality by configuring the context by applying such differentiated data levels, thereby prioritizing the inclusion of information highly relevant to the user's current interests within a limited context size and simplifying the remaining information. The generated context can be structured in the form of a prompt and transmitted to an LLM. Based on the received prompt, the LLM can accurately understand the intent of the user's utterance and generate an appropriate response.

[0174] According to one embodiment, the electronic device (101) can dynamically adjust the importance criteria or readjust the amount of information according to each importance level when the context size exceeds a specific limit.

[0175] According to one embodiment, the electronic device (101) can interpret a response received from the LLM to generate voice response and visual information to be provided to the user, and store the generated voice response and visual information as context data to be used as context for subsequent user input.

[0176] According to one embodiment, an electronic device (101) can interpret a response received from an LLM and generate a voice response and visual information to be provided to a user. In the process of interpreting the LLM response, the electronic device (101) can convert the text generated by the LLM into an appropriate form by utilizing a natural language generation (NLG) module and a view handler module. The voice response refers to text generated by the LLM that has been converted into a form similar to a human voice through Text-to-Speech (TTS) technology, and the visual information may refer to screen elements composed of various forms of UI components such as text, images, lists, and cards.

[0177] According to one embodiment, the electronic device (101) can adjust the weight of voice responses and visual information according to the user's utterance intent. For example, for a simple question such as "What is the weather like today?", weather information can be displayed in the form of a card along with a concise voice response, and for a request such as "Recommend a good restaurant nearby," a list of restaurants can be visually provided along with detailed information. The electronic device (101) can store the generated voice responses and visual information as context data and control them to be used as context for subsequent user input. The electronic device (101) can store the response content provided to the user in memory (e.g., memory (130) of FIG. 1) so that it can be referenced in future conversations. The response information stored in this way can play an important role in helping the electronic device (101) understand what "the first one" refers to when the user asks a follow-up question such as "Tell me more about the first one among them." In addition, information regarding the layout and components of visual information is also stored, allowing users to accurately identify specific items displayed on the screen when referring to them.

[0178] FIG. 11a illustrates an interface and context data extraction process provided by an electronic device according to one embodiment in response to a user's utterance.

[0179] According to one embodiment, an electronic device (101) may receive a first utterance (1101) from a user saying, "Show me nearby restaurants." This utterance may be converted into text and processed through speech recognition technology, and the electronic device (101) may analyze this utterance to determine that the user is requesting information about nearby restaurants. Based on the result of the utterance analysis, the electronic device (101) may search for information about restaurants within a specified distance relative to the current location and create an interface to visually display the searched information.

[0180] According to one embodiment, the electronic device (101) may display search results on a first interface (1110) along with a guidance message stating, "32 restaurants found nearby." The first interface (1110) contains information on various restaurants in the form of cards, and among these, restaurant information such as "A Sundaeguk" and "B Chinese Restaurant" currently visible on the screen may be displayed on a second interface (1112). Each restaurant card may include information on the restaurant name, rating (4.6, 4.8), price range (10,000~20,000 won, 7,000~30,000 won), operating status (OPEN), phone number (031-111-1111, 031-222-2222), and delivery availability. The types of information that may be included in a restaurant card are merely examples and are not limited thereto, and may vary depending on the settings.

[0181] According to one embodiment, the electronic device (101) can provide the user with information about various restaurants through this screen. The user may ask additional questions through a second utterance (1103), such as 'How many calories is the second food?' The electronic device (101) can utilize the restaurant information currently being displayed as context to process these subsequent questions.

[0182] According to one embodiment, an electronic device (101) can construct context data by extracting important information from a UI component displayed on a screen. The extracted component (1120) represents each restaurant information in the form of structured data and may have a structure similar to JSON (JavaScript Object Notation) format. Each restaurant data may include a unique identifier (id), a card title (title-card), image information (image-url), and various attributes (e.g., slot1, slot2). The types of data are merely examples and are not limited thereto, and may vary depending on the settings.

[0183] According to one embodiment, the electronic device (101) can generate a context capable of interpreting a user's subsequent utterance based on the component data extracted in this way. At this time, the electronic device (101) may determine that restaurant information currently displayed on the screen (e.g., A Sundaeguk, B Chinese Restaurant) has high importance because it is being directly viewed by the user, and may include all attribute information in the context. On the other hand, the electronic device (101) may determine that other restaurant information that can only be viewed by scrolling has relatively low importance and may include only basic attribute information in the context. The electronic device (101) can efficiently manage the size of the context to be transmitted to the LLM and provide information that can accurately respond to the user's subsequent question (e.g., second utterance (1103)).

[0184] FIG. 11b illustrates the process of an electronic device according to one embodiment classifying the importance of information items according to the user's display state and generating context data of different levels.

[0185] According to one embodiment, the electronic device (101) classifies information items into three categories based on the user's viewing state and can include different levels of attribute information in the context data for each category. Specifically, the electronic device (101) can classify information items into items that the user has already viewed, items currently being viewed, and items that have not yet been viewed, and accordingly, different data levels can be applied. The number and types of categories are merely examples and are not limited thereto, and may vary depending on the settings.

[0186] According to one embodiment, the first context (1131) may refer to context data for information items that the user has already viewed. The electronic device (101) may include key attribute information for these items. As can be seen in FIG. 11b, information such as [1, 'E Gimbap Shop', '031-111-0301'] and [2, 'D Gimbap Shop', '031-111-0302'] may be included in the first context as key attribute information. Since this information has already been verified by the user, it is not necessary to include all detailed information, but key information such as an identifier (id), business name, and contact information may be included to enable basic reference when the user asks about previously viewed items. All detailed information is merely an example and may vary depending on the settings.

[0187] According to one embodiment, the second context (1133) may refer to context data for information items currently being viewed by the user. The electronic device (101) may include all attribute information for these items. In FIG. 11b, it can be seen that detailed information is included, such as [24, 'A Sundaeguk', 'img / ...A.png', '4.6', '10,000~20,000 won', 'OPEN', ...] and [25, 'B Jjamppong House', 'img / ...B.png', '4.8', '7,000~30,000 won', 'OPEN', ...]. Since the information item currently within the user's field of vision is the most likely to prompt the user to ask follow-up questions, the electronic device (101) may include all attribute information, such as image URLs, ratings, price ranges, and operating status, in the context to prepare to answer any question. All detailed information is merely an example and may vary depending on the settings.

[0188] According to one embodiment, the third context (1135) may refer to context data for information items that the user has not yet seen. The electronic device (101) may include only basic attribute information for these items. In FIG. 11b, minimal information including only an identifier (id) and a business name, such as [32, 'D Snack Shop'], is included in the third context (1135). Since items that the user has not yet seen are unlikely to be the subject of a follow-up question, the electronic device (101) can efficiently manage the context size by including only basic information indicating the existence of the item for these items. The types or number of contexts shown in FIG. 11b are merely examples and may vary depending on the settings.

[0189] According to one embodiment, the electronic device (101) can provide an optimal information configuration that matches the user's current interests within a limited context size. When the display state of information items changes as the user scrolls the screen or moves through pages, the electronic device (101) can dynamically re-evaluate the importance of each item and update the context data. The electronic device (101) can provide an accurate voice assistant experience by always providing the context most relevant to the current situation to the LLM during a continuous conversation with the user.

[0190] FIG. 11c illustrates the structure of a prompt that an electronic device according to one embodiment transmits to a large-scale language model (LLM).

[0191] According to one embodiment, an electronic device (101) may configure a prompt to be transmitted to a large-scale language model (LLM) to process a user's utterance. This prompt may be divided into several sections and structured so that the LLM can accurately grasp the user's intent and generate an appropriate response. The prompt may include a role definition section (1141), context (1143), previous conversation content (1145), and the currently entered user utterance (1147). The configuration of the prompt is merely an example and is not limited thereto, and may vary depending on the settings.

[0192] According to one embodiment, the role definition part (1141) is an instruction specifying what role the LLM should perform, and may be configured in a form such as, "You are a voice assistant. Generate the next command to be executed based on the following information." The role definition part (1141) may serve to help the LLM understand its identity and response method as a voice assistant. The electronic device (101) may configure the role definition differently depending on the domain. The location and content of the role definition part (1141) are merely examples and may vary depending on the settings.

[0193] According to one embodiment, the context portion (1143) may refer to a portion containing information necessary to answer a user's question. The context portion (1143) may include information items optimized by the electronic device (101). In FIG. 11c, data with a differentiated amount of information according to the user's display state is listed under the [Context] section. Specifically, [1, 'E Gimbap House', '031-111-0301'] and [2, 'D Gimbap House', '031-111-0302'] may refer to key attribute information regarding information that the user has already viewed. [24, 'A Sundaeguk', 'img / ...A.png', '4.6', '10,000~20,000 Won', 'OPEN', ...] and [25, 'B Jjamppong House', 'img / ...B.png', '4.8', '7,000~30,000 Won', 'OPEN', ...] may represent all attribute information regarding the information currently being viewed by the user. [32, 'D Bunsikjeom'] may represent basic attribute information regarding information that the user has not yet viewed.

[0194] According to one embodiment, the previous conversation history (1145) may refer to a section containing the history of conversations between the user and the system up to the present. The previous conversation history (1145) may be displayed under the [History] section in the form of 'U: Show me nearby restaurants' and 'B: I found 32 restaurants nearby.' Here, 'U' may represent the utterance of the user, and 'B' may represent the response of the bot or LLM. The electronic device (101) can support the LLM in understanding the current conversation context and maintaining a continuous conversation flow through this conversation history.

[0195] According to one embodiment, the currently entered user utterance (request) (1147) may refer to the most recent question or command entered by the user. The currently entered user utterance (1147) may be displayed under the [Request] section in a form such as 'U: How many calories is the second food?'. The electronic device (101) can use the currently entered user utterance (1147) to clearly specify the current question to which the LLM must respond. The electronic device (101) can control the LLM to generate an appropriate response by comprehensively considering the currently entered user utterance (1147), the previously provided context, and the conversation history.

[0196] According to one embodiment, the electronic device (101) can provide an optimal information configuration within a limited context size by differentiating the amount of information included in the context according to the user's 'viewing state'. The electronic device (101) can provide all attribute information, including image URLs, for the information currently being viewed by the user, thereby enabling a detailed response. Additionally, the electronic device (101) can automatically determine importance by analyzing visual characteristics such as the size of text displayed in the UI, and can perform effective context optimization without the input of separate weight information.

[0197] According to one embodiment, if the electronic device (101) includes a request that the LLM response is insufficient with the current context data, it can generate additional context data for a specific information item requested and generate a new prompt including the additional context data and control it to retransmit to the LLM.

[0198] According to one embodiment, if the electronic device (101) includes a request that the current context data is insufficient for the LLM response, it can control the generation of additional context data for the requested specific information item and generate a new prompt containing the additional context data to be retransmitted to the LLM. For example, when a user asks, "What are the business hours of the third restaurant?", if the context transmitted to the LLM does not include information on the business hours of the third restaurant, the LLM may return a response such as "Information on the business hours of the third restaurant is needed." Upon receiving such a response, the electronic device (101) may identify the requested specific information (e.g., the business hours of the third restaurant) and generate additional context data by extracting the information from a database or original data source.

[0199] According to one embodiment, when additional context data is generated, the electronic device (101) may construct a new prompt including the original user question, previously provided context, and newly generated additional context. The electronic device (101) may retransmit the new prompt to the LLM to obtain a more accurate response. Even if the electronic device (101) does not initially include all possible information in the context, it can respond accurately to the user question while efficiently managing context size limits by gradually adding information as needed.

[0200] According to one embodiment, the electronic device (101) may control the inclusion of source data used to create information items, along with information about a plurality of information items, into the context data. Source data may refer to original data used to create information items displayed on the UI. The source data may take the form of at least one of a database record, an API response, or a document in a file system. For example, when displaying restaurant information, only some information such as the business name, representative image, and rating may be displayed on the UI, but more information (e.g., detailed address, business hours, phone number, full menu, reviews) may be stored in the actual database. The electronic device (101) may include additional fields of this source data in the context as well as the information displayed on the UI. The types of information presented are merely examples and may vary depending on the settings.

[0201] According to one embodiment, the electronic device (101) can receive questions and provide responses regarding information that the user cannot directly see on the screen. For example, even if only the restaurant name and rating are displayed on the screen, if the source data includes information on business hours when the user asks, "What are the business hours of the first restaurant?", the LLM can respond to this.

[0202] The embodiments of this document disclosed in this specification and drawings are provided merely as specific examples to facilitate the explanation of the technical content according to the embodiments of this document and to aid in understanding the embodiments of this document, and are not intended to limit the scope of the embodiments of this document. Accordingly, the scope of the embodiments of this document should be interpreted to include all modifications or variations derived based on the technical concept of the embodiments of this document, in addition to the embodiments disclosed herein.

Claims

1. In an electronic device, Memory that stores instructions and includes one or more storage media; It includes at least one processor comprising processing circuitry, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device Receive user input, Displays a user interface (UI) including a plurality of information items in response to the above user input, and Checking user input (interaction) or viewing state for multiple information items displayed on the above user interface (UI), and Based on the above-mentioned confirmed user input or display state, the importance of each of the plurality of information items is determined, and Based on the importance determined above, the amount of information (data level) of context data to be provided to the LLM (Large Language Model) from each information item is determined, and Context data is extracted from the plurality of information items based on the above-determined amount of information, and A prompt including the extracted context data and the user input is generated and transmitted to the LLM, and Receives a response to the prompt from the LLM and controls the execution of an operation based on the received response, The above data level refers to the amount of information to be included in the context data for each information item, and The above user input refers to an electronic device that means an action in which the user directly interacts with the corresponding information item.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Classify the above display state into either the state currently being viewed by the user or the state that the user has not yet seen, and It is determined that the information item currently being viewed by the user includes all attribute information of the said information item, and It is determined that information items that the user has not yet seen contain only the basic attribute information of the relevant information item, and An electronic device in which the above basic attribute information refers to the minimum information required to identify the corresponding information item.

3. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device The above display state is classified into one of the state currently being viewed by the user, the state that the user has passed by, and the state that the user has not yet seen, and The above importance is determined as one of the first importance, second importance, and third importance based on the above classified display state, and The first importance is assigned to the information item that the user is currently viewing, the second importance is assigned to the information item that the user has passed by, and the third importance is assigned to the information item that the user has not yet viewed. For the information item of the first importance level mentioned above, it is determined to include all attribute information of the said information item, and For the information item of the second importance mentioned above, it is determined to include intermediate level attribute information of the said information item, and For the information item of the third importance level mentioned above, it is decided to include only the basic attribute information of the relevant information item, and The above intermediate-level attribute information includes key attributes related to identification, content, and characteristics as part of all attribute information, and An electronic device in which the above basic attribute information refers to the minimum information required to identify the corresponding information item.

4. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device The importance is determined based on at least one of the font size, font weight, or display position of each information item, and An electronic device that controls assigning a relatively higher importance as the font size increases, the font weight increases, or the display position is located at the top.

5. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Check the predefined weight associated with the above information item or the function that generated the information item, and An electronic device that controls the application of the aforementioned predefined weights preferentially when determining importance, if the aforementioned predefined weights exist.

6. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Check the context size limit for the above LLM, and An electronic device that controls the amount of information to dynamically adjust based on the above context size limit.

7. In Paragraph 1, An electronic device in which the above display state is determined based on the screen visibility of the information items at the time of user input or the scroll state of the user interface.

8. In Paragraph 1, When the above plurality of information items include an image, the importance is An electronic device determined based on at least one of whether text is included in the video, the video playback time, or the user's video viewing history.

9. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Interpreting the response received from the above LLM to generate voice response and visual information to be provided to the user, and An electronic device that stores the generated voice response and visual information as context data and controls them to be used as context for subsequent user input.

10. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device If the request includes that the LLM response is insufficient with the current context data, Generate additional context data for the requested specific information item, and An electronic device that controls the generation of a new prompt containing the above additional context data and retransmission to the LLM.

11. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device An electronic device that controls the inclusion of source data used to generate the information items, along with information about the plurality of information items, into the context data.

12. A computer-readable non-transient storage medium storing one or more programs comprising instructions executable by a processor of an electronic device, When the above instructions are executed, the electronic device, Receive user input, Displays a user interface (UI) including a plurality of information items in response to the above user input, and Checking user input (interaction) or viewing state for multiple information items displayed on the above user interface (UI), and Based on the above-mentioned confirmed user input or display state, the importance of each of the plurality of information items is determined, and Based on the importance determined above, the amount of information (data level) of context data to be provided to the LLM (Large Language Model) from each information item is determined, and Context data is extracted from the plurality of information items based on the above-determined amount of information, and A prompt including the extracted context data and the user input is generated and transmitted to the LLM, and Receives a response to the prompt from the LLM and controls the execution of an operation based on the received response, The above data level refers to the amount of information to be included in the context data for each information item, and The above user input refers to a computer-readable, non-transient storage medium in which the user directly interacts with the information item.

13. In Paragraph 12, When the above instructions are executed, the electronic device Classify the above display state into either the state currently being viewed by the user or the state that the user has not yet seen, and It is determined that the information item currently being viewed by the user includes all attribute information of the said information item, and It is determined that information items that the user has not yet seen contain only the basic attribute information of the relevant information item, and The above basic attribute information refers to a computer-readable non-transient storage medium that means the minimum information for identifying the relevant information item.

14. In Paragraph 12, When the above instructions are executed, the electronic device The above display state is classified into one of the state currently being viewed by the user, the state that the user has passed by, and the state that the user has not yet seen, and The above importance is determined as one of the first importance, second importance, and third importance based on the above classified display state, and The first importance is assigned to the information item that the user is currently viewing, the second importance is assigned to the information item that the user has passed by, and the third importance is assigned to the information item that the user has not yet viewed. For the information item of the first importance level mentioned above, it is determined to include all attribute information of the said information item, and For the information item of the second importance mentioned above, it is determined to include intermediate level attribute information of the said information item, and For the information item of the third importance level mentioned above, it is decided to include only the basic attribute information of the relevant information item, and The above intermediate-level attribute information includes key attributes related to identification, content, and characteristics as part of all attribute information, and The above basic attribute information refers to a computer-readable non-transient storage medium that means the minimum information for identifying the relevant information item.

15. In Paragraph 12, When the above instructions are executed, the electronic device The importance is determined based on at least one of the font size, font weight, or display position of each information item, and A computer-readable non-transient storage medium that controls assigning relatively higher importance as the font size increases, the font weight increases, or the display position is located at the top.