Electronic device and control method therefor

Abstract

An electronic device and a control method therefor are disclosed. The present electronic device includes a memory storing instructions, and a processor, wherein the instructions, when executed individually or collectively by the processor, cause the electronic device to: when a trigger event is detected, activate an artificial intelligence (AI) assistant application; when a user input is received while the assistant application is activated, identify at least one of a plurality of databases on the basis of a lock state of the electronic device and load the identified at least one of the plurality of databases into a language model; obtain response information for the user input by inputting the user input into the language model into which the identified database has been loaded; and output the obtained response information.

Description

Electronic device and control method thereof

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more specifically, to an electronic device and a method for controlling the same for obtaining response information to user input using a language model.

[0002] As the transition from rule-based assistant applications to LLM (Large Language Model)-based assistant applications occurred, electronic devices were able to engage in general conversation through the assistant applications, rather than just controlling specific functions of the electronic device.

[0003] However, information can be provided to the user even if the electronic device is not unlocked while performing a conversation with the electronic device in this manner. When providing response information to the user while the device remains locked, there were issues such as the exposure of personal privacy or the requirement of additional unlocking actions.

[0004] Meanwhile, the information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] According to one embodiment of the present disclosure, an electronic device comprises: a memory for storing instructions; and a processor; wherein, when the instructions are executed individually or collectively by the processor, the electronic device activates an Artificial Intelligence (AI) assistant application when a trigger event is detected, and when user input is received while the assistant application is activated, it identifies at least one of a plurality of databases based on the lock state of the electronic device and loads it into a Language Model, inputs the user input into the Language Model in which the identified database is loaded to obtain response information for the user input, and outputs the obtained response information.

[0006] When the above instructions are executed individually or collectively by the processor, the electronic device may identify a public database and load it into the language model when user input is received while the electronic device is in a locked state, and input the user input into the language model loaded with the public database to obtain first response information.

[0007] When the above instructions are executed individually or collectively by the processor, the electronic device may, when user input is received while the electronic device is in an unlocked state, identify a private database associated with at least one of the usage state of the electronic device and the user input and load it into the language model, and input the user input into the language model loaded with the private database to obtain second response information.

[0008] When the above instructions are executed individually or collectively by the processor, the electronic device may, when user input is received while the electronic device is in an unlocked state, identify a private database corresponding to at least one application running in the background and load it into the language model, and input the user input into the language model in which the private database corresponding to the at least one application is loaded to obtain the second response information.

[0009] When the above instructions are executed individually or collectively by the processor, the electronic device may identify the application associated with the user input among the at least one application and load a private database corresponding to the application associated with the user input into the language model when user input is received while the electronic device is unlocked.

[0010] When the above instructions are executed individually or collectively by the processor, the electronic device may, when user input is received while the electronic device is unlocked, identify a private database storing personal information and load it into the language model, and input the user input into the language model loaded with the private database storing personal information to obtain the second response information.

[0011] When the above instructions are executed individually or collectively by the processor, the electronic device may identify at least one of a plurality of databases based on the lock state of the electronic device and load it into a generative AI model if the user input is a user input requesting content creation, and input the user input into the generative AI model loaded with the identified database to obtain generated content for the user input.

[0012] The above trigger event may be one of an event in which a user voice containing a preset wake word is input, an event in which a preset button is selected, an event in which a preset gesture is detected, and an event in which a preset touch input is received on a touch screen.

[0013] When the above instructions are executed individually or collectively by the processor, the electronic device provides an alarm message to indicate that the AI ​​assistant application is activated when the AI ​​assistant application is activated, and the alarm message may include at least one of an audio-based alarm message and a visual-based alarm message.

[0014] Meanwhile, a control method for an electronic device according to one embodiment of the present disclosure comprises: a step of activating an AI (Artificial Intelligence) assistant application when a trigger event is detected; a step of identifying at least one of a plurality of databases and loading it into a language model based on the lock state of the electronic device when user input is received while the assistant application is activated; a step of inputting the user input into the language model loaded with the identified database to obtain response information for the user input; and a step of outputting the obtained response information.

[0015] The loading step involves identifying a public database and loading it into the language model when user input is received while the electronic device is in a locked state, and the acquiring step involves inputting the user input into the language model loaded with the public database to acquire first response information.

[0016] The loading step involves, when user input is received while the electronic device is in an unlocked state, identifying a private database associated with at least one of the usage state of the electronic device and the user input and loading it into the language model, and the acquiring step involves inputting the user input into the language model loaded with the private database to acquire second response information.

[0017] The loading step involves, when user input is received while the electronic device is unlocked, identifying a private database corresponding to at least one application running in the background and loading it into the language model, and the acquiring step involves inputting the user input into the language model loaded with the private database corresponding to the at least one application to acquire the second response information.

[0018] The loading step may include: a step of identifying an application among the at least one application that is associated with the user input when the electronic device is unlocked and user input is received; and a step of loading a private database corresponding to the application associated with the user input into the language model.

[0019] The loading step involves, when user input is received while the electronic device is unlocked, identifying a private database storing personal information and loading it into the language model, and the acquiring step involves inputting the user input into the language model loaded with the private database storing personal information to acquire the second response information.

[0020] The above control method may include: a step of identifying at least one of a plurality of databases based on the lock state of the electronic device and loading it into a generative AI model if the user input is a user input requesting content creation; and a step of inputting the user input into the generative AI model loaded with the identified database to obtain generated content for the user input.

[0021] The above trigger event may be one of an event in which a user voice containing a preset wake word is input, an event in which a preset button is selected, an event in which a preset gesture is detected, and an event in which a preset touch input is received on a touch screen.

[0022] The above control method further includes the step of providing an alarm message to indicate that the AI ​​assistant application is activated when the AI ​​assistant application is activated; and the alarm message may include at least one of an audio-based alarm message and a visual-based alarm message.

[0023] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.

[0024] FIG. 2 illustrates an example of a block diagram of an electronic device according to one embodiment of the present disclosure.

[0025] FIG. 3 is a block diagram illustrating a configuration for generating content using a generative AI model according to one embodiment.

[0026] FIG. 4 is a flowchart illustrating an embodiment of obtaining response information to user input when an electronic device is in a locked state, according to one embodiment of the present disclosure.

[0027] FIG. 5 is a flowchart illustrating an embodiment of obtaining response information to user input when an electronic device is in an unlocked state, according to one embodiment of the present disclosure.

[0028] FIG. 6 is a flowchart illustrating an embodiment of an electronic device acquiring response information to user input depending on whether it is in a locked state, according to one embodiment of the present disclosure.

[0029] FIGS. 7a and 7b are drawings for explaining an embodiment of obtaining response information for user input using an LLM loaded with a public database when an electronic device is in a locked state, according to one embodiment of the present disclosure.

[0030] FIGS. 8a to 9 are drawings for explaining an embodiment of obtaining response information for user input using an LLM loaded with a private database when an electronic device is in an unlocked state, according to one embodiment of the present disclosure.

[0031] FIGS. 10a and FIGS. 11b are drawings for illustrating an embodiment of an electronic device acquiring response information to user input depending on whether it is in a locked state, according to one embodiment of the present disclosure.

[0032] FIGS. 12a and FIGS. 12b are drawings for illustrating an embodiment of an electronic device that generates content for user input depending on whether it is in a locked state, according to one embodiment of the present disclosure.

[0033] FIGS. 13a and FIGS. 13b are drawings for illustrating an embodiment of an electronic device according to one embodiment of the present disclosure, which generates content for user input using different response parameters depending on whether the electronic device is in a locked state.

[0034] The present disclosure will be described in detail below with reference to the attached drawings.

[0035] The electronic device according to the various embodiments disclosed in this document may be a device of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this disclosure is not limited to the devices described above.

[0036] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, each of phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any component (e.g., the first) is referred to as "coupled" or "connected" to another component (e.g., the second), with or without the terms "functionally" or "communicationally," it means that said component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0037] Throughout the specification, singular expressions include plural expressions unless the context clearly indicates otherwise. Wherever a part of the specification states that it "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0038] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0039] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments.

[0040] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0041] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0042] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0043] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0044] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0045] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0046] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0047] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0048] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0049] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0050] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0051] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0052] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0053] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0054] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0055] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0056] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0057] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0058] An antenna module (197) can transmit a signal or power to an external source (e.g., an external electronic device) or receive it from an external source. According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0059] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0060] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0061] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used.

[0062] The electronic device (101) can provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within the second network (199). The electronic device (101) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0063] FIG. 2 illustrates an example of a block diagram of an electronic device according to one embodiment. The electronic device (101) of FIG. 1 may correspond to the electronic device (200) of FIG. 2. The electronic device (200) of FIG. 2 may include the electronic device (101) of FIG. 1. The electronic device (200) of FIG. 2 may include at least one component among the components of the electronic device (101) illustrated in FIG. 1. The components illustrated in FIG. 2, their relationships, and their functions are merely exemplary and are not intended to limit the implementations described or claimed herein.

[0064] Referring to FIG. 2, the electronic device (200) may include at least one processor (210) (hereinafter referred to as processor (210)), at least one memory (220) (hereinafter referred to as memory (220)), at least one display (230) (hereinafter referred to as display (230)), and at least one communication circuit (240) (hereinafter referred to as communication circuit (240)). The processor (210), memory (220), display (230), and communication circuit (240) of FIG. 2 may each perform the same functions by being implemented substantially identically to the processor (120), memory (130), display module (160), and communication module (190) of FIG. 1.

[0065] The components illustrated in FIG. 2 are merely exemplary. For example, the electronic device (200) may include other components (e.g., at least one image sensor, at least one sensor, a PMIC (power management integrated circuitry), an audio processing circuit, an antenna, a rechargeable battery, and / or an input / output interface). For example, some components may be omitted from the electronic device (200). For example, some components may be integrated into a single component.

[0066] Memory (220) may include one or more storage media (or one or more storage devices). For example, memory (220) may include a memory assembly comprising one or more storage media. For example, the one or more storage media may include a hard drive, flash memory, permanent memory (e.g., non-volatile memory) such as ROM (read-only memory), semi-permanent memory (e.g., volatile memory) such as RAM (random access memory), any other suitable type of storage (or storage assembly), or any combination thereof. Memory (220) may include a cache memory, which is one or more different types of memory used to temporarily store data for a function or feature of the electronic device (200). As an example, but not limited to, the cache memory may be included within the processor (210). The memory (220) can be fixedly embedded in the electronic device (200) or incorporated into one or more suitable types of components (e.g., a SIM (subscriber identity module) card and / or an SD (secure digital) card) that can be repeatedly inserted into and removed from the electronic device (200).

[0067] For example, memory (220) may store one or more software applications, such as operating system (or system) software applications, firmware software applications, driver software applications, plugin (e.g., add-in, add-on, and / or applet) software applications, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by the processor (210). For example, memory (220) may store instructions that can be called by an application programming interface (API). For example, memory (220) may store instructions within a library.

[0068] The display (230) is controlled by the processor (210) and can output visualized information to the user. The visualized information may include visual objects displayed on the display (230). For example, the visual objects may include screens, images, icons, GUI (graphic user interface), UI (user interface) elements, etc.

[0069] For example, the display (230) can be implemented as a flat panel display (FPD), a curved display, or a flexible display. For example, the display (230) can be implemented as various types of displays such as an LCD (Liquid Crystal Display), an AMOLED (Active Matrix Organic Light Eemitting Diodes) display, an LED (Light Eemitting Diodes), a micro LED, a mini LED, etc.

[0070] The display (230) may include a touch detection circuit configured to detect touch. The touch detection circuit may acquire user input that touches (or selects) a visual object displayed on the display (230). For example, the touch detection circuit may detect an input (e.g., touch input or hovering input) on a specific location by measuring a change in a signal (e.g., voltage, light intensity, resistance, or charge) for a specific location on the display (230), and provide information regarding the detected input to the processor (210).

[0071] The communication circuit (240) can perform data communication with an external electronic device under the control of the processor (210). For example, the communication circuit (240) can communicate with an external electronic device through a network. For example, the processor (210) can receive data from an external electronic device through the communication circuit (240) and transmit data to an external electronic device through the communication circuit (240).

[0072] The communication circuit (240) may include hardware components to support the transmission and / or reception of electrical signals between the electronic device (200) and an external electronic device. For example, the communication circuit (240) may support the transmission and / or reception of electrical signals based on various types of protocols such as Ethernet, LAN (local area network), WAN (wide area network), Wi-Fi (wireless fidelity), Bluetooth, BLE (bluetooth low energy), ZigBee, LTE (long term evolution), 5G NR (new radio) and / or 6G.

[0073] The processor (210) can control the overall operations of the electronic device (200). For example, the processor (210) can control the operations of the electronic device (200) by being operatively connected to the memory (220), the display (230), and the communication circuit (240). Alternatively, the processor (210) can control the operations of the electronic device (200) according to the present disclosure by executing one or more instructions stored in the memory (220). The processor (210) may be composed of one or more processors.

[0074] The processor (210) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing operations. The processor (210) may include at least one electrical circuit and may process instructions (or programs, data, etc.) stored in memory (220) individually or collectively in a distributed manner. The processor (210) may include a processor assembly comprising one or more processing circuits. The processor (210) may include any processing circuit that is operative to control the performance and operation of one or more components of the electronic device (200) (e.g., memory (220), display (230), communication circuit (240)). For example, the processor (210) (e.g., application processor (AP)) may be implemented as a system on chip (SoC) (e.g., a single chip or chipset). For example, the processor (210) may be implemented with a plurality of cores (or at least one core circuit), a plurality of chips, or a plurality of chipsets. For example, the processor (210) may include one or more processing circuits. For example, the processor (210) may include one or more processing circuits configured to perform the various functions of the present disclosure individually and / or collectively. As an example without limitation, at least a portion of the processor (210) may be included in a first chip of the electronic device (200), and at least another portion of the processor (210) may be included in a second chip of the electronic device (200) different from the first chip of the electronic device (200).

[0075] For example, the processor (210) may include a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a display controller, a memory controller, a storage controller, a communication processor (CP), and / or a sensor interface. These components of the processor (210) are merely exemplary. For example, the processor (210) may include other components. For example, some components of the processor (210) may be omitted from the processor (210). For example, some components of the processor (210) may be included as separate components of the electronic device (200) outside the processor (210). For example, some components of the processor (210) (e.g., a memory controller) may be included within other components (e.g., at least a portion of the memory (220), an interface (e.g., available for connection to at least one component of the electronic device (200)), a display (230)).

[0076] The processor (210) may cause other components of the electronic device (200) to perform various operations by executing instructions stored in memory (220). The CPU (or central processing circuit) may be configured to control the components of the processor (210) based on the execution of instructions stored in memory (220) (e.g., volatile memory and / or non-volatile memory). The GPU (or graphics processing circuit) may be configured to execute parallel operations (e.g., rendering). The NPU (or neural processing circuit, or AI (artificial intelligence) chip) may be configured to execute operations for an artificial intelligence model (e.g., convolution computation). The ISP (or image signal processing circuit) may be configured to process a raw image acquired through an image sensor into a format suitable for components within the electronic device (200) or components of the processor (210). A display controller (or display control circuit, or DPU (display processing unit)) may be configured to process an image obtained from a CPU, GPU, ISP, or memory (220) (e.g., volatile memory) into a format suitable for display (230). A memory controller (or memory control circuit) may be configured to control reading data from volatile memory and writing data to volatile memory. A storage controller (or storage control circuit) may be configured to control reading data from non-volatile memory and writing data to non-volatile memory. A CP (communication processing circuit) may be configured to process data obtained from a component of the processor (210) into a format suitable for transmitting to another electronic device via a communication circuit, or to process data obtained from another electronic device via a communication circuit into a format suitable for processing by a component of the processor (210).For example, the communication circuit may include one or more communication circuits. The sensor interface (or sensing data processing circuit, sensor hub) may be configured to process data regarding the state of the electronic device (200) and / or the state around the electronic device (200), obtained through the sensor, into a format suitable for the components of the processor (210).

[0077] For example, the processor (210) can perform the method according to an embodiment of the present disclosure by executing one or more instructions stored in memory (220).

[0078] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0079] The processor (210) may be implemented as a single-core processor including one core, or as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When the processor (210) is implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. Each of the multiple cores included in the multicore processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one embodiment of the present disclosure.

[0080] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0081] In the embodiments of the present disclosure, a processor may mean a system-on-chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, and the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto.

[0082] In particular, the processor (210) executes an instruction stored in memory (220) to activate an AI (Artificial Intelligence) assistant application when a trigger event is detected, and when user input is received while the assistant application is activated, it identifies at least one of a plurality of databases based on the lock state of the electronic device (200) and loads it into a language model, inputs the user input into the language model in which the identified database is loaded to obtain response information for the user input, and outputs the obtained response information.

[0083] According to one or more embodiments, when user input is received while the electronic device (200) is in a locked state, the processor (210) identifies a public database and loads it into a language model, and inputs the user input into the language model loaded with the public database to obtain first response information.

[0084] According to one or more embodiments, when user input is received while the electronic device (200) is in an unlocked state, the processor (210) identifies a private database associated with at least one of the usage state of the electronic device (200) and the user input and loads it into a language model, and inputs the user input into the language model loaded with the private database to obtain second response information.

[0085] According to one or more embodiments, when user input is received while the electronic device (200) is in an unlocked state, the processor (210) identifies a private database corresponding to at least one application running in the background and loads it into a language model, and inputs the user input into the language model in which the private database corresponding to at least one application is loaded to obtain second response information.

[0086] According to one or more embodiments, when user input is received while the electronic device (200) is unlocked, the processor (210) may identify an application associated with the user input among at least one application and load a private database corresponding to the application associated with the user input into a language model.

[0087] According to one or more embodiments, when user input is received while the electronic device (200) is unlocked, the processor (210) identifies a private database containing personal information and loads it into a language model, and inputs the user input into the language model in which the private database containing personal information is loaded to obtain second response information.

[0088] According to one or more embodiments, if the user input is a user input requesting content creation, the processor (210) identifies at least one of a plurality of databases based on the lock state of the electronic device (200) and loads it into a generative AI model, and inputs the user input into the generative AI model loaded with the identified database to obtain generated content for the user input.

[0089] According to one or more embodiments, the trigger event may be one of an event in which a user voice including a preset call word is input, an event in which a preset button is selected, an event in which a preset gesture is detected, and an event in which a preset touch input is received on a touch screen.

[0090] According to one or more embodiments, the processor (210) may provide an alarm message to notify that the AI ​​assistant application is activated when the AI ​​assistant application is activated, and the alarm message may include at least one of an audio-based alarm message and a visual-based alarm message.

[0091]

[0092] FIG. 3 is a block diagram illustrating a configuration for generating content using a generative AI model according to one embodiment. Referring to FIG. 3, an electronic device (200) may include a User Query / Response Interface (310), an AI framework (320), a Generative AI Model (330), a knowledge database (340), and an application / service component (350). The configuration illustrated in FIG. 3 may be implemented in software, but this is merely one embodiment and may be implemented as a combination of software and hardware. Some components of FIG. 3 (e.g., the Generative AI Model (330) or the knowledge database (340), etc.) may be stored on an external server linked to the electronic device (200) rather than on the electronic device (200).

[0093] The user query / response interface (310) can receive user input (or user query). User input may include information regarding natural language, images, and / or videos. When acquiring user input, the user query / response interface (310) may also acquire context information. When acquiring user input, context information may include various additional information. For example, context information may include information about the application currently being used by the user or the user's location information. User input may be acquired in a mixed form of the aforementioned natural language, images, sounds, and context information. User input may also be in a non-natural language form, such as user input selecting a menu. The user query / response interface (310) may output results (or response information) of the generative AI model (330) to the user. The output may be in the form of natural language or specific content, and may be provided in the form of actions requested by the user.

[0094] The AI ​​framework (320) can receive user input and coordinate and control each component necessary to perform the user's intent based on the user's query. The AI ​​framework (320) may include a prompt design component (321), an API / Plug-in management component (322), and an output modification component (323).

[0095] User input received from the user query / response interface (310) can be transmitted to a prompt design component (321). The prompt design component (321) can generate prompts suitable for inputting user input into a generative AI model (330) (e.g., a Large Language Model (LLM) or a Large Multi Modal Model (LMM)). The prompt design component (321) may be an AI component that uses machine learning algorithms or neural networks to develop better prompts over time. The prompt design component (321) can generate prompts by accessing a knowledge database (340) containing user preference data, a prompt library, and prompt examples based on user input, and can transmit the generated prompts to the LLM or LMM.

[0096] The APIs / plugin management component (322) can perform the role of communicating with external information when there is a request for additional information when transmitting user input as input to the generative AI model (330). The APIs / plugin management component (322) establishes a channel to communicate with the outside of the AI ​​Interface through an API (Application Programming Interface), and can enable access to various knowledge databases (340) through the established channel. If the APIs / plugin management component (322) needs to perform an action that ultimately executes user input rather than an intermediate result in the application or service, it can request that action from the application / service component (350) via the API. Information obtained from the outside may be used to generate a prompt in the prompt design component (321) along with user input, or it may be transmitted as input to the generative AI model (330).

[0097] The output modification component (323) can modify (or tune) the output output from the generative AI model (330) in detail. For example, the output modification component (323) can verify whether the content generated through LLM and / or LMM is irrelevant, contains biased content, or contains harmful content. The output modification component (323) can also determine how well the output matches the desired result and, if additional processing is required, proceed with that process. Additionally, the output modification component (323) can configure and provide hints to the user to avoid unwanted output.

[0098] A generative AI model (330) generally refers to an artificial intelligence neural network that generates new forms of data based on user input information. A generative AI model (330) may include a model that generates images and / or a model that generates language. Models that generate images include, but are not limited to, GANs (generative adversarial networks) and VAEs (variational autoencoders), and examples include Diffusion-based generative models that use VAEs and Transformer structures. Models that generate language are models trained to output the most statistically appropriate output value based on input values, and examples include LLMs. There are also LMMs that can recognize various forms of data input, such as text, images, and voice, and generate new data corresponding to them.

[0099] According to one embodiment, an electronic device (100) may operate in one of a first state and a second state. The first state refers to a state where the electronic device is locked, and the second state refers to a state where the electronic device is unlocked. In the first state, since the electronic device (100) is not unlocked, a voice assistant application included in the electronic device (100) can receive voice or text input from a user, embed the input, and use it as an input to the LLM, and the LLM can generate and provide to the user. In the second state, when user input is received while the electronic device (100) is unlocked, the received user input is embedded, and the context of the query is verified using the result of loading at least one vector database based on the query or user's context, and the verified result is received as an input to the LLM, through which the LLM can generate a response reflecting the user query or the user's context-based vector database and provide it to the user. In the second state, unlike the first state, by providing LLM with responses that reflect personal information or personal context, users can receive customized information necessary for the situation.

[0100] Hereinafter, with reference to FIG. 4, an embodiment for obtaining response information for user input in a first state (i.e., a locked state) will be described in detail, and with reference to FIG. 5, an embodiment for obtaining response information for the user in a second state (i.e., an unlocked state) will be described in detail.

[0101] FIG. 4 is a flowchart illustrating an embodiment of obtaining response information to user input when an electronic device is in a locked state, according to one embodiment of the present disclosure.

[0102] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0103] According to one or more embodiments, operations 410 to 450 may be understood to be performed in a processor (e.g., processor (210) of FIG. 2) of an electronic device (e.g., electronic device (200) of FIG. 2).

[0104] According to one or more embodiments, the electronic device (200) may maintain a locked state. The locked state is a state in which the functions of the electronic device (200) are partially restricted to limit use by unauthorized users, and may be referred to by various terms such as a blocked state, a standby state, a protected state, or a disabled state.

[0105] Referring to operation 410, the electronic device (200) can detect a trigger event. For example, a trigger event may be an event for activating an AI assistant application to receive various services through the AI ​​assistant application. For example, the trigger event may be an event in which a user voice containing a preset wake word (e.g., Bixby) is input. According to one embodiment, the trigger event may be an event in which a physical button placed in a preset area of ​​the electronic device (200) is selected. For example, the trigger event may be an event in which a preset gesture is detected through a camera. For example, the preset gesture input through the camera may include user input using the user's gaze. For example, the trigger event may be an event in which a preset touch input is received on a touch screen. For example, this is merely one embodiment, and it goes without saying that it can be implemented with various trigger events.

[0106] Referring to operation 420, the electronic device (200) may receive at least one user input. The user input is an input for obtaining a response or content requested by the user. According to one embodiment, as an example of the user input, at least one word may be recognized and activated as a trigger event, or a pre-specified user input may be detected using at least one sensing means installed in the electronic device. For example, the user input may include at least one of audio (e.g., user voice), text (e.g., text entered by the user via a touch screen), an image, and a video. In particular, the user input may include not only one type but also multiple types simultaneously. For example, the user input may include the user voice along with an image.

[0107] Referring to operation 430, the electronic device (200) can obtain an embedding vector by inputting user input into an embedding model. The embedding model is a model for converting text, sentences, or other forms of data into a high-dimensional vector space, and may be a model for converting text into a numerical representation. The embedding vector is a numerical representation generated by the embedding model and can be input into an LLM model.

[0108] Referring to operation 440, the electronic device (200) can input an embedding vector into a Large Language Model (LLM). The LLM is a language model composed of an artificial neural network containing numerous parameters. The LLM can be trained on a significant amount of unlabeled corpus text using self-supervised or semi-self-supervised learning. In this case, the LLM may not only have the ability to generate answers to user inquiries but may also include reasoning capabilities and the ability to formulate and execute plans on its own. Meanwhile, the LLM may be referred to by various terms such as a large language model, an AI chatbot model, etc.

[0109] The LLM can be stored within the electronic device (200), but this is merely one example, and it is obvious that it can be stored on an external server.

[0110] Referring to operation 450, the electronic device (200) can obtain response information from the LLM. The response information may be implemented in text form, but this is merely one example and may be implemented in other forms (e.g., images).

[0111] The electronic device (200) can obtain response information from the LLM solely through user input without context information for a separate user while in a locked state.

[0112] FIG. 5 is a flowchart illustrating an embodiment of obtaining response information to user input when an electronic device is in an unlocked state, according to one embodiment of the present disclosure.

[0113] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0114] According to one embodiment, operations 510 to 560 may be understood to be performed in a processor (e.g., processor (210) of FIG. 2) of an electronic device (e.g., electronic device (200) of FIG. 2).

[0115] According to one embodiment, the electronic device (200) may maintain an unlocked state. The unlocked state is a state in which the functions of the electronic device (00) can be used through user authentication (e.g., password authentication, biometric authentication, etc.) from a locked state, and may be referred to by various terms such as unlocked state, activated state, access allowed state, etc.

[0116] Referring to operation 510, the electronic device (200) can detect a trigger event. The trigger event may be, but is not limited to, an event in which a user voice containing a preset wake word (e.g., Bixby) is input, an event in which a physical button placed in a preset area of ​​the electronic device (200) is selected, an event in which a preset gesture is detected through a camera, or an event in which a preset touch input is received on a touch screen. After the trigger event is detected, the electronic device (200) can check information related to the lock of the electronic device (200). In one or more embodiments, the electronic device (200) can identify whether the state of the electronic device (200) is locked or unlocked.

[0117] Referring to operation 520, the electronic device (200) can receive user input. The user input is an input for obtaining a response or content requested by the user, and the user input may include at least one of audio (e.g., user voice), text (e.g., text entered by the user through a touch screen), an image, and a video.

[0118] Referring to operation 530, the electronic device (200) can obtain an embedding vector by inputting user input into an embedding model.

[0119] Referring to operation 540, the electronic device (200) can obtain context information corresponding to an embedding vector. The electronic device (200) can obtain context information corresponding to an embedding vector through a DB containing personal information or usage information of the user currently using the electronic device. The context information may include the user's personal information, information about the application the user is currently using the electronic device (200), etc.

[0120] Referring to operation 550, the electronic device (200) can input embedding vectors and context information into the LLM. For example, the electronic device (200) can input embedding vectors and context information into the LLM, respectively, but this is merely one example, and new user input can be obtained and input into the LLM using embedding vectors and context information.

[0121] Referring to operation 560, the electronic device (200) can obtain response information from the LLM. The response information is information obtained by using context information together with user input, rather than just user input, and may include information necessary for the current user situation.

[0122] For example, the electronic device (200) can obtain response information from the LLM by using user context information as well as user input while in an unlocked state. For example, the electronic device can obtain a response suitable for the current user situation by using the response information obtained from the LLM.

[0123] FIG. 6 is a flowchart illustrating an embodiment of an electronic device acquiring response information to user input depending on whether it is in a locked state, according to one embodiment of the present disclosure.

[0124] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0125] According to one embodiment, operations 610 to 680 may be understood to be performed in a processor (e.g., processor (210) of FIG. 2) of an electronic device (e.g., electronic device (200) of FIG. 2).

[0126] Referring to operation 610, the electronic device (200) can detect a trigger event. The trigger event may be an event in which a user voice containing a preset wake word (e.g., Bixby) is input, an event in which a physical button placed in a preset area of ​​the electronic device (200) is selected, an event in which a preset gesture is detected through a camera, or an event in which a preset touch input is received on a touch screen, but is not limited thereto.

[0127] Referring to operation 620, the electronic device (200) can activate an AI assistant application. The AI ​​assistant application may be an AI-based application that performs functions such as assisting the user with daily tasks, answering questions, or automating tasks. The AI ​​assistant application can understand and process the user's commands by utilizing natural language processing and machine learning technologies. In particular, when the AI ​​assistant application is activated, the electronic device (200) may provide an alarm message to notify that the AI ​​assistant application is activated. The alarm message may include at least one of an audio-based alarm message and a visual-based alarm message. For example, the electronic device (200) may provide a beep sound or a voice message such as "The AI ​​assistant is activated," and may provide at least one of an object, icon, or text corresponding to the AI ​​assistant on a part of the screen. The electronic device (200) may change and display the attributes of the object corresponding to the AI ​​assistant while the AI ​​assistant application is activated. For example, the electronic device (200) can display at least one of the size, color, brightness, transparency, shape, and design of the object.

[0128] Referring to operation 630, while the AI ​​assistant application is active, the electronic device (200) may receive user input. The user input may be user voice, but this is merely one example and may be various inputs such as text, images, gestures, etc. In particular, the user input may be referred to by various terms such as user query, prompt, etc.

[0129] Referring to operation 640, the electronic device (200) can determine whether the state of the electronic device (200) is locked.

[0130] If the state of the electronic device (200) is locked (640-Y), referring to operation 650, the electronic device (200) can identify a public database based on the locked state of the electronic device (200) and load it into a language model. A "Language Model (LM)" (hereinafter referred to as "LM") is a model that predicts the next word in a given text or calculates the probability of a sentence. Depending on the embodiment, the language model may be implemented as a massive language model (LLM). "Loading" may mean that the language model accesses the database so that it can obtain (or generate) response information based on information stored in the database. A public database is a database that stores public information (e.g., weather information, traffic information, news information, etc.) rather than the user's personal information, and may be stored in the electronic device (200), but this is merely one embodiment, and it is obvious that it may be stored on an external server. A public database may be a RAG database that records public information, but is not limited to this. Meanwhile, a public database may be called a first database.

[0131] When the state of the electronic device (200) is unlocked (640-N), referring to operation 660, the electronic device (200) can identify a private database associated with the usage state of the electronic device (200) and at least one of the user inputs and load it into a language model. The private database may be a database containing user information (e.g., personal information, usage information, etc.). Meanwhile, the private database may be referred to as a second database.

[0132] When user input is received while the electronic device (200) is in an unlocked state, the electronic device (200) can identify a private database corresponding to at least one application running in the background and load it into a language model. For example, the electronic device (200) can utilize all private databases corresponding to at least one application running in the background as input to the language model. Alternatively, the electronic device (200) can identify an application among at least one application that is related to the user input and load a private database corresponding to the application related to the user input into the language model. For example, if the user input is related to a video playback application, the electronic device (200) can identify a video playback application among at least one application and load a private database corresponding to the video playback application into the language model.

[0133] For example, when user input is received while the electronic device (200) is unlocked, the electronic device (200) can identify a private database in which personal information is stored and load it into a language model. The personal information may include user identification information, user search information, user shopping information, user schedule information, etc.

[0134] Meanwhile, according to one embodiment of the present disclosure, the private database may be a LORA (Low-Rank Adaptation) database learned based on user information or a RAG database recording user information, but is not limited thereto.

[0135] Referring to operation 670, the electronic device (200) can obtain response information for user input by inputting user input into a language model loaded with an identified database.

[0136] When the electronic device (200) is in a locked state, the electronic device (200) can obtain first response information by inputting user input into a language model loaded with a public database. When the electronic device (200) is in an unlocked state, the electronic device (200) can provide second response information by inputting user input into a language model loaded with a private database. The second response information is information different from the first response information and may be response information obtained based on information related to the electronic device (200) (e.g., user's personal information and usage information of the electronic device (200), etc.). Even if user input of the same question is received, the electronic device (200) can generate different response information depending on the current state of the electronic device (200).

[0137] Referring to operation 680, the electronic device (200) can output the acquired response information. The electronic device (200) can provide the acquired response information through a speaker or a display.

[0138] In FIG. 6, the neural network model providing response information is described as a language model, but this is merely one example and can be implemented as a generative AI model for content creation. For example, if the user input is a user input requesting content creation, the electronic device (200) identifies at least one of a plurality of databases based on the lock state of the electronic device and loads it into a generative AI model, and inputs the user input into the generative AI model in which the identified database is loaded to obtain generated content for the user input.

[0139] Hereinafter, various embodiments of the present disclosure will be described with reference to the drawings.

[0140] FIG. 7a is a diagram illustrating an embodiment of obtaining response information for user input using an LLM loaded with a public database when an electronic device is in a locked state, according to one embodiment of the present disclosure.

[0141] According to one or more embodiments, while the electronic device (200) maintains a locked state, the electronic device (200) may detect a trigger event (710). The trigger event (710) may be an event in which a user voice including a preset wake word (e.g., Bixby) is input, an event in which a physical button placed in a preset area of ​​the electronic device (200) is selected, an event in which a preset gesture is detected through a camera, or an event in which a preset touch input is received on a touch screen, but is not limited thereto.

[0142] The electronic device (200) can activate an AI assistant application (720). When the AI ​​assistant application (720) is activated, the electronic device (200) may provide an alarm message on the standby screen indicating that the AI ​​assistant application (720) has been activated. Alternatively, the electronic device (200) may provide at least one affordance object indicating that the AI ​​assistant application (720) has been activated, for example, the affordance object may be provided as an image, text, sound, screen effect, etc.

[0143] The electronic device (200) can receive user input (730). For example, as shown in FIG. 7b, the electronic device (200) can receive a user voice from the user (10) saying "Tell me about a good restaurant nearby."

[0144] The electronic device (200) can identify a public database (740) based on user input. The electronic device can identify a restaurant information database, which is a public database (740) associated with user input (730). The restaurant information database may be stored in the electronic device (200), but this is merely one example, and it is obvious that it may be stored on an external server.

[0145] And, the electronic device (200) can allow the public database to be loaded into the LLM (750).

[0146] The electronic device (200) can input user input (730) into an LLM (750) loaded with an identified public database (740). The LLM (750) may be stored within the electronic device (200), but this is merely one example, and it is obvious that it may be stored on an external server.

[0147] The electronic device (200) can obtain response information (760). The electronic device (200) can output the response information (760) through an output device (e.g., a display (230) or a speaker). For example, as shown in FIG. 7b, the electronic device (200) can output a response message (770) containing response information (760) through the display (230) such as “Near the current user, xxx restaurant and yyy restaurant have high ratings.”

[0148] In one or more embodiments, if a public database is stored in an electronic device (200), the electronic device (200) may periodically update the public database to maintain the latest information. In one or more embodiments, the electronic device (200) may update the information stored in the public database prior to user input based on the current location information of the electronic device (200).

[0149] In the above-described embodiment, it was explained that the electronic device (200) stores the public database (740), but this is merely one embodiment, and the public database (740) may not be stored in the electronic device (200). If the public database (740) is not stored in the electronic device (200), the electronic device (200) can obtain information related to user input by using an external search (e.g., web search, etc.). The electronic device (200) can obtain response information by inputting the information related to user input into the LLM (750) along with the user input. For example, if a search result related to user input is obtained through a web search, such as “The restaurant found at the user’s current location “XX” is YY Restaurant, and the visitor rating is high,” the electronic device (200) can obtain response information (760) by inputting the search result into the LLM (750) as a prompt along with the user input (730).

[0150] FIG. 8a is a diagram illustrating an embodiment of obtaining response information for user input using an LLM loaded with a private database when an electronic device is in an unlocked state, according to one embodiment of the present disclosure.

[0151] According to one or more embodiments, while the electronic device (200) maintains an unlocked state, the electronic device (200) may detect a trigger event (810). The trigger event (810) may be an event in which a user voice including a preset wake word (e.g., Bixby) is input, an event in which a physical button placed in a preset area of ​​the electronic device (200) is selected, an event in which a preset gesture is detected through a camera, or an event in which a preset touch input is received on a touch screen, but is not limited thereto.

[0152] The electronic device (200) can activate an AI assistant application (820). When the AI ​​assistant application (820) is activated, the electronic device (200) may provide an alarm message on a screen (or application execution screen) containing multiple application execution icons indicating that the AI ​​assistant application (820) has been activated.

[0153] The electronic device (200) can receive user input (830). For example, as shown in FIG. 8c, the electronic device (200) can receive a user voice from the user (10) saying "Tell me about a good restaurant nearby."

[0154] The electronic device (200) can identify at least one private database (840-1) and a public database (840-2). The electronic device (200) can identify at least one private database (840-1) based on the user's (10) personal information or the usage information of the electronic device (200). The electronic device (200) can identify at least one of a first private database that stores the user's personal information, and a second and third private database corresponding to the first and second applications currently running in the background. The electronic device (200) can load at least one identified private database into the LLM (850). The private database may be a database created based on information collected from the corresponding application, and may be one of a RAG database that stores information collected from the application or a LORA database that stores the results of learning information collected from multiple applications.

[0155] The electronic device (200) can identify a public database (840-2) based on user input. For example, the electronic device can identify a restaurant information database, which is a public database (840-2) associated with user input (830). The restaurant information database may be stored in the electronic device (200), but this is merely one embodiment, and it is obvious that it may be stored on an external server.

[0156] According to one embodiment, an electronic device (200) can input user input (830) into an LLM (850) loaded with at least one private database (840-1). The LLM (850) may be stored within the electronic device (200), but this is merely one embodiment, and it is obvious that it may be stored on an external server. The electronic device (200) can perform a search on the private database and transmit the information obtained by the search to the LLM (850) along with the user input (830). For example, if information such as “a restaurant visited on June 6, 2023, and left a review saying it was delicious” is found through at least one private database (840-1), the electronic device (200) can transmit the searched information and the user input (830) to the LLM (850).

[0157] According to one embodiment, the electronic device (200) can obtain response information (860). The electronic device (200) can output the response information (860) through an output device (e.g., a display (230) or a speaker). For example, as illustrated in FIG. 8c, if information about a restaurant that the user has previously visited or is interested in is obtained through at least one private database (840-1), the electronic device (200) can output a user response (880) through the display (230) saying, "There is a restaurant called yyy that you visited last year near the user. How about the cold noodles sold at the yyy restaurant on a day like today?"

[0158] According to one embodiment, as shown in FIG. 7b, a general response is provided in the locked state, but as shown in FIG. 8c, a response reflecting user information rather than a general response is provided in the unlocked state, thereby enabling the user to have a variety of user experiences.

[0159] According to one embodiment, FIG. 8a describes that a single private database (840-1) is loaded, but this is merely one embodiment, and multiple private databases may be loaded. The electronic device (200) may load all of the multiple private databases, but this is merely one embodiment, and an application related to user input may be selected, and among the multiple private databases, the private database corresponding to the selected application may be loaded into the LLM. This will be explained with reference to FIG. 8b.

[0160] FIG. 8b is a diagram illustrating an embodiment according to another embodiment of the present disclosure in which response information for user input is obtained using an LLM loaded with a private database when an electronic device is in an unlocked state. Duplicate descriptions in FIG. 8a are omitted.

[0161] According to one embodiment, while the electronic device (200) maintains an unlocked state, the electronic device (200) can detect a trigger event (810). When in an unlocked state, the electronic device (200) may grant permission to access the user information stored in the electronic device (200) before receiving user input, or load a Low-Rank Adaptation (LORA) database learned based on user information or a RAG database recording user information, etc., so that it can be accessed in an LLM model. The electronic device (200) can activate an AI assistant application (820).

[0162] The electronic device (200) can receive user input (830) (e.g., user input related to sound source search).

[0163] The electronic device (200) can select (840) one of the applications among the multiple private databases (840-1, 840-2, 840-3). The electronic device (200) can identify the application related to user input among the multiple applications running in the background. For example, if the user input (830) is user input related to sound source search, the electronic device (200) can identify the application related to sound source search. Then, the electronic device (200) can select the second private database (840-2) corresponding to the application related to sound source search among the multiple private databases (840-1, 840-2, 840-3). The electronic device (200) can load the second private database (840-2) among the multiple private databases (840-1, 840-2, 840-3) into the LLM (850).

[0164] The electronic device (200) can input user input (830) into the LLM (850) loaded with the second private database (840-2).

[0165] The electronic device (200) can obtain response information (860). The electronic device (200) can output the response information (860) through an output device (e.g., a display (230) or a speaker). For example, the response information (860) may be a response obtained based on a database in which a user's audio playlist is stored.

[0166] FIG. 9 is a diagram illustrating an embodiment according to another embodiment of the present disclosure in which response information for user input is obtained using an LLM loaded with a private database when an electronic device is in an unlocked state. Duplicate descriptions in FIG. 8b are omitted.

[0167] According to one or more embodiments, while the electronic device (200) maintains an unlocked state, the electronic device (200) may detect a trigger event (910). The electronic device (200) may maintain a locked state. And, as illustrated in FIG. 9, a calendar application, an SMS application, and a video application may be running in the background. And, the electronic device (200) may store a first private database (950-1) corresponding to the calendar application, a second private database (950-2) corresponding to the SMS application, and a third private database (950-3) corresponding to the video application. The first private database (950-1) may store information about the user's schedule, the second private database (950-2) may store information about conversations and content exchanged with other users, and the third private database (950-3) may store information about video content watched by the user.

[0168] The electronic device (200) can activate the AI ​​assistant application (920).

[0169] The electronic device (200) can receive user input (930) (e.g., user input related to sound source search).

[0170] According to one embodiment, the electronic device (200) can select at least one private database among a plurality of private databases (950-1, 950-2, 950-3) using a database selector (940). For example, the electronic device (200) can identify an application related to user input among a plurality of applications running in the background. The electronic device (200) can select a private database corresponding to the application identified through the database selector (940). For example, if the user input (830) is a user input related to a query about a previous user conversation (e.g., "Who sent the message last night?"), the electronic device (200) can identify an SMS application related to the "message" included in the user input. Then, the electronic device (200) can select a second private database (950-2) corresponding to the SMS application among a plurality of private databases (950-1, 950-2, 950-3). And, the electronic device (200) can load the second private database (950-2) among the multiple private databases (950-1, 950-2, 950-3) into the LLM (960).

[0171] The electronic device (200) can receive user input (930) into the LLM (960) loaded with the second private database (950-2).

[0172] According to one embodiment, the electronic device (200) can obtain response information (970). The electronic device (200) can output the response information (970) through an output device (e.g., a display (230) or a speaker). For example, the response information (970) can provide response information such as "The person who sent the message last night is XXX. It was a message asking for amusement park usage tips." which is information obtained through an SMS database.

[0173] Various embodiments of the present invention will be described below.

[0174] FIG. 10a is a diagram illustrating an embodiment of obtaining response information for user input while an electronic device maintains a locked state, according to one embodiment of the present disclosure. In this embodiment, the electronic device (200) may be in a locked state and the AI ​​assistant application may be activated.

[0175] As illustrated in FIG. 10a, for example, the electronic device (200) can receive user voice (1010) such as "Tell me the weather tomorrow" spoken by the user.

[0176] According to one embodiment, the electronic device (200) can load a RAG database (1040) that stores real-time weather information, which is a public database, into an LLM (Foundation model) (1030).

[0177] The electronic device (200) can input a user voice (1010) (or text corresponding to the user voice (1010)) into an LLM (1030) loaded with a RAG database (1040) to obtain response information (1020), such as "The weather on xx tomorrow is clear," which is a user response. Then, the electronic device (200) can display the response information (1020), such as "The weather on xx tomorrow is clear," on a display (230) or output it through a speaker.

[0178] FIG. 10b is a diagram illustrating an embodiment of obtaining response information to user input while an electronic device maintains an unlocked state according to one embodiment of the present disclosure. In this embodiment, the electronic device (200) may have an AI assistant application activated while in an unlocked state.

[0179] As illustrated in FIG. 10b, the electronic device (200) can receive user voice (1010), such as "Tell me the weather tomorrow," spoken by the user.

[0180] According to one embodiment, the electronic device (200) can load a RAG database (1060-1) that stores real-time weather information, which is a public database, and a RAG database (1060-2) that corresponds to a schedule application related to user voice, which is a private database, into an LLM (Foundation model) (1030).

[0181] According to one embodiment, the electronic device (200) inputs a user voice (1010) (or text corresponding to the user voice (1010)) into an LLM (1030) loaded with a RAG database (1060-1) and a private database (1060-2) corresponding to a schedule application, and obtains response information (1050) such as a user response, "The weather in xx tomorrow is clear. However, there is expected to be a shower at the place you are going on vacation tomorrow, so please bring an umbrella." Then, the electronic device (200) can display the response information (1050), "The weather in xx tomorrow is clear. However, there is expected to be a shower at the place you are going on vacation tomorrow, so please bring an umbrella," on a display (230) or output it through a speaker.

[0182] According to one embodiment, as illustrated in FIG. 10a and FIG. 10b, the electronic device (200) can provide different responses to the same user voice (1010) depending on the unlocked state of the electronic device (200).

[0183] FIG. 11a is a diagram illustrating an embodiment according to one embodiment of the present disclosure in which an electronic device obtains response information for a plurality of types of user inputs (or multimodal inputs) while maintaining a locked state. In this embodiment, the electronic device (200) may be in a locked state and an AI assistant application may be activated.

[0184] As illustrated in FIG. 11a, according to one embodiment, the electronic device (200) can receive user voice (1120), such as “Who is this person?” spoken by the user while the first person image (1110) is displayed.

[0185] According to one embodiment, the electronic device (200) can load a public database (1150) that stores information about a general person, which is a public database, into the LLM (1140).

[0186] According to one embodiment, the electronic device (200) inputs a first person image (1110) and a user voice (1120) (or text corresponding to the user voice (1120)) into an LLM (1140) loaded with a public database (1150) to obtain response information (1130), such as a user response, "The person shown now is an actor who appeared in xxx." According to one embodiment, the electronic device (200) can display the response information (1130), "The person shown now is an actor who appeared in xxx," on a display (230) or output it through a speaker.

[0187] FIG. 11b is a diagram illustrating an embodiment of obtaining response information for user input (or multimodal input) while an electronic device maintains an unlocked state according to one embodiment of the present disclosure. In this embodiment, the electronic device (200) may have an AI assistant application activated while in an unlocked state.

[0188] As illustrated in FIG. 11a, the electronic device (200) can receive user voice (1120), such as “Who is this person?” spoken by the user while the first person image (1110) is displayed.

[0189] The electronic device (200) can load a public database (1170-1) that stores information about a general person, which is a public database, and a private database (1170-2) corresponding to a gallery application into the LLM (1140).

[0190] According to one embodiment, the electronic device (200) inputs a first person image (1110) and a user voice (1120) (or text corresponding to the user voice (1120)) into an LLM (1140) loaded with a public database (1170-1) and a private database (1170-2) corresponding to a gallery application, thereby obtaining response information (1130), such as a user response, "The person shown now is a father." Then, the electronic device (200) can display the response information (1160), "The person shown now is a father," on a display (230) or output it through a speaker.

[0191] FIG. 12a is a drawing illustrating an embodiment of generating content for user input when an electronic device is in a locked state, according to one embodiment of the present disclosure. In this embodiment, the electronic device (200) may be in a locked state and the AI ​​assistant application may be activated.

[0192] As illustrated in FIG. 12a, the electronic device (200) can receive user voice (1210), such as “draw a baby,” spoken by the user.

[0193] The electronic device (200) can load a public database (1230) that stores general baby photos into a generative AI model (1220).

[0194] According to one embodiment, an electronic device (200) can acquire (or generate) a first baby photo (1240) as illustrated in FIG. 12a by inputting a user voice (1210) (or text corresponding to the user voice (1210)) into a generative AI model (1220) loaded with a public database (1230).

[0195] FIG. 12b is a drawing illustrating an embodiment of generating content for user input while an electronic device maintains an unlocked state according to one embodiment of the present disclosure. In this embodiment, the electronic device (200) may have an AI assistant application activated while in an unlocked state.

[0196] As illustrated in FIG. 12a, the electronic device (200) can receive user voice (1210), such as “draw a baby,” spoken by the user.

[0197] The electronic device (200) can load a private database (1250) corresponding to a gallery application into a generative AI model (1220).

[0198] According to one embodiment, the electronic device (200) can acquire (or generate) a second baby photo (1260) as illustrated in FIG. 12b by inputting a user voice (1210) (or text corresponding to the user voice (1210)) into a generative AI model (1220) loaded with a private database (1250).

[0199] As illustrated in FIGS. 12a and 12b, the electronic device (200) can generate and provide different content depending on the unlocked state of the electronic device (200) even for the same user voice (1210).

[0200] Depending on whether the electronic device (200) is in a locked state, the electronic device (200) may generate a general response or provide a user-specific response generated by the LLM. When the electronic device (200) generates a response based on a user's query, it may provide response information using response parameters related to a first persona when the electronic device (200) is in a locked state, and provide a user response using response parameters related to a second persona when answering the query when the electronic device (200) is unlocked. Personas may be referred to by other terms such as virtual user, character, user type, or user model.

[0201] FIG. 13a is a diagram illustrating an embodiment of generating content for user input using response parameters corresponding to a first persona when an electronic device is in a locked state, according to one embodiment of the present disclosure. In this embodiment, the electronic device (200) may be in a locked state and the AI ​​assistant application may be activated.

[0202] As illustrated in FIG. 13a, according to one embodiment, the electronic device (200) can receive user voice (1320), such as “tell me a funny story,” spoken by the user while the lock screen (1310) is displayed.

[0203] According to one embodiment, the electronic device (200) can load a database (1350) that stores response parameters corresponding to a first persona (e.g., General persona) into the LLM (1340).

[0204] According to one embodiment, an electronic device (200) may obtain first response information (1330) by inputting a user voice (1320) (or text corresponding to the user voice (1320)) into an LLM (1340) loaded with a database that stores response parameters corresponding to a first persona. According to one embodiment, the electronic device (200) may provide the first response information (1330), which includes content generated based on the user voice (1320), in the form of a general persona. The first response information (1330) may be displayed on a display (230) or output through a speaker.

[0205] FIG. 13b is a diagram illustrating an embodiment of generating content for user input using response parameters corresponding to a second persona while an electronic device maintains an unlocked state according to one embodiment of the present disclosure. In this embodiment, the electronic device (200) may be in a locked state and the AI ​​assistant application may be activated.

[0206] As illustrated in FIG. 13b, according to one embodiment, the electronic device (200) can receive user voice (1320), such as "tell me a funny story," spoken by the user while a general background screen (1360) is displayed.

[0207] According to one embodiment, the electronic device (200) can load a database (1380) that stores response parameters corresponding to a second persona (e.g., Custom persona) into the LLM (1340).

[0208] According to one embodiment, an electronic device (200) may obtain second response information (1370) by inputting a user voice (1320) (or text corresponding to the user voice (1320)) into an LLM (1340) loaded with a database that stores response parameters corresponding to a second persona. According to one embodiment, the electronic device (200) may provide the second response information (1370), which includes content generated based on the user voice (1320), in the form of a persona specialized for the user. The second response information (1730) may be displayed on a display (230) or output through a speaker.

[0209] For example, the first persona may be a pre-designated public persona, and the second persona may be a persona set by the user, a user-specific persona frequently used by the user, or a persona preferred by the user, but is not limited thereto.

[0210] In the above-described embodiment, it was explained that different databases are loaded into the LLM depending on the lock status of the electronic device (200); however, this is merely one embodiment, and different databases can be loaded into the LLM depending on status information such as whether the electronic device (200) is authenticated or whether the electronic device (200) is worn. For example, if user authentication for the electronic device (200) is not performed, the electronic device (200) can load a public database into the LLM, but if user authentication for the electronic device (200) is performed, the electronic device (200) can load a private database into the LLM. As another example, if the user is not wearing the electronic device (200) (or is placed in a cradle), the electronic device (200) can load a public database into the LLM, but if the user is wearing the electronic device (200), the electronic device (100) can load a private database into the LLM.

[0211]

[0212] Various embodiments of this document may be implemented as software comprising one or more instructions stored in a storage medium (e.g., memory (620)) readable by a machine (e.g., electronic device (200)). For example, a processor (e.g., processor (210)) of the machine (e.g., electronic device (200)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. , 'non-transient' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0213] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). ™It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0214] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0215] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

In electronic devices, Memory for storing instructions; and Includes a processor; When the above instructions are executed individually or collectively by the processor, the electronic device, When a trigger event is detected, activate the AI ​​(Artificial Intelligence) assistant application, and When user input is received while the above-mentioned assistant application is active, at least one of a plurality of databases is identified based on the lock state of the electronic device and loaded into a language model. The user input is input into a language model loaded with the identified database to obtain response information regarding the user input, and An electronic device that outputs the above-mentioned acquired response information. In paragraph 1, When the above instructions are executed individually or collectively by the processor, the electronic device, When user input is received while the electronic device is locked, it identifies a public database and loads it into the language model, and An electronic device that obtains first response information by inputting the user input into a language model loaded with the above public database. In paragraph 1, When the above instructions are executed individually or collectively by the processor, the electronic device, When user input is received while the electronic device is unlocked, a private database associated with at least one of the usage state of the electronic device and the user input is identified and loaded into the language model, and An electronic device that obtains second response information by inputting the user input into a language model loaded with the private database. In paragraph 3, When the above instructions are executed individually or collectively by the processor, the electronic device, When user input is received while the electronic device is unlocked, it identifies a private database corresponding to at least one application running in the background and loads it into the language model, and An electronic device that obtains the second response information by inputting the user input into a language model loaded with a private database corresponding to at least one application. In paragraph 3, When the above instructions are executed individually or collectively by the processor, the electronic device, When the electronic device is unlocked and receives user input, it identifies the application associated with the user input among the at least one application, and An electronic device that loads a private database corresponding to an application related to the above user input into the language model. In paragraph 3, When the above instructions are executed individually or collectively by the processor, the electronic device, When user input is received while the electronic device is unlocked, it identifies a private database storing personal information and loads it into the language model, and An electronic device that obtains the second response information by inputting the user input into a language model loaded with a private database in which the above personal information is stored. In paragraph 1, When the above instructions are executed individually or collectively by the processor, the electronic device, If the above user input is a user input for requesting content creation, at least one of a plurality of databases is identified based on the lock state of the electronic device and loaded into a generative AI model, and An electronic device that inputs the user input into a generative AI model loaded with the identified database to obtain generated content for the user input. In paragraph 1, The above trigger event is, An electronic device that is one of the following events: an event in which a user voice containing a preset wake word is input, an event in which a preset button is selected, an event in which a preset gesture is detected, and an event in which a preset touch input is received on a touch screen. In paragraph 1, When the above instructions are executed individually or collectively by the processor, the electronic device, When the above AI assistant application is activated, an alarm message is provided to notify that the above AI assistant application is activated, and The above alarm message is an electronic device comprising at least one of an audio-based alarm message and a visual-based alarm message. In a method for controlling an electronic device, A step of activating an AI (Artificial Intelligence) assistant application when a trigger event is detected; When user input is received while the above-mentioned assistant application is active, a step of identifying at least one of a plurality of databases based on the lock state of the electronic device and loading it into a language model; A step of inputting the user input into a language model loaded with the identified database to obtain response information for the user input; and A control method comprising the step of outputting the above-mentioned acquired response information. In Paragraph 10, The above loading step is, When user input is received while the electronic device is locked, it identifies a public database and loads it into the language model, and The above-mentioned acquisition step is, A control method for obtaining first response information by inputting the user input into a language model loaded with the above public database. In Paragraph 10, The above loading step is, When user input is received while the electronic device is unlocked, a private database associated with at least one of the usage state of the electronic device and the user input is identified and loaded into the language model, and The above-mentioned acquisition step is, A control method for obtaining second response information by inputting the user input into a language model loaded with the private database. In Paragraph 12, The above loading step is, When user input is received while the electronic device is unlocked, it identifies a private database corresponding to at least one application running in the background and loads it into the language model, and The above-mentioned acquisition step is, A control method for obtaining the second response information by inputting the user input into a language model loaded with a private database corresponding to at least one application. In Paragraph 12, The above loading step is, When the electronic device is unlocked and receives user input, the step of identifying an application among the at least one application associated with the user input; and A control method comprising the step of loading a private database corresponding to an application related to the above user input into the language model. In Paragraph 12, The above loading step is, When user input is received while the electronic device is unlocked, it identifies a private database storing personal information and loads it into the language model, and The above-mentioned acquisition step is, A control method for obtaining the second response information by inputting the user input into a language model loaded with a private database in which the above personal information is stored.

Images