Electronic device, method, and non-transitory computer-readable storage medium for generating routine

Abstract

This electronic device comprises a memory storing instructions, and at least one processor comprising processing circuitry, wherein the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: when a user request comprising a natural language sentence is received, identify a condition and an action included in the natural language sentence; acquire first data corresponding to the condition and second data corresponding to the action; acquire routine generation data on the basis of the first data, the second data, and information about the electronic device; and on the basis of the routine generation data, generate a routine causing the electronic device to automatically execute a specific action when information identified by the electronic device corresponds to a specific condition.

Description

Electronic device, method, and non-transient computer-readable storage medium for generating a routine

[0001] The present disclosure relates to an electronic device, a method, and a non-transient computer-readable storage medium for generating a routine.

[0002] Recently, the distribution of various types of portable electronic devices, such as smartphones, tablet PCs, wireless earphones, and smartwatches, is expanding.

[0003] Recently, electronic devices provide a wide range of functions and services, and accordingly, there is a growing demand for features that allow users to directly automate the functions they desire. One such feature is a routine, and it is important to enable users to create and use them directly. However, existing methods for creating routines required users to write code themselves or required complex settings.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] According to one embodiment, the electronic device comprises: a memory for storing instructions; and at least one processor including processing circuitry; wherein, when the instructions are executed individually or collectively by the at least one processor, the electronic device, upon receiving a user request including a natural language sentence, identifies a condition and an action included in the natural language sentence, obtains first data corresponding to the condition and second data corresponding to the action, obtains routine generation data based on the first data, the second data and information about the electronic device, and generates a routine that causes the electronic device to automatically execute a specific action if the information identified by the electronic device based on the routine generation data corresponds to a specific condition.

[0006] According to one embodiment, a control method for an electronic device comprises: an operation of identifying conditions and operations included in the natural language sentence when a user request including a natural language sentence is received; an operation of obtaining first data corresponding to the condition and second data corresponding to the operation; an operation of obtaining routine generation data based on the first data, the second data and information about the electronic device; and an operation of generating a routine that causes the electronic device to automatically execute a specific operation when information identified in the electronic device based on the routine generation data corresponds to a specific condition.

[0007] According to one embodiment, in a non-transient computer-readable medium storing instructions that cause the electronic device to perform an operation when executed by a processor of the electronic device, the operation comprises: an operation of identifying a condition and an operation included in the natural language sentence when a user request including a natural language sentence is received; an operation of obtaining first data corresponding to the condition and second data corresponding to the operation; an operation of obtaining routine generation data based on the first data, the second data and information about the electronic device; and an operation of generating a routine that causes the electronic device to automatically execute a specific operation when information identified in the electronic device based on the routine generation data corresponds to a specific condition.

[0008] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken together with the accompanying drawings.

[0009] FIGS. 1a and FIGS. 1b are drawings for schematically illustrating the operation of an electronic device according to one embodiment.

[0010] FIG. 2 illustrates an example of a block diagram of an electronic device according to one embodiment.

[0011] FIG. 3 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0012] FIG. 4 is a diagram illustrating a generative artificial intelligence model according to one embodiment.

[0013] FIGS. 5A and FIGS. 5B are drawings for explaining a method for receiving a user request according to one embodiment.

[0014] FIG. 6 is a diagram illustrating an example of a routine generation method according to one embodiment.

[0015] FIG. 7 is a drawing for illustrating an example of a first prompt and a response to the first prompt according to one embodiment.

[0016] FIG. 8 is a drawing for illustrating an example of a first prompt and a response to the first prompt according to one embodiment.

[0017] FIG. 9 is a drawing for illustrating an example of a first prompt and a response to the first prompt according to one embodiment.

[0018] FIG. 10 is a drawing for illustrating an example of a first prompt and a response to the first prompt according to one embodiment.

[0019] FIG. 11 is a diagram illustrating the schema of a condition vector database and an action vector database according to one embodiment.

[0020] FIGS. 12a and FIGS. 12b are drawings for illustrating examples of vector data according to one embodiment.

[0021] FIGS. 13a and FIGS. 13b are drawings for illustrating examples of vector data related to an operation according to one example.

[0022] FIG. 14 is a diagram illustrating an example of routine generation data according to one embodiment.

[0023] FIGS. 15a and FIGS. 15b are drawings for illustrating examples of routine generation data according to one embodiment.

[0024] FIG. 16 is a drawing for explaining an example of a UI provision method according to one embodiment.

[0025] FIGS. 17 and FIGS. 18 are drawings for explaining an example of a method for providing a UI according to one embodiment.

[0026] FIG. 19 is a diagram illustrating an example of a scenario for generating a routine according to one embodiment.

[0027] FIG. 20 is a block diagram of an electronic device in a network environment according to various embodiments.

[0028] The present disclosure will be described in detail below with reference to the attached drawings.

[0029] The terms used in the embodiments of this disclosure have been selected to be as widely used and general as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, or the emergence of new technologies. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the description section of the disclosure. Therefore, terms used in this disclosure should be defined based on their meanings and the overall content of this disclosure, rather than merely their names (e.g., call, message, analyzed schedule).

[0030] In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of the above features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0031] The expression "at least one of A and / or B" should be understood as representing either "A" or "B" or "A and B".

[0032] Expressions such as "first," "second," "first," or "second" used in this specification may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0033] Where it is stated that a component (e.g., a first component) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., a second component), it should be understood that the component may be directly connected to the other component or connected through the other component (e.g., a third component).

[0034] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as “comprising” or “consisting” are intended to specify the existence of the features, numbers, actions, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, actions, actions, components, parts, or combinations thereof.

[0035] In the embodiments, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor, except for a "module" or "part" that needs to be implemented in specific hardware.

[0036] In the present disclosure, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0037] The various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0038] Embodiments of the present disclosure will be described in more detail below with reference to the attached drawings.

[0039] FIGS. 1a and FIGS. 1b are drawings for schematically illustrating the operation of an electronic device according to one embodiment.

[0040] Figure 1a is a diagram illustrating the operation of generating a routine according to one example.

[0041] According to FIG. 1a, in order to generate a routine in an electronic device (100), the user must manually search for and select conditions and actions through a UI screen (10) provided by a routine generation application and input detailed condition parameter values. This may be a complex and difficult process for a general user.

[0042] FIG. 1b is a diagram illustrating the operation of generating a routine according to one example.

[0043] According to FIG. 1b, to generate a routine in the electronic device (100), a user may input natural language commands through a UI (20) for receiving voice and / or text input. For example, natural language commands may include natural language sentences composed of language that people use in their daily lives, such as “Turn on Bluetooth and Wi-Fi when I get in the car at 6 a.m..”

[0044] According to one example, when an electronic device (100) receives a user's request to generate a routine containing a natural language sentence, it can generate a routine based on technologies such as Artificial Intelligence (AI), Natural Language Processing (NLP), and / or Deep Learning. Artificial Intelligence is a technology that enables a machine to think and make judgments like a human, and can identify the user's intent through natural language commands and process it appropriately. Natural Language Processing can convert natural language commands entered by the user into a form that the computer can understand. For example, Natural Language Processing can analyze text data to extract meaning and perform various tasks such as syntactic analysis, named entity recognition, and sentiment analysis. Deep Learning is a technology that uses artificial neural networks to learn large amounts of data and identifies complex patterns based on this, and can be used for more precise natural language processing.

[0045] Hereinafter, various embodiments of generating routines through natural language commands using artificial intelligence, natural language processing, and / or deep learning technologies will be described.

[0046] FIG. 2 illustrates an example of a block diagram of an electronic device according to one embodiment.

[0047] According to various embodiments, the electronic device (100) of FIG. 2 may be at least partially similar to the electronic device (2401) of FIG. 20, or may include other embodiments of the electronic device.

[0048] In one embodiment, in terms of being owned by a user, the electronic device (100) may be referred to as a terminal (or user terminal). The terminal may include, for example, a personal computer (PC) such as a laptop and a desktop. The terminal may include, for example, a smartphone, a smartpad, and / or a tablet PC. The terminal may include smart accessories such as a smartwatch and / or a head-mounted device (HMD). According to one embodiment, the electronic device (100) may include a deformable housing. Based on the deformability, the housing of the electronic device (100) may be divided into a plurality of parts. According to one example, the electronic device (100) may be implemented as a user terminal (40) illustrated in FIG. 1.

[0049] According to one embodiment, the electronic device (100) may include at least one of a processor (110), a memory (120), a display (130), a communication circuit (140), a camera (150), a sensor (160), or a microphone (170). The processor (110), memory (120), display (130), communication circuit (140), camera (150), sensor (160), or microphone (170) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus.

[0050] In one embodiment, the hardware of the electronic device (100) being operatively coupled may mean that a direct or indirect connection between the hardware is established via wired or wireless means so that the second hardware is controlled by the first hardware among the hardware. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the hardware of FIG. 2 (e.g., at least some of the processor (110), memory (120), and communication circuit (140)) may be included in a single integrated circuit, such as a system on a chip (SoC). The type and / or number of hardware included in the electronic device (100) is not limited to that shown in FIG. 2. For example, the electronic device (100) may include only some of the hardware components shown in FIG. 2.

[0051] According to one embodiment, the processor (110) of the electronic device (100) may include hardware for processing data based on one or more instructions. The hardware for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), a graphic processing unit (GPU), a neural processing unit (NPU), and / or an application processor (AP). The number of processors (110) may be one or more. For example, the processor (110) may have the structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

[0052] The processor (110) can control the operations of the electronic device (100) by executing instructions stored in the memory (120). For example, the processor (110) may correspond to a plurality of processors that divide and collectively perform a plurality of operations among the processors.

[0053] A CPU (central processing unit) is a general-purpose processor capable of performing not only general operations but also artificial intelligence operations, and it can efficiently execute complex programs through a multi-layered cache structure. The CPU is advantageous for serial processing methods, which enable the organic linkage between previous and next calculation results through sequential computation. General-purpose processors are not limited to the examples mentioned above, except for cases specified as the aforementioned CPU.

[0054] A GPU (graphic processing unit) is a processor designed for massive computations, such as floating-point operations used in graphics processing, and can perform large-scale computations in parallel by integrating a large number of cores. In particular, GPUs may be advantageous over CPUs for parallel processing methods such as convolution operations. Additionally, GPUs can be used as co-processors to complement the functions of CPUs. Processors for massive computation are not limited to the examples mentioned above, except for cases specified as GPUs.

[0055] A Neural Processing Unit (NPU) is a processor specialized for artificial intelligence computations using artificial neural networks, and each layer constituting the neural network can be implemented in hardware (e.g., silicon). In this case, since the NPU is designed specifically according to the specifications required by the vendor, it has a lower degree of flexibility compared to CPUs or GPUs, but it can efficiently process the artificial intelligence computations required by the vendor. Meanwhile, as a processor specialized for artificial intelligence computations, the NPU can be implemented in various forms such as Tensor Processing Units (TPUs), Intelligence Processing Units (IPUs), and Vision Processing Units (VPUs). Artificial intelligence processors are not limited to the examples mentioned above, except for cases specified as the aforementioned NPU.

[0056] According to one embodiment, the memory (120) of the electronic device (100) may include a hardware component for storing data and / or instructions that are input and / or output to the processor (110). The memory (120) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, solid status drive (SSD), or embedded multimedia card (eMMC).

[0057] According to one embodiment, within the memory (120) of the electronic device (100), one or more instructions (or commands) representing operations and / or operations to be performed on data by the processor (110) may be stored. A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the electronic device (100) and / or the processor (110) may perform various operations when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. In the following, the statement that an application is installed on an electronic device (100) means that one or more instructions provided in the form of an application are stored in the memory (120) of the electronic device (100), and that the one or more applications are stored in an executable format (e.g., a file having an extension specified by the operating system of the electronic device (100)) that is executable by the processor (110) of the electronic device (100).

[0058] At least one processor (110) controls the processing of input data according to a predefined operation rule or AI model (artificial-intelligence model) stored in memory (120). The predefined operation rule or AI model is characterized by being created through learning. Being created through learning means that a predefined operation rule or AI model with desired characteristics is created by applying a learning algorithm to a number of learning data. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is performed, or it may be performed through a separate server / system.

[0059] An AI model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs the layer's operation through the result of the operation of the previous layer and at least one defined operation. Examples of neural networks include convolutional neural networks (CNN), recurrent neural networks (RNN), deep neural networks (DNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN), deep Q-networks, and Transformers; however, the neural networks in this disclosure are not limited to the aforementioned examples except where specified.

[0060] A learning algorithm is a method of training a specific target device (e.g., a robot) using a number of learning data to enable the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithms in this disclosure are not limited to the aforementioned examples except where specified.

[0061] According to one embodiment, a display (130) of an electronic device (100) can output visualized information to a user. For example, the display (130) can be controlled by a controller, such as a GPU (graphic processing unit), to output visualized information to a user. The display (130) may include an OLED (organic light emitting diodes) display, an LED (light emitting diodes), a micro LED, a mini LED, a PDP (plasma display panel), a QD (quantum dot) display, a QLED (quantum dot light-emitting diodes) and / or an e-ink display and / or an e-paper display. According to one example, the display (130) may be implemented as a flat display, a curved display, a folding and / or rolling flexible display.

[0062] A communication circuit (140) of an electronic device (100) according to one embodiment may include hardware for supporting the transmission and / or reception of electrical signals between the electronic device (100) and an external device (e.g., a server). The communication circuit (140) may include, for example, at least one of a modem (modulator and demodulator), an antenna, and an optic / electronic converter. The communication circuit (140) may support the transmission and / or reception of electrical signals based on various types of protocols such as Ethernet, LAN (local area network), WAN (wide area network), WiFi (wireless fidelity), NFC (near field communication), Bluetooth, BLE (bluetooth low energy), ZigBee, LTE (long term evolution), 5G NR (new radio), and / or 6G.

[0063] According to one example, the electronic device (100) may be connected to a server and to each other based on a wired network and / or a wireless network. The wired network may include a network such as the Internet, a LAN (local area network), a WAN (wide area network), Ethernet, or a combination thereof. The wireless network may include a network such as LTE (long term evolution), 5G NR (new radio), WiFi (wireless fidelity), Zigbee, NFC (near field communication), Bluetooth, BLE (bluetooth low-energy), or a combination thereof. According to one example, the electronic device (100) and the server may be connected indirectly through an intermediate node within the network.

[0064] A camera (150) of an electronic device (100) according to one embodiment can convert a captured image into an electrical signal and generate image data based on the converted signal. For example, the camera (150) may include at least one of a general (or basic) camera, a depth camera, and an ultra-wide angle camera.

[0065] A sensor (160) of an electronic device (100) according to one embodiment can sense various user information. The sensor (160) can be implemented as various types of sensors capable of user sensing. For example, the sensor (160) may include at least one sensor among a time of flight (ToF) sensor, an ultrasonic sensor, a radio detection and ranging (RADAR) sensor, a photodiode sensor, a proximity sensor, a passive infrared (PIR) sensor, a pinhole sensor, a pinhole camera, an infrared human body detection sensor, a complementary metal oxide semiconductor (CMOS) image sensor, a thermal sensor, a light sensor, and a motion sensor.

[0066] The sensor (160) may include a touch sensor that detects touch actions, having a form such as a touch film, a touch sheet, or a touch pad.

[0067] The sensor (160) may include at least one of a CO2 sensor and an atmospheric pressure sensor. The CO2 sensor is a sensor for measuring carbon dioxide concentration. The atmospheric pressure sensor is a sensor for sensing ambient pressure.

[0068] The sensor (160) may further include at least one sensor capable of sensing ambient illuminance, ambient temperature, and the direction of incidence of light. In this case, the sensor (160) may be implemented as an illuminance sensor, a temperature sensing sensor, a light intensity sensing layer, and a camera.

[0069] The sensor (160) may further include at least one of an acceleration sensor (or gravity sensor), a geomagnetic sensor, and a gyro sensor. For example, the acceleration sensor may be a 3-axis acceleration sensor. The 3-axis acceleration sensor may measure gravitational acceleration by axis and provide raw data to the processor (140). The geomagnetic sensor or the gyro sensor may be used to obtain attitude information. Here, the attitude information may include at least one of roll information, pitch information, or yaw information.

[0070] A microphone (170) of an electronic device (100) according to one embodiment is configured to receive user voice or other sounds and convert them into audio data.

[0071] FIG. 3 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0072] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0073] According to one embodiment, operations 310 to 350 can be understood as being performed in the processor (110) of the electronic device (100).

[0074] According to FIG. 3, in operation 310, an electronic device (100) according to one embodiment can identify whether a user request containing a natural language sentence is received. A natural language sentence (or a sentence in the form of a natural language) may be a sentence composed of language that people use in their daily lives. For example, a natural language sentence may include various elements such as grammar, vocabulary, semantics, and context.

[0075] For example, a user request may be a user request for routine generation. Routine generation (or automatic script generation) may be a function configured to automatically perform desired tasks under specific conditions and / or times.

[0076] According to one embodiment, when an electronic device (100) receives a user request corresponding to the selection of at least one of a specific icon or a specific physical button, the electronic device (100) can identify that the user request is a user request for routine generation. According to one example, when an electronic device (100) receives a user request through at least one of a specific icon or a specific physical button, the received user request can identify that it contains a natural language sentence for routine generation.

[0077] According to one example, the electronic device (100) may receive a user request containing a natural language sentence via the user's voice. For example, the electronic device (100) may receive voice input by activating the microphone (170) when a first icon provided on the display (130) is selected by touch operation. For example, the electronic device (100) may receive voice input by activating the microphone (170) when a first physical button is selected by press operation.

[0078] According to one example, the electronic device (100) may receive a user request containing a natural language sentence through text input. For example, the electronic device (100) may receive text input by providing a virtual keyboard on the display (130) when a second icon provided on the display (130) is selected by touch operation. For example, the electronic device (100) may receive text input by providing a virtual keyboard on the display (130) when a second physical button is selected by press operation. For example, the virtual keyboard may be a keyboard implemented in software provided on the touchscreen of the display (130).

[0079] According to one example, the second icon for receiving text input may be the same as or different from the first icon for receiving user voice. For example, if the second icon is the same as the first icon, the microphone (170) may be activated or a virtual keyboard may be provided on the display (130) by a different type of touch operation.

[0080] According to one example, the second physical button for receiving text input may be the same as or different from the first physical button for receiving user voice. For example, if the second physical button is the same as the first physical button, the microphone (170) may be activated or a virtual keyboard may be provided on the display (130) by a different type of pressing operation.

[0081] However, the above examples are not limited to the examples described above, and a user request for routine creation may be received through at least one of a general voice recognition assistant service and a general chatbot service. According to one example, when an electronic device (100) receives a natural language sentence through at least one of a voice recognition assistant service and a chatbot service, it can identify whether the natural language sentence contains a pre-set keyword. According to one example, if it is identified that a natural language sentence received through the electronic device (100) contains a pre-set keyword, it can be identified as a user request for routine creation. For example, the pre-set keyword may include words such as "routine" or "create". This is to filter whether a user request received in a general service environment is a request for routine creation.

[0082] When a user request containing a natural language sentence is received (310:Y), in operation 320, the electronic device (100) according to one embodiment can identify conditions and actions contained in the natural language sentence. If a user request containing a natural language sentence is not received (310:N), subsequent operations may not proceed.

[0083] According to one embodiment, an electronic device (100) can identify conditions and actions included in a natural language sentence by inputting a first prompt to a learned artificial intelligence model to output text corresponding to the natural language sentence by distinguishing it into conditions and actions. The prompt may be a text input that instructs a specific algorithm or artificial intelligence model to perform a desired task. According to one example, the electronic device (100) may obtain a first prompt for outputting text by distinguishing it into conditions and actions according to a pre-set rule. Rule-based prompt generation may be a method of generating a prompt using at least one of pre-defined rules, patterns, formats, and templates.

[0084] According to one example, the artificial intelligence model may be an on-device model included in the electronic device (100) or a model included in an external device (e.g., a server). According to one example, at least some functions of the artificial intelligence model may be executed in the electronic device (100) and at least some functions may be implemented in an external device (e.g., a server). However, for the convenience of explanation, the following description will assume that the artificial intelligence model is an on-device model included in the electronic device (100).

[0085] For example, an artificial intelligence model trained may include a large language model (LM) trained to identify conditions and actions contained in natural language sentences. For example, the LLM may be an artificial neural network-based language model that has learned a large amount of text data through prior training. For example, the LLM may include far more parameters (e.g., more than 10 billion) than a general language model. For example, the LLM can perform natural language processing (NLP) on text.

[0086] According to one example, a condition may be at least one of a trigger, situation, environment, or rule for a routine to be executed when a certain situation or event occurs. For example, the condition may include at least one of time, date, day of the week, location, battery level, communication connection status, sensor information, app execution, and notification reception.

[0087] According to one example, the action may be a task that is automatically performed by the electronic device (100) when a condition is met. For example, the action may include at least one of network settings, changing sound mode, adjusting screen brightness, launching an app, closing an app, making a phone call, media and / or voice control, providing notifications and / or information, and smart home device control actions.

[0088] According to one example, when a user voice is received, the electronic device (100) can acquire text corresponding to the user voice and input the acquired text into an artificial intelligence model. For example, the electronic device (100) can preprocess the received audio signal (e.g., noise removal, segmentation, normalization, etc.) and convert the preprocessed audio data into text data. For example, the electronic device (100) can convert the preprocessed audio data into text data using Automatic Speech Recognition (ASR) technology.

[0089] According to one embodiment, the electronic device (100) can convert (or translate) a natural language sentence into a preset language and identify conditions and actions included in the natural language sentence converted into the preset language.

[0090] According to one example, the electronic device (100) can identify conditions and actions included in a natural language sentence by inputting a first prompt to a learned artificial intelligence model that distinguishes text corresponding to a natural language sentence into conditions and actions and outputs them in a preset language. For example, the preset language may be a language determined based on the processing language of the artificial intelligence model. For example, if the processing performance of the artificial intelligence model for English is superior to that of other languages, the electronic device (100) can input a first prompt to the artificial intelligence model that translates a natural language sentence of a non-English language (e.g., Korean) into English and distinguishes and outputs conditions and actions.

[0091] According to one example, the electronic device (100) can identify conditions and actions by inputting a first prompt obtained based on a natural language sentence converted into a preset language into an artificial intelligence model. For example, the preset language may be a language determined based on the processing language of the artificial intelligence model. For example, if the processing performance of the artificial intelligence model for English is superior to that of other languages, the electronic device (100) can convert a natural language sentence of a non-English language (e.g., Korean) into English and input a first prompt obtained based on the natural language sentence converted into English into the artificial intelligence model.

[0092] In operation 330, an electronic device (100) according to one embodiment can obtain first data corresponding to a condition and second data corresponding to an operation.

[0093] According to one example, the electronic device (100) can retrieve a plurality of first vector data related to conditions included in a natural language sentence by searching a first database that stores vector data (or feature data) for conditions.

[0094] According to one example, the electronic device (100) may search a second database that stores vector data (or feature data) for an action to obtain second data including a plurality of second vector data related to an action included in a natural language sentence.

[0095] For example, vector data may be data converted into a structure that can be understood or processed by an artificial intelligence model (e.g., LLM) by vectorizing text data. For example, vector data may be data in JSON (JavaScript Object Notation) format (hereinafter referred to as JSON data), but is not limited to this as long as it is a data format that can be understood or processed by an artificial intelligence model (e.g., LLM). For example, JSON data is a lightweight data exchange format and may consist of key-value pairs and / or arrays.

[0096] According to one example, the first database may be a database capable of systematically storing, managing, and retrieving vector data that quantifies (or measures) specific conditions. For example, the first database may store vector data in which various conditions (e.g., time, weather, user status) are converted into vectors, and may process similarity searches and / or condition-based queries. For example, each vector data may represent a specific condition in a high-dimensional space.

[0097] According to one example, the first database can search for similar conditions by calculating similarity between vectors. For example, the first database can calculate similarity between vectors based on cosine similarity, Euclidean distance, and score-based search. For example, cosine similarity can calculate similarity based on the angle between vectors. For example, Euclidean distance can calculate similarity based on the straight-line distance between vectors. For example, score-based search can assign vector scores according to specific criteria.

[0098] For example, the first database may store vector data and metadata in conjunction. For instance, the vector data may be high-dimensional numeric data in which conditions are numerical, and the metadata may be attribute information (e.g., date, time, location) associated with the vector.

[0099] According to one example, the first database can index vector data for fast searching. For example, the first database can index vector data using at least one of a Spatial Partition Tree (SPT), an Approximate Nearest Neighbor (ANN) algorithm, and Hierarchical Clustering.

[0100] According to one example, the second database may be a database capable of systematically storing, managing, and retrieving vector data that quantifies (or measures) data related to actions. For example, the vector data related to actions may be data that quantifies or measures at least one of the steps of each action, the continuity between steps, or the result. For example, the second database may store vector data that converts various actions (e.g., network settings, changing sound mode, adjusting screen brightness, launching an app, closing an app) into vectors, and may process similarity searches and / or action-based queries. For example, each vector data may represent a specific condition in a high-dimensional space.

[0101] For example, a second database can search for similar conditions by calculating similarity between vectors. For instance, the second database can calculate similarity between vectors based on cosine similarity, Euclidean distance, and score-based search. For instance, cosine similarity can calculate similarity based on the angle between vectors.

[0102] For example, a second database may store vector data and metadata in conjunction. For instance, the vector data may be high-dimensional numeric data in which conditions are quantified, and the metadata may be attribute information associated with the vector (e.g., network settings, sound mode change, screen brightness adjustment, app launch, app termination).

[0103] According to one example, the second database can index vector data for fast searching. For example, the second database can index vector data using at least one of a Spatial Partition Tree (SPT), an Approximate Nearest Neighbor (ANN) algorithm, and Hierarchical Clustering.

[0104] According to one example, multiple first vector data related to conditions included in the first data may be vector data related to conditions included in natural language sentences retrieved through the first database.

[0105] According to one example, multiple second vector data related to actions included in the second data may be vector data related to actions included in natural language sentences retrieved through the second database.

[0106] In operation 340, an electronic device (100) according to one embodiment can obtain routine generation data for routine generation based on first data and second data.

[0107] According to one embodiment, the electronic device (100) may obtain routine generation data for routine generation based on first data, second data, and information about the electronic device (100). According to one example, the information about the electronic device (100) may include at least one of device information, type information, model information, hardware specification information, and installed application information. For example, the hardware specification information may include at least one of display information, camera information, battery information, memory information, operating system information, and network information.

[0108] According to one example, an electronic device (100) can input first data and second data into a learned artificial intelligence model to obtain routine generation data for routine generation.

[0109] According to one example, the electronic device (100) can obtain routine generation data for routine generation by inputting the first data, the second data, and information about the electronic device (100) into a learned artificial intelligence model.

[0110] According to one example, the artificial intelligence model trained may include a large language model (LM) trained to acquire routine generation data. According to one example, the artificial intelligence model trained may be the same artificial intelligence model as the artificial intelligence model that distinguishes conditions and actions in natural language sentences, but is not limited thereto.

[0111] According to one example, an artificial intelligence model trained can obtain routine generation data by inputting a second prompt to the trained artificial intelligence model to output routine generation data based on text corresponding to a natural language sentence, first data, and second data.

[0112] According to one example, an artificial intelligence model trained can obtain routine generation data by inputting a second prompt to the trained artificial intelligence model to output routine generation data based on text corresponding to a natural language sentence, first data, second data, and information about an electronic device (100).

[0113] According to one example, the electronic device (100) may obtain a second prompt for obtaining routine generation data according to a preset rule. Rule-based prompt generation may be a method of generating a prompt using at least one of a preset rule, pattern, format, and template.

[0114] Routine generation data obtained through an artificial intelligence model trained according to one example may include detailed parameter information related to at least one of a condition or action included in a natural language sentence. For example, the detailed parameter information may include numerical information used for routine generation that is not included in the natural language sentence.

[0115] According to one embodiment, when conditions and actions are distinguished in a natural language sentence, the electronic device (100) supplements (or modifies) the conditions and / or actions based on information about the electronic device (100), and can obtain first data and second data based on the supplemented conditions and / or actions. According to one example, the electronic device (100) can obtain routine generation data based on the first data and second data obtained based on the supplemented conditions and / or actions.

[0116] In operation 350, the electronic device (100) according to one embodiment can generate a routine executable in the electronic device (100) based on routine generation data.

[0117] According to one example, the routine may be at least one set of program code and instructions so that when information identified in the electronic device (100) corresponds to a specific condition, the electronic device (100) automatically executes a specific action. According to one example, the electronic device (100) may identify whether the identified information corresponds to a specific condition based on at least one of data received through the communication circuit (140) or data acquired through the sensor (160).

[0118] According to one example, the electronic device (100) can set the execution conditions of the routine based on condition information included in the routine generation data.

[0119] According to one example, the electronic device (100) can set the operation to be performed under execution conditions based on the operation information included in the routine generation data.

[0120] According to one embodiment, when routine generation data is obtained, the electronic device (100) may display a UI screen containing information included in the routine generation data through a display (130). According to one example, the electronic device (100) may display a UI screen containing condition information and operation information included in the routine generation data through a display (130). For example, the electronic device (100) may display a UI screen containing detailed parameter information used for routine generation through a display (130). For example, the detailed parameter information may include numeric information used for routine generation that is not included in the natural language sentence. For example, the numeric information may include information about at least one of a date and a time.

[0121] According to one example, the electronic device (100) can modify routine generation data based on user input through a UI screen. According to one example, the electronic device (100) can generate a routine based on the modified routine generation data.

[0122] According to one embodiment, the electronic device (100) can generate a routine based on routine generation data and information about the electronic device (100). According to one example, if information about the electronic device (100) is not used when acquiring routine generation data, the electronic device (100) can supplement (or modify) the routine generation data based on the information about the electronic device (100) and generate a routine based on the supplemented routine generation data.

[0123] According to one embodiment, information regarding an electronic device (100) used for at least one of routine data generation, first data and second data supplementation, and routine generation (or routine data supplementation) may further include at least one of context information of the electronic device (100) or user usage history information. The context information of the electronic device (100) may include at least one of an internal context or an external context of the electronic device (100). The user usage history information may include at least one of the user's real-time usage information, the user's past usage information, or the user's habit information.

[0124] Context information may include various information such as at least one of time information, environment information, location information, and physiological state information. For example, time information may include time-related information such as the current time, day of the week, season, etc. For example, environment information may include at least one of weather information, temperature information, and illuminance information as information about the surrounding environment. For example, physiological state information may include health-related data and / or physiological information such as at least one of heart rate, sleep patterns, and activity levels through a wearable device.

[0125] As illustrated in FIG. 3, the order of all operations is mapped for convenience of explanation, but the embodiment is not limited thereto and one or more operations may be performed in any order, for example, in parallel.

[0126] FIG. 4 is a diagram illustrating a generative artificial intelligence model according to one embodiment.

[0127] According to one embodiment, an artificial intelligence model that distinguishes conditions and actions in natural language sentences can be implemented as a Generative AI Model (405) as shown in FIG. 4.

[0128] According to one embodiment, an artificial intelligence model that outputs routine generation data can be implemented as a Generative AI Model (405) as shown in FIG. 4.

[0129] According to FIG. 4, the User Query / Response Interface (401) can receive user input. The user input may be in the form of natural language, images and / or videos.

[0130] For example, user input may include user voice received through a microphone. However, it is not limited thereto, and user input may include text corresponding to the voice generated by a speech-to-text (STT) model in addition to voice. Furthermore, context information may be transmitted along with the user input. Context information may include various additional information at the time of user input. For example, this may include information about the application currently being used by the user or the user's location information. Additionally, user input may take the form of a mixture of the aforementioned natural language, images, sounds, and context information. Furthermore, user input may also take the form of non-natural language input, such as menu selection.

[0131] The User Query / Response Interface (401) can output results from the Generative AI system to the user. The output can be in the form of natural language or specific content, and it can also be provided in the form of an action requested by the user. The User query interface (401) can output results from the Generative AI system to the user. The output can be in the form of natural language or specific content, and it can also be provided in the form of an action requested by the user. For example, the User Query / Response Interface (401) can output content generated by the Generative AI Model (405) based on voice received from the user.

[0132] The AI ​​framework (402) receives user input and can coordinate and control each component necessary to perform the user's intent based on the user's query.

[0133] User input received from the User Query / Response Interface (401) can be sent to the Prompt design component (402-1). The Prompt design component (402-1) can be used to generate a prompt suitable for inputting user input into a large language model (LLM) or a large multimodal model (LMM). The Prompt design component (402-1) may be an AI component that uses machine learning algorithms or neural networks to develop better prompts over time. The Prompt design component (402-1) can generate a prompt by accessing knowledge repositories (403) containing user preference data, prompt libraries, and prompt examples based on user input, and can send the generated prompt to the LLM or LMM.

[0134] The API / Plug-in management component (402-2) can perform the role of communicating with external information when there is a request for additional information when passing user input as input to a generative model. The API / Plug-in management component (402-2) establishes a channel to communicate with the outside of the AI ​​Interface via the API, and can enable access to various data sources through the established channel. Additionally, the API / Plug-in management component (402-2) can request an action via the API if the application or service needs to perform an action that executes the user input as a final step, rather than an intermediate result. Information obtained from the outside (e.g., the Applications / service component (404)) may be used to generate a prompt in the Prompt design component (402-1) along with the user input, or it may be passed as input to the generative model.

[0135] The Output modification Component (402-3) (or Refiner component) can fine-tune the output of the generative model. For example, the Output modification Component (402-3) can verify whether the content generated through the LLM and / or LMM is irrelevant, contains biased content, or contains harmful content. Additionally, the Output modification Component (402-3) can determine the extent to which the output matches the desired result and, if additional processing is required, proceed with that process. Furthermore, the Output modification Component (402-3) can configure and provide hints to the user to avoid unwanted output.

[0136] A Generative AI Model (405) generally refers to an artificial intelligence neural network that generates new forms of data based on user input information. A Generative AI Model (405) may include models that generate images and / or models that generate language. Models that generate images include, but are not limited to, GANs (generative adversarial networks) and VAEs (variational autoencoders), and examples of Diffusion-based generative models using VAEs and Transformer structures. Models that generate language are models trained to output the most statistically appropriate output value based on input values, and examples of which include CHAT-GPT 3 and CHAT-GPT 4 (e.g., CHAT-GPT 4o). There are also LMMs (large multimodal models) that can recognize various forms of data input, such as text, images, and voice, and generate new data corresponding to them.

[0137] According to an embodiment, when a prompt is input from the Prompt design component (402-1), the Generative AI Model (405) generates content corresponding to the prompt based on the instructions and can output the content through the User Query / Response Interface (401).

[0138] FIGS. 5A and FIGS. 5B are drawings for explaining a method for receiving a user request according to one embodiment.

[0139] According to one embodiment, the electronic device (100) can receive a user request for routine generation using at least one of a specific icon or a specific physical button.

[0140] According to the example illustrated in FIG. 5a, the electronic device (100) may provide a UI screen (510) containing a specific icon (511). According to one example, when the specific icon (511) is selected by touch operation, the electronic device (100) may activate a microphone (170) to receive voice input. For example, the electronic device (100) may receive user voice containing a natural language sentence such as “Create a routine to reduce screen brightness and turn on the blue light function when it gets dark in the evening,” convert the received user voice into text, and display the converted text (512).

[0141] According to the example illustrated in FIG. 5b, the electronic device (100) can receive text input by providing a UI screen (520) that includes a virtual keyboard (521). For example, the electronic device (100) can display text (522) containing a natural language sentence, “Create a routine to reduce screen brightness and turn on the blue light function when it gets dark in the evening,” which is entered through the virtual keyboard (521).

[0142] FIG. 6 is a diagram illustrating an example of a routine generation method according to one embodiment.

[0143] According to one embodiment, each module (620, 630) illustrated in FIG. 6 may be implemented with at least one software, at least one hardware, and / or a combination thereof. For example, each module (620, 630) may be implemented to use a predefined algorithm, a predefined formula, and / or an artificial intelligence model. Each module (620, 630) may be contained within an electronic device (100), but may be distributed to at least one external device according to one example.

[0144] According to an embodiment illustrated in FIG. 6, when an electronic device (100) receives a user voice containing a natural language sentence for routine generation, the electronic device (100) can convert the natural language sentence contained in the user voice into text (614). In one example, the electronic device (100) can convert the natural language sentence contained in the user voice into text using speech-to-text (STT) technology. For example, STT technology may be a technology that converts human voice into digital text. For example, STT technology may convert human voice into digital text using at least one of preprocessing, an acoustic model, a language model, and decoding. For example, preprocessing may include at least one of sampling, noise filtering, and frequency analysis of the voice signal. For example, an acoustic model may analyze the voice signal to learn the relationship between sounds (e.g., phonemes) and specific text. For example, a language model may analyze the grammatical and statistical patterns of the text to provide more accurate conversion results. For example, decoding may combine the results of the acoustic model and the language model to generate the final text.

[0145] According to one example, the electronic device (100) can obtain a natural language sentence in the form of text, such as “create a routine to reduce screen brightness and turn on the blue light function when it gets dark in the evening,” based on the user’s voice.

[0146] According to one embodiment, the routine generation module (620) can use the LLM (630) to identify (or separate or distinguish) conditions and actions included in a natural language sentence (612).

[0147] According to one example, the routine generation module (620) can generate a first prompt to distinguish conditions and actions included in a natural language sentence and input it into the LLM (630).

[0148] For example, when the LLM (630) receives a first prompt to distinguish between conditions and actions included in a natural language sentence, “Generate a routine to reduce screen brightness and turn on the blue light function when it gets dark in the evening,” it can distinguish between the condition “when it gets dark” and the action “reduce brightness and turn on the blue light filter” and output them as a response to the first prompt. For example, the routine generation module (620) can identify the conditions and actions included in the natural language sentence based on the response output from the LLM (630).

[0149] FIGS. 7 and 8 are drawings for illustrating examples of a first prompt and a response to the first prompt according to one embodiment.

[0150] According to one example, the first prompt may include a request to identify conditions and actions for the corresponding user request ("Create a routine to reduce screen brightness and turn on the blue light function when it gets dark in the evening") as illustrated in FIG. 7, a request to respond in a specific format, and an example of the response.

[0151] According to one example, the response to the first prompt may include a condition identified for the user request ("when it gets evening") and a response ("reduce brightness and turn on the blue light filter") as illustrated in FIG. 8.

[0152] FIGS. 9 and FIGS. 10 are drawings for illustrating examples of a first prompt and a response to the first prompt according to one embodiment.

[0153] According to one embodiment, the electronic device (100) can convert a natural language sentence into a preset language and identify conditions and actions included in the natural language sentence converted into the preset language. According to one example, the electronic device (100) can identify conditions and actions included in the natural language sentence by inputting a first prompt into a learned artificial intelligence model to distinguish text corresponding to the natural language sentence into conditions and actions and output them in the preset language.

[0154] According to one example, the first prompt may include a request to identify conditions and responses by translating the corresponding user request ("Create a routine to reduce screen brightness and turn on the blue light function when it gets dark in the evening") into English as shown in FIG. 9, a request to respond in a specific format, and an example of a response.

[0155] According to one example, the response to the first prompt may include the corresponding user request ("Create a routine to reduce screen brightness and turn on blue light filter when it gets dark in the evening") as illustrated in FIG. 8, and the condition ("In the evening") and response ("reduce screen brightness and turn on blue light filter") identified in the sentence translated into English ("In the evening, reduce screen brightness and turn on blue light filter").

[0156] Returning to FIG. 6, according to one embodiment, the routine generation module (620) may search the condition vector database (641) to obtain a vector data set associated with conditions identified in natural language sentences (613). According to one example, the condition vector database (641) may include a data set in which at least one parameter is stored in association with each other for conditions executable in the electronic device (100). For example, the condition vector database (641) may store data in the form of vector embeddings, which are unstructured data such as text, images, and / or audio converted into numerical vectors. For example, vector embeddings may be located in a multidimensional space, and similar data may be located close to each other in space. For example, the routine generation module (620) may perform a similarity-based search. For example, the routine generation module (620) may convert query data into vectors and calculate the distance to the vector embeddings stored in the condition vector database (641) to search for the most similar results. This may mean that the search is performed based on semantic similarity rather than keyword-based exact matching.

[0157] According to one embodiment, the routine generation module (620) may search the action vector database (642) to obtain a vector data set related to actions identified in natural language sentences (613). According to one example, the action vector database (642) may include a data set in which at least one parameter is stored in association with each other for actions that can be executed on the electronic device (100). For example, the action vector database (642) may store data in the form of vector embeddings, which are unstructured data such as text, images, and / or audio converted into numerical vectors. For example, vector embeddings are located in a multidimensional space, and similar data may be located close to each other in space. For example, the routine generation module (620) may perform a similarity-based search. For example, the routine generation module (620) may convert query data into vectors and calculate the distance to the vector embeddings stored in the action vector database (642) to search for the most similar results. This may mean that the search is performed based on semantic similarity rather than keyword-based exact matching.

[0158] FIG. 11 is a diagram illustrating the schema of a condition vector database and an action vector database according to one embodiment.

[0159] According to one example illustrated in FIG. 11, a condition vector database may include a tag corresponding to a condition executable in an electronic device (100), a description of the condition, and at least one of a parameter key (param_key), a parameter type (param_type), and a description of the parameter (param_description) as parameters associated with the condition.

[0160] According to one example illustrated in FIG. 11, the action vector database may include a tag corresponding to an action executable in an electronic device (100), a description of the action, and at least one of a parameter key (param_key), a parameter type (param_type), and a description of the parameter (param_description) as parameters associated with the action.

[0161] FIGS. 12a and FIGS. 12b are drawings for illustrating examples of vector data related to conditions according to one embodiment.

[0162] According to one embodiment, when a condition and an action included in a natural language sentence included in a user request are identified, the electronic device (100) may retrieve vector data for the condition and the action to obtain first data related to the condition and second data related to the action. According to one example, if the natural language sentence is "In the evening, reduce screen brightness and turn on blue light filter", the condition may be "In the evening" and the action may be "reduce screen brightness and turn on blue light filter".

[0163] According to one example illustrated in FIG. 12a and FIG. 12b, the first data associated with the condition "In the evening" may include a plurality of first vector data associated with "In the evening".

[0164] For example, as illustrated in FIG. 12a, a plurality of first vector data including a plurality of parameter keys (param_key), each parameter type (param_type), and a description of the parameter (param_description) can be obtained as parameters associated with “In the evening” in an electronic device (100).

[0165] For example, as shown in FIG. 12a and FIG. 12b, multiple parameters related to “tag”: “specific_time”, “version”: “v2”, “package”: “com.samsung.android.app.routines”, “description”: “Conditions for operating according to a specific time schedule specified by the user” may each include parameter keys “0”, “1”, “2”, “3”, “4”, “5”, “6”, “7”, “8”, “9”.

[0166] For example, as illustrated in FIG. 12a, the parameter type corresponding to parameter key "0" may be "NUMBER" and the parameter description may be "Parameter that represents repeating days of week which is a form of bits". For example, the parameter type corresponding to parameter key "1" may be "NUMBER" and the parameter description may be "Parameter that represents the hour of day". For example, the parameter type corresponding to parameter key "2" may be "NUMBER" and the parameter description may be "Parameter that represents the minute of day". For example, the parameter type corresponding to parameter key "3" may be "NONE" and the parameter description may be "Deprecated". For example, the parameter type corresponding to parameter key "4" may be "NUMBER" and the parameter description may be "Parameter that represents time type such as normal time (=0), sunrise (=1), sunset (=2)".

[0167] For example, as illustrated in FIG. 12b, the parameter type corresponding to parameter key "5" may be "NUMBER" and the parameter description may be "Parameter that represents delayed minute from the decided time". For example, the parameter type corresponding to parameter key "6" may be "BOOLEAN" and the parameter description may be "Parameter that represents this time is disabled during holiday". For example, the parameter type corresponding to parameter key "7" may be "NUMBER" and the parameter description may be "Parameter that represents timer id". For example, the parameter type corresponding to parameter key "8" may be "NUMBER" and the parameter description may be "Parameter of repeat type such as every day(=0), every week(=1), every month(=2)". For example, the parameter type corresponding to parameter key "9" may be "STRING" and the parameter description may be "Parameter that represents repeating dates of month in the form of bits (made of only 0 and 1)".

[0168] FIGS. 13a and FIGS. 13b are drawings for illustrating examples of vector data related to an operation according to one embodiment.

[0169] According to one example illustrated in FIG. 13a and FIG. 13b, the first data associated with the operation “reduce screen brightness and turn on blue light filter” may include a plurality of first vector data associated with “reduce screen brightness and turn on blue light filter”.

[0170] For example, as illustrated in FIG. 13a, multiple parameters associated with "tag": "sec_eye_comfort_shield_routine", "version": "v3", "package": "com.android.settings", "description": "Actions to modify eye comfort shield states" may each include parameter values ​​"blue_light_filter_switch", "blue_light_filter_scheduled", "blue_light_filter_opacity", "blue_light_filter_type", and "blue_light_filter_adaptive_mode".

[0171] For example, as illustrated in FIG. 13a, the parameter type corresponding to the parameter key "blue_light_filter_switch" may be "NUMBER", and the parameter description may be "Parameter that represents whether the eye comfort shield needs to be turned on (=1) or off (=0)". For example, the parameter type corresponding to the parameter key "blue_light_filter_scheduled" may be "NUMBER", and the parameter description may be "Parameter that represents whether the custom mode for eye comfort shield need to be turned on (=1) or off (=0)". For example, the parameter type corresponding to the parameter key "blue_light_filter_opacity" may be "NUMBER", and the parameter description may be "Parameter that represents the eye comfort shield value or opacity for custom mode". For example, the parameter type corresponding to the parameter key "blue_light_filter_type" may be "NUMBER", and the parameter description may be "Parameter that represents the schedule type always on with value equals to 0 always". For example, the parameter type corresponding to the parameter key "blue_light_filter_adaptive_mode" is "NUMBER", and the parameter description can be "Parameter that represents whether the adaptive mode for eye comfort shield need to be turned on(=1) or off(=0)".

[0172] For example, as illustrated in FIG. 13b, multiple parameters related to "tag": "brightness", "version": "v2", "package": "com.samsung.android.app.routines", "description": "Action to change the brightness level setting setting of the mobile phone" may each include parameter values ​​"0", "1", and "2".

[0173] For example, as illustrated in FIG. 13b, the parameter type corresponding to parameter key "0" may be "NUMBER" and the parameter description may be "Parameter that represents mode of brightness such as manual (=0) and automatic (=1)". For example, the parameter type corresponding to parameter key "1" may be "NUMBER" and the parameter description may be "Parameter that represents brightness level". For example, the parameter type corresponding to parameter key "2" may be "NUMBER" and the parameter description may be "Parameter that represents whether extra brightness to be turned on (=1) or off (=0)".

[0174] For example, as illustrated in FIG. 13b, the parameter associated with "tag": "adaptive_brightness", "version": "v3", "package": "com.samsung.android.app.routines", "description": "Action to turn on or off the adaptive brightness setting of the mobile phone" may include a parameter key "toggle_value". For example, as illustrated in FIG. 13b, the parameter type corresponding to the parameter key "toggle_value" may be "BOOLEAN", and the parameter description may be "Parameter that determines whether adaptive brightness should be applied (=true) on main screen or not (=false)".

[0175] Returning to FIG. 6, according to one embodiment, the routine generation module (620) may request routine generation based on a vector data set related to conditions and a vector data set related to actions using the LLM (630) (614). For example, the LLM (630) may output routine generation data based on the vector data set related to conditions and the vector data set related to actions.

[0176] FIG. 14 is a diagram illustrating an example of routine generation data according to one embodiment.

[0177] According to one embodiment, an electronic device (100) can obtain routine generation data by inputting a prompt obtained based on first data related to conditions and second data related to actions retrieved through a vector database into a learned artificial intelligence model. For example, the electronic device (100) can input a prompt obtained based on text corresponding to a natural language sentence, first data, second data, and a pre-set routine generation data related template into a learned artificial intelligence model.

[0178] According to one example illustrated in FIG. 14, routine generation data may include a routine name, a type corresponding to a condition, a tag, a version, a package, at least one parameter information, and a type corresponding to an operation, a tag, a version, a package, and at least one parameter information. The at least one parameter information may include at least one of a parameter key (param_key), a parameter type (param_type), a description of the parameter (param_description), and a parameter value (param_value).

[0179] FIGS. 15a and FIGS. 15b are drawings for illustrating examples of routine generation data according to one embodiment.

[0180] According to one example illustrated in FIGS. 15a and 15b, when the natural language sentence is "In the evening, reduce screen brightness and turn on blue light filter," routine generation data including multiple parameters for conditions and multiple parameters for actions can be obtained. According to one example, the routine generation data (or automation script) may include detailed parameter values ​​of conditions and / or actions (e.g., brightness values, quantified time values) that are not included in the natural language sentence. For example, a trained artificial intelligence model may be trained to output routine generation data that includes detailed parameter values ​​of conditions and / or actions not included in the natural language sentence, based on vector data related to the natural language sentence obtained from a vector database.

[0181] According to FIG. 15a, multiple parameters related to "tag": "sec_eye_comfort_shield_routine", "version": "v3", "package": "com.android.settings", "description": "Actions to modify eye comfort shield states", "type": "condition", "tag": "time", "version": "v2", "package": "com.samsung.android.app.routines" may include parameter keys "0", "1", "2", "3", "4".

[0182] For example, the parameter type corresponding to parameter key "0" is "NUMBER", the parameter description is "Parameter that represents repeating days of week which is a form of bits", and the parameter value can be "127". For example, the parameter type corresponding to parameter key "1" is "NUMBER", the parameter description is "Parameter that represents the starting hour of day", and the parameter value can be "18". For example, the parameter type corresponding to parameter key "2" is "NUMBER", the parameter description is "Parameter that represents the starting minute of day", and the parameter value can be "0". For example, the parameter type corresponding to parameter key "3" is "NUMBER", the parameter description is "Parameter that represents the ending hour of day", and the parameter value can be "23". For example, the parameter type corresponding to parameter key "4" is "NUMBER", the parameter description is "Parameter that represents the ending minute of day", and the parameter value can be "59".

[0183] According to FIG. 15b, multiple parameters related to "type": "action", "tag": "brightness", "version": "v2", "package": "com.samsung.android.app.routines" may include parameter keys "0", "1".

[0184] For example, the parameter type corresponding to parameter key "0" is "NUMBER", the parameter description is "Parameter that represents mode of brightness such as manual(=0) and automatic(=1)", and the parameter value can be "0". For example, the parameter type corresponding to parameter key "1" is "NUMBER", the parameter description is "Parameter that represents brightness level", and the parameter value can be "10".

[0185] According to FIG. 15b, parameters associated with "type": "action", "tag": "sec_eye_comfort_shield_routine", "version": "v3", "package": "com.android.settings" may include the parameter key "blue_light_filter_switch".

[0186] For example, the parameter type corresponding to the parameter key "blue_light_filter_switch" is "NUMBER", the parameter description is "Parameter that represents whether the eye comfort shield needs to be turned on(=1) or off(=0)", and the parameter value can be "1".

[0187] Returning to FIG. 6, according to one embodiment, the routine generation module (620) can generate a routine based on routine generation data output from the LLM (630) (615). According to one example, the routine generation module (620) can generate a routine by setting an execution condition value of the routine based on condition data included in the routine generation data and setting an action to be performed in the execution condition based on action data included in the routine generation data.

[0188] FIG. 16 is a drawing for explaining an example of a UI provision method according to one embodiment.

[0189] According to one embodiment, when routine generation data is obtained, the electronic device (100) may display a UI screen containing condition information and operation information included in the routine generation data through a display (130). According to one example, the electronic device (100) may display a UI screen containing detailed parameter information through a display (130). For example, the detailed parameter information may include numerical information used for routine generation that is not included in the natural language sentence. For example, the numerical information may include information about at least one of a date and a time.

[0190] According to FIG. 16, the electronic device (100) may display a UI screen (1610) including a routine name "Evening screen brightness adjustment", a condition "Every day at 6 PM", an action "Brightness 3%", and an action "View screen comfortably". For example, the user may add and / or modify at least some of the details of the conditions and actions provided through the UI screen (1610).

[0191] FIGS. 17 and FIGS. 18 are drawings for explaining an example of a method for providing a UI according to one embodiment.

[0192] According to the embodiment illustrated in FIG. 17, when a natural language sentence is input through voice and / or text input, the electronic device (100) may display a UI screen (1710) containing the input natural language sentence. For example, the UI screen (1710) may include text corresponding to the input natural language sentence and a button for generating a routine based on the natural language sentence.

[0193] According to one example, the electronic device (100) may display the input natural language sentence as is, or display the input natural language sentence with some modifications. For example, the electronic device (100) may modify the natural language sentence by excluding unnecessary modifiers.

[0194] For example, the input natural language sentence itself could be “When evening comes, lower the screen brightness and turn on the blue light.”

[0195] For example, the input natural language sentence is “Create a routine to reduce screen brightness and turn on the blue light function when it gets dark in the evening,” but a slightly modified sentence could be “Reduce screen brightness and turn on the blue light when it gets evening.”

[0196] According to the embodiment illustrated in FIG. 18, the electronic device (100) may display a UI screen (1810) containing information included in routine generation data obtained based on a natural language sentence. For example, the UI screen (1810) may include a routine name "Adjust screen brightness in the evening," a condition "Every day at 6 PM," an action "Brightness 3%," and an action "View screen comfortably." For example, the natural language sentence included in the user request was "When it is evening, lower the screen brightness and turn on the blue light," but the final generated routine may include details not included in the natural language sentence (e.g., "Every day at 6 PM," "Brightness 3%").

[0197] FIG. 19 is a diagram illustrating an example of a scenario for generating a routine according to one embodiment.

[0198] According to FIG. 19, an electronic device (100) can generate routines using a routine generation module (1910), a vector database (1920), and an LLM (1930). In one example, at least one of the routine generation module (1910), the vector database (1920), and the LLM (1930) may be implemented in at least one software, at least one hardware, and / or a combination thereof. In one example, at least one of the routine generation module (1910), the vector database (1920), and the LLM (1930) may be contained within the electronic device (100), but in one example, may be distributed to at least one external device.

[0199] According to one embodiment, in the LLM embedding information generation step (1940), an LLM embedding conversion for the conditions and operations of the entire routine may be requested to the LLM (1930) (1941).

[0200] LLM embeddings can be the result of converting text data into numeric vectors in Natural Language Processing (NLP) models. For example, embedding is a technique that represents high-dimensional text data as low-dimensional, continuous numeric vectors so that computers can process them; this allows for increased computational efficiency while preserving the meaning of the text.

[0201] According to one example, the LLM (1930) can convert the conditions and actions of the entire routine into LLM embeddings (1942).

[0202] In one example, when the conversion of LLM embeddings by LLM (1930) is completed, the acquired LLM embeddings for the conditions and operations of the entire routine can be stored in a vector database (1920) (1943).

[0203] In the step (1950) of analyzing a user natural language request according to one embodiment, the user may request the creation of a routine based on a user request that includes a natural language sentence (1951). For example, the user request may include a natural language sentence such as "Create a routine that notifies me via Beep when my mom texts me asking where I am."

[0204] For example, the routine generation module (1910) can input a prompt to the LLM (1930) saying, "Give me a DB query with conditions and actions that match the user request" (1952). For example, the user request can be input to the LLM (1930) along with the prompt, saying, "Give me a routine that tells me where I am when I text my mom where she is" (1953).

[0205] According to one example, LLM (1930) can analyze the prompt and generate a response to the prompt (1954).

[0206] According to one example, the routine generation module (1910) may receive a response to a prompt from the LLM (1930) (1955). For example, the response to the prompt may be "Condition Query is 'condition when receiving character' and Action Query is 'beep'".

[0207] According to one example, the routine generation module (1910) can search the vector database (1920) for a 'when a character is received' condition and a 'beep' action, respectively, based on the received response (1956, 1957). For example, the routine generation module (1910) can search the condition 'when a character is received' in the condition vector database and search the 'beep' action in the action vector database.

[0208] According to one embodiment, in the LLM routine generation step (1960), the routine generation module (1910) according to one example may input a prompt to the LLM (1930) saying, "Generate a routine that matches the user request (Generate a routine that notifies me via Beep when Mom texts me asking where I am)" (1961). For example, the routine generation module (1910) may input routine condition and operation information obtained through a Jason template for routine generation and a Better database into the LLM (1930) by attaching them to the prompt (1961).

[0209] According to one example, LLM (1930) can analyze input prompts and attached data to generate a response to the prompt (1963).

[0210] According to one example, the routine generation module (1910) may receive a response to a prompt from the LLM (1930) (1964). For example, the response to the prompt may be "Generated routine jason information is as follows. Attachment: Generated routine jason format result".

[0211] According to one example, the routine generation module (1910) can provide guidance information such as “The following routine has been generated” based on routine information received from the LLM (1930) (1965).

[0212] FIG. 20 is a block diagram of an electronic device in a network environment according to various embodiments. The electronic device (2401) may be implemented as the electronic device (100) shown in FIG. 2 according to one example.

[0213] Referring to FIG. 20, in a network environment (2400), an electronic device (2401) may communicate with an electronic device (2402) through a first network (2498) (e.g., a short-range wireless communication network) or with at least one of an electronic device (2404) or a server (2408) through a second network (2499) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (2401) may communicate with the electronic device (2404) through a server (2408). According to one embodiment, the electronic device (2401) may include a processor (2420), memory (2430), input module (2450), sound output module (2455), display module (2460), audio module (2470), sensor module (2476), interface (2477), connection terminal (2478), haptic module (2479), camera module (2480), power management module (2488), battery (2489), communication module (2490), subscriber identification module (2496), or antenna module (2497). In some embodiments, at least one of these components (e.g., connection terminal (2478)) may be omitted from the electronic device (2401), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (2476), camera module (2480), or antenna module (2497)) may be integrated into a single component (e.g., display module (2460)).

[0214] The processor (2420) can, for example, execute software (e.g., program (2440)) to control at least one other component (e.g., hardware or software component) of the electronic device (2401) connected to the processor (2420) and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (2420) can store commands or data received from other components (e.g., sensor module (2476) or communication module (2490)) in volatile memory (2432), process the commands or data stored in volatile memory (2432), and store the resulting data in non-volatile memory (2434). According to one embodiment, the processor (2420) may include a main processor (2421) (e.g., a central processing unit or an application processor) or an auxiliary processor (2423) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (2401) includes a main processor (2421) and an auxiliary processor (2423), the auxiliary processor (2423) may be configured to use lower power than the main processor (2421) or to be specialized for a specified function. The auxiliary processor (2423) may be implemented separately from the main processor (2421) or as part thereof.

[0215] The auxiliary processor (2423) may control at least some of the functions or states associated with at least one component of the electronic device (2401) (e.g., display module (2460), sensor module (2476), or communication module (2490)) on behalf of the main processor (2421) while the main processor (2421) is in an inactive (e.g., sleep) state, or together with the main processor (2421) while the main processor (2421) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (2423) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (2480) or communication module (2490)). According to one embodiment, the auxiliary processor (2423) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (2401) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (2408)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, an artificial intelligence model may include a software structure, either additionally or substantially.

[0216] The memory (2430) can store various data used by at least one component of the electronic device (2401) (e.g., processor (2420) or sensor module (2476)). The data may include, for example, input data or output data for software (e.g., program (2440)) and related commands. The memory (2430) may include volatile memory (2432) or non-volatile memory (2434).

[0217] The program (2440) may be stored as software in memory (2430) and may include, for example, an operating system (1442), middleware (1444), or an application (1446).

[0218] The input module (2450) can receive commands or data to be used for a component of the electronic device (2401) (e.g., processor (2420)) from outside the electronic device (2401) (e.g., user). The input module (2450) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0219] The sound output module (2455) can output a sound signal to the outside of the electronic device (2401). The sound output module (2455) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0220] The display module (2460) can visually provide information to an external (e.g., user) of the electronic device (2401). The display module (2460) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (2460) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0221] The audio module (2470) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (2470) can acquire sound through the input module (2450) or output sound through the sound output module (2455) or an external electronic device (e.g., electronic device (2402)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (2401).

[0222] The sensor module (2476) can detect the operating state of the electronic device (2401) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (2476) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0223] The interface (2477) may support one or more specified protocols that can be used for the electronic device (2401) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (2402)). According to one embodiment, the interface (2477) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0224] The connection terminal (2478) may include a connector through which the electronic device (2401) can be physically connected to an external electronic device (e.g., electronic device (2402)). According to one embodiment, the connection terminal (2478) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0225] The haptic module (2479) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (2479) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0226] The camera module (2480) can capture still images and video. According to one embodiment, the camera module (2480) may include one or more lenses, image sensors, image signal processors, or flashes.

[0227] The power management module (2488) can manage power supplied to the electronic device (2401). According to one embodiment, the power management module (2488) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0228] The battery (2489) can supply power to at least one component of the electronic device (2401). According to one embodiment, the battery (2489) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0229] The communication module (2490) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (2401) and an external electronic device (e.g., electronic device (2402), electronic device (2404), or server (2408)), and the performance of communication through the established communication channel. The communication module (2490) may include one or more communication processors that operate independently of the processor (2420) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (2490) may include a wireless communication module (2492) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (1494) (e.g., LAN (local area network) communication module, or power line communication module). Among these communication modules, the communication module described above can communicate with an external electronic device (2404) through a first network (2498) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (2499) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (2492) can identify or authenticate the electronic device (2401) within a communication network such as the first network (2498) or the second network (2499) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (2496).

[0230] The wireless communication module (2492) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (2492) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (2492) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (2492) can support various requirements specified in the electronic device (2401), external electronic device (e.g., electronic device (2404)), or network system (e.g., second network (2499)). According to one embodiment, the wireless communication module (2492) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0231] An antenna module (2497) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (2497) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (2497) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (2498) or a second network (2499), may be selected from the plurality of antennas, for example, by a communication module (2490). A signal or power may be transmitted or received between the communication module (2490) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (2497).

[0232] According to various embodiments, the antenna module (2497) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0233] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0234] According to one embodiment, commands or data may be transmitted or received between the electronic device (2401) and an external electronic device (2404) through a server (2408) connected to a second network (2499). Each of the external electronic devices (2402, or 2404) may be the same or a different type of device as the electronic device (2401). According to one embodiment, all or part of the operations performed on the electronic device (2401) may be performed on one or more of the external electronic devices (2402, 2404, or 2408). For example, if the electronic device (2401) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (2401) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (2401). The electronic device (2401) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (2401) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (2404) may include an Internet of Things (IoT) device. The server (2408) may be an intelligent server using machine learning and / or neural networks.According to one embodiment, an external electronic device (2404) or server (2408) may be included within the second network (2499). The electronic device (2401) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0235] According to one embodiment, the electronic device comprises: a memory for storing instructions; and at least one processor including processing circuitry; wherein, when the instructions are executed individually or collectively by the at least one processor, the electronic device, upon receiving a user request including a natural language sentence, identifies a condition and an action included in the natural language sentence, obtains first data corresponding to the condition and second data corresponding to the action, obtains routine generation data based on the first data, the second data and information about the electronic device, and generates a routine that causes the electronic device to automatically execute a specific action if the information identified by the electronic device corresponds to a specific condition based on the routine generation data.

[0236] According to one embodiment, the routine may be at least one set of program code and instructions that cause the electronic device to automatically execute a specific operation when information identified in the electronic device corresponds to a specific condition.

[0237] According to one embodiment, the electronic device further comprises a communication circuit and a sensor; and when the instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to automatically execute the specific operation if information identified based on at least one of data received through the communication circuit or data acquired through the sensor corresponds to the specific condition based on the routine.

[0238] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may search a first database storing vector data for conditions to obtain the first data including a plurality of first vector data related to conditions included in the natural language sentence, and search a second database storing vector data for actions to obtain the second data including a plurality of second vector data related to actions included in the natural language sentence.

[0239] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may input a first prompt to an artificial intelligence model to output text corresponding to the natural language sentence, distinguishing it into conditions and actions, thereby identifying the conditions and actions included in the natural language sentence.

[0240] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may convert the natural language sentence into a preset language and identify conditions and actions included in the natural language sentence converted into the preset language. The preset language may be a language determined based on the processing language of the artificial intelligence model.

[0241] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may input a second prompt to an artificial intelligence model to output routine generation data based on text corresponding to the natural language sentence, the first data, the second data, and a pre-set routine generation data related template, thereby obtaining the routine generation data.

[0242] According to one embodiment, the routine generation data may include detailed parameter information related to at least one of the conditions or actions included in the natural language sentence.

[0243] According to one embodiment, the detailed parameter information may include numeric information used to generate a routine that is not included in the natural language sentence.

[0244] According to one embodiment, the electronic device further includes a display, wherein when the instructions are executed individually or collectively by the at least one processor, the electronic device may, upon obtaining the routine generation data, display a UI screen containing numerical information used for routine generation through the display, modify the routine generation data based on user input through the UI screen, and generate the routine based on the modified routine generation data.

[0245] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the electronic device may generate a routine executable by the electronic device by setting an execution condition of the routine based on condition information included in the routine generation data and setting an operation to be performed under the execution condition based on operation information included in the routine generation data.

[0246] According to one embodiment, a control method for an electronic device may include: an operation of identifying conditions and operations included in a natural language sentence when a user request including a natural language sentence is received; an operation of obtaining first data corresponding to the conditions and second data corresponding to the operations; an operation of obtaining routine generation data based on the first data, the second data and information about the electronic device; and an operation of generating the electronic device to automatically execute a specific operation based on the routine generation data when information identified in the electronic device corresponds to a specific condition.

[0247] According to one embodiment, the routine may be at least one set of program code and instructions that cause the electronic device to automatically execute a specific operation when information identified in the electronic device corresponds to a specific condition.

[0248] According to one embodiment, the control method may further include an operation in which the electronic device automatically executes the specific operation when information identified based on at least one of data received through the communication circuit or data acquired through the sensor based on the routine corresponds to the specific condition.

[0249] According to one embodiment, the operation of obtaining first data corresponding to the condition and second data corresponding to the operation may include: an operation of searching a first database storing vector data for the condition to obtain the first data including a plurality of first vector data related to the condition included in the natural language sentence; and an operation of searching a second database storing vector data for the operation to obtain the second data including a plurality of second vector data related to the operation included in the natural language sentence.

[0250] According to one embodiment, the operation of identifying conditions and actions included in the natural language sentence may include the operation of identifying conditions and actions included in the natural language sentence by inputting a first prompt to an artificial intelligence model to output text corresponding to the natural language sentence as conditions and actions.

[0251] According to one embodiment, the operation of identifying conditions and actions included in the natural language sentence may include converting the natural language sentence into a preset language and identifying conditions and actions included in the natural language sentence converted into the preset language. The preset language may be a language determined based on the processing language of the artificial intelligence model.

[0252] According to one embodiment, the operation of acquiring the routine generation data may include the operation of acquiring the routine generation data by inputting a second prompt to an artificial intelligence model to output the routine generation data based on text corresponding to the natural language sentence, the first data, the second data, and a pre-set routine generation data related template.

[0253] According to one embodiment, the routine generation data may include detailed parameter information related to at least one of the conditions or actions included in the natural language sentence.

[0254] According to one embodiment, the detailed parameter information may include numeric information used to generate a routine that is not included in the natural language sentence.

[0255] According to one embodiment, when the routine generation data is obtained, the method may further include: an operation of displaying a UI screen containing numerical information used for routine generation; and an operation of modifying the routine generation data based on user input through the UI screen and generating the routine based on the modified routine generation data.

[0256] According to one embodiment, a non-transient computer-readable medium storing instructions that cause the electronic device to perform an operation when executed by a processor of the electronic device may include: an operation of identifying a condition and an operation included in the natural language sentence when a user request including a natural language sentence is received; an operation of obtaining first data corresponding to the condition and second data corresponding to the operation; an operation of obtaining routine generation data based on the first data, the second data and information about the electronic device; and an operation of generating a routine based on the routine generation data that causes the electronic device to automatically execute a specific operation if the information identified in the electronic device corresponds to a specific condition.

[0257] According to one example, by using user utterances that are used identically for conditions and actions (e.g., both a WiFi On / Off condition and a WiFi On / Off action exist), it can be tested whether the vector DBs of conditions and actions are separated; if normal operation is achieved, it can be inferred that the same method as the present disclosure was used.

[0258] According to one example, if normal operation is achieved by utilizing utterances that are unlikely to be properly searched when searching a vector DB with the user utterance itself (e.g., "If you connect to the home internet, please XX" -> the condition needs to be changed to "WiFi connection" which is not in the utterance to search), it can be inferred that the same method as the present disclosure was used.

[0259] According to one example, if detailed parameters not included in the user utterance are included in the generated routine, it can be inferred that the same method as the present disclosure was used.

[0260] According to the various embodiments described above, routines can be generated through a user's natural language commands, allowing for easy creation of routines without prior knowledge of routine generation applications. For example, this reduces the time required for a user to generate routines and lowers the cognitive cost of remembering the locations of conditions and actions for input. Consequently, the user can utilize electronic devices more efficiently.

[0261] According to the various embodiments described above, since a routine is generated by searching the vector database separately for each condition and vector, the accuracy of routine generation can be improved. Accordingly, the stability of the system can be improved.

[0262] Although the various embodiments described above use multiple individual neural network models, the operation of at least two of the multiple neural network models may be implemented in a single neural network model.

[0263] Each operation according to the various embodiments described above may be performed by the processor (110), but if necessary, a module for each operation may be used. For example, each module may be implemented with at least one software, at least one hardware, and / or a combination thereof. Each module may be implemented to use a predefined algorithm, a predefined formula, and / or a learned artificial intelligence model to perform the operation. However, at least some modules may be distributed to external devices.

[0264] The methods according to the various embodiments of the present disclosure described above may be implemented in the form of an application that can be installed on an existing electronic device. Alternatively, the methods according to the various embodiments of the present disclosure described above may be performed using a deep learning-based artificial neural network (or deep artificial neural network), that is, a learning network model.

[0265] The methods according to the various embodiments of the present disclosure described above can be implemented by software upgrades or hardware upgrades alone for existing electronic devices.

[0266] The various embodiments of the present disclosure described above may also be performed through an embedded server equipped in an electronic device or an external server of the electronic device.

[0267] According to a specific example of the present disclosure, the various embodiments described above may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions. When instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or by using other components under the control of the processor. Instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means only that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

[0268] Additionally, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0269] Additionally, each component (e.g., module or program) according to the various embodiments described above may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in the various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the functions performed by each of the respective components prior to integration in the same or similar manner. The operations performed by the module, program, or other components according to the various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added.

[0270] Although an embodiment of the present disclosure has been illustrated and described above, the embodiments are not limited to the specific embodiment described above. It is understood that various modifications can be made by those skilled in the art without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In an electronic device (100), Memory (120) for storing instructions; and It includes at least one processor (110) including a processing circuitry; and When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When a user request containing a natural language sentence is received, the condition and action included in the natural language sentence are identified, and Acquire first data corresponding to the above conditions and second data corresponding to the above operation, and Routine generation data is obtained based on the first data, the second data, and information about the electronic device, and An electronic device that generates a routine to cause the electronic device to automatically execute a specific operation when information identified in the electronic device corresponds to a specific condition, based on the routine generation data.

2. In Paragraph 1, The above routine is, It is at least one set of program code and instructions that causes the electronic device to automatically execute a specific operation when information identified in the electronic device corresponds to a specific condition, and The above electronic device is, Communication circuit; and It further includes a sensor; When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that causes the electronic device to automatically execute the specific operation when information identified based on at least one of data received through the communication circuit or data acquired through the sensor based on the above routine corresponds to the specific condition.

3. In Paragraph 1 or 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, A first database storing vector data for conditions is searched to obtain the first data including a plurality of first vector data related to conditions included in the natural language sentence, and An electronic device that searches a second database storing vector data for an action to obtain said second data including a plurality of second vector data related to an action included in said natural language sentence.

4. In Paragraph 1 or 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that inputs a first prompt, which outputs text corresponding to the above natural language sentence by distinguishing it into conditions and actions, into an artificial intelligence model to identify the conditions and actions included in the above natural language sentence.

5. In Paragraph 4, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, Converts the above natural language sentence into a pre-set language and identifies conditions and actions included in the natural language sentence converted into the pre-set language, and The language previously set above is, An electronic device, which is a language determined based on the processing language of the above artificial intelligence model.

6. In Paragraph 1 or 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that inputs a second prompt to an artificial intelligence model to output routine generation data based on text corresponding to the above natural language sentence, the above first data, the above second data, and a pre-set routine generation data related template, thereby obtaining the routine generation data.

7. In Paragraph 1 or 2, The above routine generation data is, An electronic device comprising detailed parameter information related to at least one of the conditions or actions included in the above natural language sentence.

8. In Paragraph 7, The above detailed parameter information is, An electronic device comprising numeric information used to generate a routine not included in the above natural language sentence.

9. In Paragraph 8, It further includes a display; When the above instructions are executed individually or collectively by the at least one processor, the electronic device, When the above routine generation data is obtained, a UI screen including numerical information used for the above routine generation is displayed through the above display, and An electronic device that modifies routine generation data based on user input through the above UI screen and generates the routine based on the modified routine generation data.

10. In Paragraph 1 or 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device, An electronic device that generates a routine executable on the electronic device by setting an execution condition of the routine based on condition information included in the routine generation data, and setting an operation to be performed under the execution condition based on operation information included in the routine generation data.

11. In a method for controlling an electronic device, When a user request including a natural language sentence is received, an action to identify conditions and actions included in the natural language sentence; An operation to acquire first data corresponding to the above conditions and second data corresponding to the above operation; An operation to obtain routine generation data based on the first data, the second data, and information about the electronic device; and A control method comprising: generating a routine that causes the electronic device to automatically execute a specific operation when information identified in the electronic device corresponds to a specific condition, based on the routine generation data.

12. In Paragraph 11, The above routine is, It is at least one set of program code and instructions that causes the electronic device to automatically execute a specific operation when information identified in the electronic device corresponds to a specific condition, and The above control method is, A control method further comprising: an operation in which, based on the above routine, the electronic device automatically executes the specific operation when information identified based on at least one of data received through the communication circuit or data acquired through the sensor corresponds to the specific condition.

13. In Paragraph 11 or 12, The operation of acquiring first data corresponding to the above conditions and second data corresponding to the above operation is, The operation of searching a first database storing vector data for conditions to obtain the first data including a plurality of first vector data related to conditions included in the natural language sentence; and A control method comprising: a second database storing vector data for an action, and an action of obtaining the second data including a plurality of second vector data related to an action included in the natural language sentence.

14. In Paragraph 11 or 12, The operation for identifying conditions and actions included in the above natural language sentence is, A control method comprising: inputting a first prompt to an artificial intelligence model to output text corresponding to the above natural language sentence, distinguishing it into conditions and actions, and identifying the conditions and actions included in the above natural language sentence.

15. A non-transient computer-readable medium storing instructions that cause said electronic device to perform an operation when executed by a processor of said electronic device, When a user request including a natural language sentence is received, an action to identify conditions and actions included in the natural language sentence; An operation to acquire first data corresponding to the above conditions and second data corresponding to the above operation; An operation to obtain routine generation data based on the first data, the second data, and information about the electronic device; and A non-transient computer-readable medium comprising: an operation to generate a routine that causes the electronic device to automatically execute a specific operation when information identified in the electronic device corresponds to a specific condition, based on the routine generation data.

Images