Electronic device and control method therefor

Abstract

An electronic device is disclosed. This electronic device comprises: a display; a memory storing at least one instruction; and a processor. On the basis of the execution of the at least one instruction, the processor: controls the display to display a plurality of UI elements respectively corresponding to a plurality of keywords included in a first prompt input from a user; when a user interaction for editing the plurality of UI elements is input, obtains a second prompt obtained by editing the first prompt on the basis of the edited UI elements; and controls the display to display an image generated by a generative AI model on the basis of the second prompt.

Description

Electronic device and control method thereof

[0001] The present disclosure relates to an electronic device and a method for controlling the same.

[0002] Recently, smart devices supporting AI functions are being developed and distributed. In particular, generative AI models are artificial intelligence models that generate images corresponding to input prompts. When a user enters a prompt, a smart device can input that prompt into a generative AI model and provide the generated image.

[0003] Based on the prompts entered by the user, unintended results may be produced, such as highlighting unimportant parts or failing to achieve the desired layout. Consequently, users experience inconvenience due to the difficulty in predicting the outcome generated through the prompts.

[0004] Therefore, the need for technology that allows prompts to be edited more effectively has emerged.

[0005] An electronic device according to the present disclosure comprises a display, a memory storing at least one instruction, and a processor, wherein the processor controls the display to display a plurality of UI elements corresponding to each of a plurality of keywords included in a first prompt received from a user based on the execution of the at least one instruction, and when a user interaction for editing the plurality of UI elements is received, the processor obtains a second prompt that edits the first prompt based on the edited plurality of UI elements, and controls the display to display an image generated by a generative AI model based on the second prompt.

[0006] In this case, the system further includes a communication unit for performing communication with an external device that stores an artificial intelligence model trained to edit a prompt based on the display state of the UI element, and the processor transmits information regarding the first prompt and the edited plurality of UI elements to the external device through the communication unit, and receives the second prompt obtained by inputting information regarding the first prompt and the edited plurality of UI elements into the artificial intelligence model from the external device through the communication unit, and the information regarding the plurality of UI elements may include at least one of the size, position, color, arrangement, and relationship between UI elements of each UI element.

[0007] In this case, the processor controls the display to decrease or increase the size of the UI element when an interaction of pinching in or pinching out is input to one of the plurality of UI elements, and the artificial intelligence model can be trained to convert the first prompt into the second prompt by adjusting the weight for the keyword corresponding to each element based on the size change of each UI element.

[0008] Meanwhile, when a drag interaction is input in which one of the plurality of UI elements is touched and dragged, the processor controls the display to move and display the UI element according to the direction of the drag interaction, and the artificial intelligence model can be trained to convert the first prompt into the second prompt by adjusting the relationship between the UI element and keywords corresponding to the other UI element based on the distance and positional relationship with at least one other UI element that changes according to the movement of the UI element.

[0009] Meanwhile, when a drag interaction is input in which one of the plurality of UI elements is touched and dragged to a position where at least a part of it overlaps with another UI element, the processor controls the display to display at least a part of the UI element and the other element overlapping according to the direction of the drag interaction, and the artificial intelligence model can be trained to convert the first prompt into the second prompt by adjusting the relationship between keywords corresponding to the UI element and the other element according to the degree of overlap between the plurality of UI elements.

[0010] Meanwhile, when an interaction to change the color of one of the plurality of UI elements is input, the processor controls the display to change the color of the UI element, and the artificial intelligence model can be trained to convert the first prompt into the second prompt by reflecting information about the UI element whose color has been modified.

[0011] Meanwhile, when an interaction is input that rotates one of the plurality of UI elements, the processor controls the display to rotate the UI element, and the artificial intelligence model can be trained to convert the first prompt into the second prompt by reflecting information about the rotated UI element.

[0012] In this case, the processor controls the display to add the new UI element to the plurality of UI elements and display it when a user interaction for adding a new UI element is input, and the artificial intelligence model can be trained to convert the first prompt into the second prompt by reflecting information about the new UI element.

[0013] Meanwhile, the processor transmits the second prompt to the external device where the generative AI model is stored through the communication unit, receives the generated image obtained by inputting the second prompt into the generative AI model from the external device through the communication unit, and the generative AI model can be trained to output an image corresponding to the second prompt.

[0014] Meanwhile, the memory stores an artificial intelligence model trained to convert a prompt based on the display state of the UI element, and the processor can input information about the first prompt and the edited UI element into the artificial intelligence model to obtain the second prompt, and control the display to display an image generated by a generative AI model based on the second prompt.

[0015] A control method for an electronic device according to the present disclosure includes the steps of: displaying a plurality of UI elements corresponding to each of a plurality of keywords included in a first prompt received from a user; when a user interaction for editing the plurality of UI elements is input, obtaining a second prompt in which the first prompt is edited based on the edited UI elements; and displaying an image generated by a generative AI model based on the second prompt.

[0016] In this case, the step of converting the first prompt into a second prompt includes the step of transmitting information regarding the first prompt and the edited plurality of UI elements to an external device, and the step of receiving the second prompt obtained by inputting the information regarding the first prompt and the edited plurality of UI elements into an artificial intelligence model from the external device, wherein the artificial intelligence model is trained to convert the prompt based on the display state of the UI elements, and the information regarding the edited plurality of UI elements may include at least one of the size, position, color, arrangement, and relationship between UI elements of each UI element.

[0017] In this case, when an interaction of pinching in or pinching out is input to one of the plurality of UI elements, the method further includes the step of displaying the UI element by decreasing or increasing its size, and the artificial intelligence model can be trained to convert the first prompt into the second prompt by adjusting the weight for the keyword corresponding to each element based on the size change of each UI element.

[0018] Meanwhile, when a drag interaction is input by touching and dragging one of the plurality of UI elements, the method further includes the step of moving and displaying the UI element according to the direction of the drag interaction, and the artificial intelligence model can be trained to convert the first prompt into the second prompt by adjusting the relationship between the UI element and keywords corresponding to the other UI element based on the distance and positional relationship with at least one other UI element that changes according to the movement of the UI element.

[0019] Meanwhile, when a drag interaction is input in which one of the plurality of UI elements is touched and dragged to a position where at least a portion overlaps with another UI element, the method further includes the step of displaying an overlap of at least a portion of the UI element and the other element according to the direction of the drag interaction, and the artificial intelligence model can be trained to convert the first prompt into the second prompt by adjusting the relationship between keywords corresponding to the UI element and the other element according to the degree of overlap between the plurality of UI elements.

[0020] Meanwhile, when an interaction that changes the color of one of the plurality of UI elements is input, the method further includes a step of changing the color of the UI element, and the artificial intelligence model can be trained to convert the first prompt into the second prompt by reflecting information about the UI element whose color has been modified.

[0021] Meanwhile, when an interaction that rotates one of the plurality of UI elements is input, the method further includes a step of rotating the UI element, and the artificial intelligence model can be trained to convert the first prompt into the second prompt by reflecting information about the rotated UI element.

[0022] In this case, when a user interaction for adding a new UI element is input, the method further includes the step of adding and displaying the new UI element to the plurality of UI elements, and the artificial intelligence model can be trained to convert the first prompt into the second prompt by reflecting information about the new UI element.

[0023] Meanwhile, the step of converting the first prompt into a second prompt includes the step of transmitting the second prompt to an external device where the generative AI model is stored, and the step of receiving the generated image obtained by inputting the second prompt into the generative AI model at the external device, and the generative AI model may be trained to output an image corresponding to the second prompt.

[0024] Meanwhile, the step of converting the first prompt into a second prompt may involve inputting information about the first prompt and the edited UI element into an artificial intelligence model to obtain the second prompt, and the artificial intelligence model may convert the prompt based on the display state of the UI element.

[0025] FIG. 1 is a schematic diagram illustrating the operation of an electronic device according to at least one embodiment of the present disclosure.

[0026] FIG. 2 is a block diagram briefly illustrating the configuration of an electronic device according to at least one embodiment of the present disclosure.

[0027] FIG. 3 is a block diagram illustrating the detailed configuration of an electronic device according to at least one embodiment of the present disclosure.

[0028] FIG. 4 is a drawing showing the operation of an electronic device and an external device according to at least one embodiment of the present disclosure.

[0029] FIG. 5 is a drawing showing a case where the size of a UI element is adjusted according to at least one embodiment of the present disclosure.

[0030] FIG. 6 is a drawing showing a case where a UI element is dragged to move its position according to at least one embodiment of the present disclosure.

[0031] FIG. 7 is a drawing showing a case where a UI element is dragged and overlapped with another UI element according to at least one embodiment of the present disclosure.

[0032] FIG. 8 is a drawing showing a case where the color of a UI element is changed according to at least one embodiment of the present disclosure.

[0033] FIG. 9 is a drawing showing a case where a UI element is rotated according to at least one embodiment of the present disclosure.

[0034] FIG. 10 is a drawing showing a case where a new UI element is added according to at least one embodiment of the present disclosure.

[0035] FIG. 11 is a flowchart illustrating a method for controlling an electronic device according to at least one embodiment of the present disclosure.

[0036] The terms used in the embodiments of this disclosure have been selected to be as widely used and general as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section of this disclosure. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0037] In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, operations, or components such as parts) and do not exclude the presence of additional features.

[0038] The expression "at least one of A or / and B" should be understood as representing either "A" or "B" or "A and B".

[0039] Expressions such as "first," "second," "first," or "second" used in this specification may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0040] Where it is stated that a component (e.g., Component 1) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., Component 2), it should be understood that the component may be directly connected to the other component or connected through the other component (e.g., Component 3).

[0041] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "consisting of" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0042] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor (not shown), except for a "module" or "part" that needs to be implemented in specific hardware.

[0043] In this specification, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0044] An embodiment of the present disclosure will be described in more detail below with reference to the attached drawings.

[0045] FIG. 1 is a schematic diagram illustrating the operation of an electronic device according to at least one embodiment of the present disclosure.

[0046] In FIG. 1, the electronic device (100) is depicted as an electronic device including a display, but various embodiments of the present disclosure may be implemented in various types of electronic devices. Even if the electronic device (100) does not directly have a display, it may be implemented as a device that provides UI elements and generated images by being connected to an external display device provided separately. For example, the electronic device (100) may be implemented in various forms such as a TV, set-top box, PC, laptop PC, smartphone, tablet PC, video wall, kiosk, etc.

[0047] Referring to FIG. 1, the electronic device (100) can receive a prompt (20) from a user and display a plurality of UI elements (30) corresponding to the prompt (20).

[0048] According to the present disclosure, "prompt" may mean input for initiating interaction with a large language model. The prompt (20) may be a text input or voice input comprising one or more texts and / or one or more sentences. In one embodiment, the prompt (20) may include natural language text. The natural language text may include various information that the large language model can use to generate a response to a user inquiry or to control a home IoT system, such as context, intent, task, and constraints. Meanwhile, the prompt (20) may be referred to by various expressions representing the same or similar concepts. The prompt (20) can be replaced with expressions such as, for example, "input," "user input," "input phrase," "user command," "directive," "starting sentence," "task query," "trigger sentence," "message," etc., but is not limited to the examples mentioned above. FIG. 1 illustrates a case where the prompt (20) "a cat wearing blue sunglasses is drinking a mojito on the beach" is input by the user.

[0049] The electronic device (100) can receive a prompt (20) from a user. For example, the electronic device (100) can receive a prompt (20) from a user through a keyboard connected to the electronic device (100). Another example is that the electronic device (100) can receive a prompt (20) by receiving touch input from a user through a touch screen that includes a display. Alternatively, the electronic device (100) can receive a voice signal from a user through a microphone and obtain a prompt (20) by obtaining text information corresponding to the voice signal using an automatic speech recognition model (ASR model) included in the electronic device (100) or a speech recognition model included in an external device.

[0050] The electronic device (100) can display an image (40) generated by inputting a prompt (20) into a generative AI model. According to one embodiment, the electronic device (100) transmits the prompt (20) to an external device (200) where the generative AI model is stored, and receives an image (40) generated by inputting the prompt (20) into the generative AI model from the external device (200). The electronic device (100) can display the received image (40). In another example, the electronic device (100) can display an image (40) generated by inputting the prompt (20) into a generative AI model stored in the electronic device (100). In this case, the generative AI model may be a model trained to generate an image (40) when the prompt (20) is input.

[0051] Meanwhile, the electronic device (100) can edit the prompt (20) according to user input. The image generated through the prompt (20) entered by the user may not reflect the user's intention. The user must regenerate the image (40) by modifying the prompt (20) until an image reflecting the intention is obtained.

[0052] The electronic device (100) may provide a mode in which a user can modify a prompt. Referring to FIG. 1, a UI element (11) corresponding to the Bubble mode and a UI element (12) corresponding to the Prompt modification mode are displayed. The Bubble mode may be a mode in which a user can edit a circular UI element (30) provided in a bubble form. By adjusting the size or placement of the UI element (30), the user can more easily reflect their intention and predict the result. The electronic device (100) may display a UI for editing the UI element (30) in the Bubble mode (11). Referring to FIG. 1, the electronic device (100) can display a UI (14) that can receive an image of a keyword corresponding to a UI element (30) from a user, a UI (16) that can add or change the color of the UI element (30), a UI (15) that can select a wider variety of colors for the UI element (30), and a UI (17) that can delete the UI element (30). However, this is merely one example of a UI for editing the UI element (30) and is not limited to the illustration of FIG. 1.

[0053] The prompt modification mode may be a mode in which the user can directly modify the prompt (20). The electronic device (100) may receive the modified prompt from the user through an input interface. In another embodiment, the electronic device (100) may visually display the result of modifying the keyword included in the prompt (20). For example, the color of the text corresponding to the keyword included in the prompt (20) may be changed or the size of the text may be increased to display it.

[0054] The UI element (30) may be a UI that visually represents the keywords included in the prompt (20) and the relationships between the keywords. Alternatively, the UI element (30) may be a bubble, a graphic object, or a UI element. For example, the UI element (30) may be a circular bubble containing the keywords. As another example, the UI element (30) may be a shape other than a circle (e.g., a square, etc.).

[0055] The electronic device (100) can identify multiple keywords included in the input prompt (20) and display multiple UI elements (30) corresponding to the multiple keywords.

[0056] Keywords may refer to words that can be generated as images (40) on the prompt (20). Keywords may include objects such as people, animals, or things, and may include backgrounds such as the space or environment where the objects are located. Keywords regarding objects may appear as subjects on the prompt (20) and may be central elements that perform specific actions or have a presence. Additionally, keywords regarding objects may be elements that are connected to actions or describe features such as appearance, size, or color through modifiers. On the other hand, keywords regarding backgrounds may be spatial or environmental elements where objects are located, identified through words or prepositional phrases that describe the location or environment. For example, when a user inputs the prompt (20) "A cat wearing blue sunglasses is drinking a mojito on the beach," the electronic device (100) may identify "sunglasses," "cat," and "mojito" as keywords regarding objects and "beach" as keywords regarding backgrounds.

[0057] Additionally, the electronic device (100) can identify information regarding the attributes of multiple keywords. The information regarding the attributes of multiple keywords may be information regarding the characteristics of each keyword or the relationships between keywords. The electronic device (100) can identify information regarding the attributes of multiple keywords through modifiers that describe the keywords included in the prompt (20). For example, the electronic device (100) can identify information regarding the attributes of keywords through expressions that describe keywords, such as color or size, or expressions that describe the positional relationships between keywords, such as 'side', 'top / bottom', 'front / back', etc. Alternatively, the electronic device (100) can identify information regarding the relationships between keywords through expressions that indicate actions. For example, when the user inputs the prompt (20) "A cat wearing blue sunglasses is drinking a mojito on the beach," the electronic device (100) can identify information regarding the color "blue" for "sunglasses." In addition, in the prompt (20), it can be identified that the ‘cat’ is wearing ‘sunglasses’ and that the ‘cat’ is drinking ‘mojito’.

[0058] According to one embodiment, the electronic device (100) can identify a plurality of keywords and the relationship between the keywords through an artificial intelligence model trained to identify a plurality of keywords included in the prompt (20).

[0059] The electronic device (100) can display multiple UI elements (30) corresponding to multiple identified keywords. At this time, the electronic device (100) can display multiple UI elements (30) corresponding to information regarding the attributes of the multiple keywords. Specifically, the electronic device (100) can identify at least one of size, position, and color corresponding to the attributes of the multiple keywords and display multiple UI elements (30) corresponding to each of the multiple keywords. For example, the electronic device (100) can display four UI elements (30) each containing the keywords 'sunglasses', 'cat', 'beach', and 'mojito'. Additionally, the UI element (30) corresponding to 'sunglasses' can be displayed in blue so that it corresponds to color information of 'blue' for 'sunglasses'.

[0060] The electronic device (100) can receive user interactions for editing a plurality of UI elements (30). User interactions for editing a plurality of UI elements (30) may include at least one of pinch-in or pinch-out interactions, drag interactions, color change interactions, rotation interactions, and UI element addition interactions. When the electronic device (100) receives user interactions for editing UI elements (30), it can adjust and display the UI elements (30) to correspond to the interaction. A detailed description of each interaction will be provided later.

[0061] The electronic device (100) can convert the prompt (20) based on information regarding a plurality of edited UI elements (30). Specifically, the electronic device (100) can convert the prompt (20) through an artificial intelligence model stored in the electronic device (100) or an external device (200) using information regarding the prompt (20) and the plurality of edited UI elements (30).

[0062] The electronic device (100) can display the generated image (40) by inputting the converted prompt (20) into a generative AI model stored in the electronic device (100) or an external device (200).

[0063] FIG. 2 is a block diagram briefly illustrating the configuration of an electronic device (100) according to at least one embodiment of the present disclosure.

[0064] As illustrated in FIG. 2, the electronic device (100) according to the present disclosure may include a display (110), a memory (120), and a processor (130).

[0065] The display (110) is configured to display a plurality of UI elements (30) and a generated image (40) under the control of the processor (130). The display (110) can be implemented as an LCD (Liquid Crystal Display Panel), OLED (Organic Light Emitting Diodes), etc., and the display (110) can also be implemented as a flexible display, a transparent display, etc. depending on the case. However, the display (110) according to the present disclosure is not limited to a specific type. Although FIG. 2 is illustrated as if the electronic device (100) directly embeds the display (110), this can be interpreted to include not only cases where the display (110) is actually mounted on the electronic device (100) (e.g., TV, kiosk, smartphone, laptop PC, tablet PC, etc.), but also cases where it is connected to a separately provided display device (e.g., monitor, TV, beam projector, electronic whiteboard, etc.) via various wired or wireless communication methods (e.g., set-top box, PC, server, etc.).

[0066] In one or more embodiments of the present disclosure, the display (110) may display a plurality of UI elements (30) and an image generated through a generative AI model. Specifically, it may display a prompt (20) received from a user and a UI element (30) corresponding to a keyword included in the prompt (20).

[0067] At least one instruction regarding an electronic device (100) may be stored in the memory (120). Additionally, an operating system (O / S) for operating the electronic device (100) may be stored in the memory (120). Furthermore, various software programs or applications for operating the electronic device (100) may be stored in the memory (120) according to various embodiments of the present disclosure. Additionally, the memory (120) may be implemented as volatile memory such as S-RAM (Static Random Access Memory) or D-RAM (Dynamic Random Access Memory), non-volatile memory such as Flash Memory, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), or EEPROM (Electrically Erasable Programmable Read Only Memory), a hard disk drive (HDD), or a solid-state drive (SSD).

[0068] Specifically, various software modules for operating an electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (120), and the processor (130) may control the operation of the electronic device (100) by executing the various software modules stored in the memory (120). That is, the memory (120) is accessed by the processor (130), and reading / writing / modifying / deleting / updating of data by the processor (130) may be performed.

[0069] The memory (120) may be a configuration provided separately from the processor (130), may be an internal memory built into the processor (130), and may also be used to include a memory card (not shown) (e.g., micro SD card, memory stick) or an external hard drive mounted on the electronic device (100).

[0070] In one or more embodiments of the present disclosure, the memory (120) may store a prompt received from a user, information about a plurality of UI elements corresponding to a plurality of keywords included in the prompt, and information about an image. Additionally, the memory (120) may store an identification model that identifies at least one keyword within the prompt received from the user. In one embodiment, the memory (120) may store an artificial intelligence model that converts the prompt based on the display state of the UI element and a generative AI model that generates an image through the prompt.

[0071] In addition, various information necessary within the scope of achieving the purpose of the present disclosure may be stored in the memory (120), and the information stored in the memory (120) may be updated as it is received from an external device or input by a user.

[0072] The processor (130) controls the overall operation of the electronic device (100). Specifically, the processor (130) is connected to the configuration of the electronic device (100) including a display (110) and a memory (120), and can control the overall operation of the electronic device (100) by executing at least one instruction stored in the memory (120) as described above.

[0073] The processor (130) can be implemented in various ways. For example, the processor (130) may include or be defined by one or more of a central processing unit (CPU) that processes digital signals, a Micro Controller Unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), or an ARM processor. Additionally, the processor (130) may be implemented as a System on Chip (SoC) or Large Scale Integration (LSI) with built-in processing algorithms, or as a Field Programmable Gate Array (FPGA). The processor (130) can perform various functions by executing computer executable instructions stored in memory (120).

[0074] The processor (130) can perform a method according to one or more embodiments of the present disclosure based on the execution of at least one instruction stored in memory (120).

[0075] According to one embodiment, the processor (130) can control the display to display a plurality of UI elements corresponding to each of the plurality of keywords included in the first prompt received from the user.

[0076] According to one embodiment, when a user interaction for editing a plurality of UI elements is input to the processor (130), the processor can obtain a second prompt that edits the first prompt based on the edited plurality of UI elements.

[0077] According to one embodiment, the processor (130) can control the display to display an image generated from a generative AI model based on a second prompt.

[0078] FIG. 3 is a block diagram illustrating the detailed configuration of an electronic device (100).

[0079] As illustrated in FIG. 3, the electronic device (100) may further include a communication unit (140) and an operation interface unit (150), in addition to a display (110), memory (120), and processor (130). However, the configurations illustrated in FIG. 2 and FIG. 3 are merely exemplary, and it is understood that in carrying out the present disclosure, new configurations may be added or some configurations may be omitted in addition to the configurations illustrated in FIG. 2. Since the basic operation description and specific examples of the display (110), memory (120), and processor (130) among the configurations of FIG. 3 have already been described in FIG. 2, a redundant description is omitted.

[0080] The communication unit (140) includes a circuit and can perform communication with an external device (200). Specifically, the processor (130) can receive various data or information from an external device (200) connected through the communication unit (140), and can also transmit various data or information to the external device (200).

[0081] The communication unit (140) may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, an NFC module, and a UWB (Ultra Wide Band) module. Specifically, the WiFi module and the Bluetooth module can each perform communication using the WiFi method and the Bluetooth method. When using the WiFi module or the Bluetooth module, various connection information such as SSID is first transmitted and received, and then various information can be transmitted and received after establishing a communication connection using this information.

[0082] In addition, the wireless communication module can perform communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), and 5G (5th Generation). Furthermore, the NFC module can perform communication using the NFC (Near Field Communication) method, which utilizes the 13.56 MHz band among various RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860~960 MHz, and 2.45 GHz. Additionally, the UWB module can accurately measure the Time of Arrival (ToA), which is the time it takes for a pulse to reach a target, and the Angle of Arrival (AoA), which is the angle of arrival of the pulse at the transmitting device, through communication between UWB antennas. Accordingly, precise distance and location recognition within an error range of tens of centimeters indoors is possible.

[0083] In various embodiments of the present disclosure, when an artificial intelligence model is stored in a separate external device (200), the communication unit (140) can transmit information regarding the first prompt and UI elements as described above to the external device (200). Additionally, the communication unit (140) can receive a second prompt modified through the artificial intelligence model from the external device (200).

[0084] In various embodiments of the present disclosure, when a generative AI model is stored in a separate external device (200), the communication unit (140) can transmit a prompt as described above to the external device (200) and receive an image generated through the generative AI model from the external device (200).

[0085] The operation interface (150) is configured to receive various user commands directly from the user. The operation interface (150) can be implemented as a touchscreen, touchpad, button, etc. For example, if implemented as a touchscreen, the user can edit UI elements (30) by touching the touchscreen and pinching out or pinching in, or dragging. In addition, the user can also input keywords to be added through an input window displayed on the touchscreen.

[0086] FIG. 4 is a drawing showing the operation of an electronic device (100) and an external device (200) according to at least one embodiment of the present disclosure.

[0087] FIG. 4 illustrates a method in which an electronic device (100) transmits and receives information to and from an external device (200) that stores an artificial intelligence model and a generative AI model.

[0088] Here, an artificial intelligence model refers to a model that inputs a specific input value into a specific function based on learned data to produce an output value. Here, the artificial intelligence model may be a model trained to transform a prompt based on the display state of multiple UI elements. The artificial intelligence model may be referred to in various ways, such as a neural network model, a deep learning model, a neural network model, or a generative artificial intelligence model, but in this disclosure, it will be collectively referred to as an artificial intelligence model.

[0089] Meanwhile, the artificial intelligence-related function according to the present disclosure can be operated by including an artificial intelligence model in an external device, transmitting an input value to an external device through a communication unit (140), and receiving an output value.

[0090] Alternatively, an artificial intelligence model may be stored in the memory (120) of the electronic device (100) and operated through the processor (130) and the memory (120).

[0091] The processor (130) can be controlled to process input data according to predefined operation rules or artificial intelligence models stored in memory (120). Alternatively, if the processor (130) is an artificial intelligence-dedicated processor, the artificial intelligence-dedicated processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model. The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0092] Here, "created through learning" means that a basic artificial intelligence model is trained using multiple learning data by a learning algorithm, thereby creating a predefined rule of operation or an artificial intelligence model configured to perform a desired characteristic (or objective). Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0093] An artificial intelligence model can be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through calculations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights can be updated during the learning process so that the loss or cost values ​​obtained by the artificial intelligence model are reduced or minimized.

[0094] Artificial neural networks may include deep neural networks (DNNs), such as, but are not limited to, Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), Generative Adversarial Networks (GANs), or Deep Q-Networks.

[0095] The external device (200) may store an artificial intelligence model and a generative AI model. The artificial intelligence model may be a model trained to convert a prompt (20) based on the display state of a UI element (30). The generative AI model may be a model trained to generate an image corresponding to the prompt (20) by inputting the prompt (20). However, the generative AI model is not limited to generating images, and may be a model trained to output search results or answers corresponding to the prompt (20). Below, the generative AI model is described as a case where an image is generated.

[0096] Alternatively, an external device (200) may store an artificial intelligence model trained to convert a prompt (20) and generate an image through the converted prompt (20). In this case, the electronic device (100) transmits the prompt (20) to the external device (200), converts the prompt (20) through the artificial intelligence model stored in the external device (200), and generates an image corresponding to the converted prompt, thereby receiving the converted prompt and the generated image from the external device (200).

[0097] According to one embodiment, an artificial intelligence model and a generative AI model may be stored together in a single external device (200). In this case, the single external device (200) can convert a prompt (20) through the artificial intelligence model and generate an image through the generative AI model. According to another embodiment, an artificial intelligence model and a generative AI model may be stored in two different external devices (200), respectively. In this case, the electronic device (100) can receive a prompt converted through the artificial intelligence model stored in the first external device (200), transmit the received prompt to the second external device (200), and receive an image generated through the generative AI model stored in the second external device (200).

[0098] In the following description, an artificial intelligence model trained to convert a prompt (20) and a generative AI model trained to generate an image through the prompt (20) are each stored in an external device (200).

[0099] The electronic device (100) can display a plurality of UI elements corresponding to a first prompt received from a user (S410). Specifically, the electronic device (100) can identify a plurality of keywords included in the first prompt received from a user and display a plurality of UI elements corresponding to the identified plurality of keywords.

[0100] The electronic device (100) can receive user interactions for editing multiple UI elements (S420). User interactions for editing multiple UI elements may include at least one of pinch-in or pinch-out interactions, drag interactions, color change interactions, rotation interactions, and UI element addition interactions. When the electronic device (100) receives user interactions for editing UI elements, it can adjust and display UI elements to correspond to the interaction.

[0101] A pinch-out interaction may be an interaction in which two points are touched and then moved so that the distance between the two points increases. Additionally, a pinch-in interaction may be an interaction in which two points are touched and then moved so that the distance between the two points decreases. When a pinch-out interaction is input, the electronic device (100) can increase the size of the UI element by the distance that has increased according to the interaction. Conversely, when a pinch-in interaction is input, the electronic device (100) can decrease the size of the UI element by the distance that has decreased according to the interaction.

[0102] A drag interaction may be an interaction in which a user touches one of a plurality of UI elements and then moves the UI element in a specific direction while maintaining the touch state. The electronic device (100) may indicate that the UI element moves in a specific direction according to the drag interaction.

[0103] Meanwhile, the electronic device (100) can receive a drag interaction input from the user and display the UI element moving to a position where it overlaps with at least a part of another UI element.

[0104] The color change interaction may be an interaction in which a user changes the color of one of a plurality of UI elements. Specifically, the electronic device (100) may receive input from the user regarding the color of one of a plurality of UI elements. For example, if the user touches one of a plurality of UI elements and then touches the UI for changing the color at the bottom, the electronic device (100) may display a color palette UI, and if the user touches a desired color in the color palette UI, the corresponding UI element may be displayed to change to the color selected by the user.

[0105] The rotation interaction may be an interaction in which a user touches one of a plurality of UI elements and then rotates it clockwise or counterclockwise while maintaining the touch state. When the rotation interaction is input, the electronic device (100) may display a keyword included in the UI element to rotate in a direction corresponding to the user's rotation interaction.

[0106] The UI element addition interaction may be an interaction for creating and adding a new UI element. When a user touches a UI element containing a '+' shape, the electronic device (100) may create and display a new UI element and display an input window for entering a keyword corresponding to the UI element. Subsequently, the electronic device (100) may change the created UI element into a form desired by the user through at least one of the aforementioned pinch-in or pinch-out interaction, drag interaction, color change interaction, and rotation interaction.

[0107] The electronic device (100) can transmit information about a first prompt and a plurality of UI elements to an external device (200) through a communication unit (140) (S430). Specifically, the electronic device (100) can transmit information about a plurality of UI elements edited through user interaction and the first prompt to the external device (200).

[0108] At this time, information regarding multiple UI elements may include at least one of the size, position, color, arrangement, and relationship between UI elements. Specifically, information regarding multiple UI elements may include information regarding the size, color, and position of each UI element. For example, information regarding multiple UI elements may include information regarding the ratio or size of the UI elements, information regarding the color of the UI elements, and information regarding coordinate values ​​on the screen.

[0109] Alternatively, information regarding multiple UI elements may include information regarding the arrangement and nesting between multiple UI elements. For example, information regarding multiple UI elements may include information regarding whether multiple UI elements are arranged horizontally or vertically. Alternatively, information regarding multiple UI elements may include information regarding whether multiple UI elements overlap each other, and if so, the direction or degree of overlap.

[0110] The external device (200) can input information regarding the received first prompt and multiple UI elements into an artificial intelligence model to convert the first prompt into a second prompt (S440). The artificial intelligence model stored in the external device (200) may be a model trained to convert the prompt based on information regarding multiple UI elements. For example, the artificial intelligence model may be a model trained to convert the prompt by modifying or adding expressions that modify keywords corresponding to multiple UI elements based on information regarding multiple UI elements. In this case, the information regarding multiple UI elements may include the size, position, color, arrangement of multiple UI elements, and the relationship between multiple UI elements. The artificial intelligence model may be trained by receiving information regarding multiple prompts and UI elements corresponding to the prompts.

[0111] The external device (200) can input a second prompt into a generative AI model to generate an image corresponding to the second prompt (S450). The generative AI model may be a model trained to generate an image when a prompt is input. The generative AI model may be trained by receiving multiple prompts and images corresponding to each prompt.

[0112] The electronic device (100) can receive a second prompt and a generated image from an external device (200) through a communication unit (140) (S460). Additionally, the electronic device (100) can display the received second prompt and the generated image (S470). The electronic device (100) can display the second prompt and the generated image together.

[0113] As another example, the electronic device (100) can receive a second prompt from an external device (200) through a communication unit (140). Subsequently, when the electronic device (100) receives input for image generation from a user, it transmits the second prompt to the external device (200), and the external device (200) inputs the second prompt into a generative AI model to receive and display the generated image.

[0114] Meanwhile, the user may modify the second prompt or multiple UI elements corresponding to the second prompt. Specifically, the generated image may not match the user's intention, or the second prompt may not match the user's intention. At this time, the electronic device (100) may receive input from the user regarding modifications to the second prompt or multiple UI elements corresponding to the second prompt.

[0115] The electronic device (100) can transmit a second prompt modified by the user to an external device (200) where a generative AI model is stored. The electronic device (100) can receive an image generated by inputting the second prompt modified by the external device (200) into the generative AI model and display it on a display (110).

[0116] In another embodiment, the electronic device (100) may receive a user interaction for editing a plurality of UI elements corresponding to a second prompt. The electronic device (100) may transmit information regarding the second prompt and the edited plurality of UI elements to an external device (200). The external device (200) may input the information regarding the second prompt and the edited plurality of UI elements into an artificial intelligence model to convert the second prompt into a third prompt. Subsequently, the electronic device (100) may receive an image generated by inputting the third prompt into a generative AI model from the external device (200) and display it on a display (110).

[0117] The following describes in detail user interactions for editing UI elements.

[0118] FIG. 5 is a drawing showing a case where the size of a UI element is adjusted according to at least one embodiment of the present disclosure.

[0119] The processor (130) can control the display (110) to decrease or increase the size of the UI element when an interaction (51) of pinching in or pinching out is input to one of the multiple UI elements.

[0120] FIG. 5 illustrates a UI element (31) corresponding to a first prompt (21) named 'cat' received from a user, which is displayed on a display (110) with its size increased by the user's pinch-out interaction (51).

[0121] According to FIG. 5, the processor (130) displays a UI element (31) corresponding to a first prompt (21) of ‘cat’ received from the user on the display (110). At this time, when a pinch-out interaction (51) is received from the user, the processor (130) can display a UI element (31) with a size increased according to the pinch-out interaction (51) on the display (110).

[0122] When a pinch-out or pinch-in interaction (51) is input from a user, the processor (130) can identify the location of two points touched by two fingers. The processor (130) can measure the distance between the two points and the distance between the two points after the pinch-out or pinch-in interaction (51) is performed, using coordinate information regarding the location of the two points. Subsequently, the processor (130) can measure the scaling ratio using the distance between the two points before the interaction (51) and the distance between the two points after the interaction (51). The processor (130) can fix the center of the UI element (31) and control the UI element (31) to be displayed on the display (110) by increasing or decreasing its size according to the measured scaling ratio.

[0123] The processor (130) can store information about the resized UI element (31) in memory (120). The information about the resized UI element (31) may include a numerical value for the size of the UI element (31), the proportion of the UI element (31) occupying the entire screen, and a resizing ratio. For example, in FIG. 5, the processor (130) displays the UI element (31) containing 'cat' by increasing its size to fill the screen through a pinch-out interaction (51). The processor (130) can store information about the proportion of the screen occupied by the UI element (31) containing 'cat' and the size of the UI element (31) in memory (120).

[0124] The processor (130) can transmit information about the size of the UI element and the first prompt (21) to an external device (200) where an artificial intelligence model is stored via the communication unit (140).

[0125] The artificial intelligence model stored in the external device (200) may be a model trained to convert the first prompt (21) into the second prompt (22) by adjusting the weight for the keyword corresponding to each element based on the size change of each UI element.

[0126] The artificial intelligence model can convert the first prompt (21) into the second prompt (22) based on information about the size of the UI elements. Specifically, the artificial intelligence model can convert the first prompt (21) by receiving information about the UI elements, adjusting the weight for the keyword corresponding to each UI element, and adding an expression for the keyword corresponding to the UI element. For example, according to FIG. 5, the first prompt (21) of ‘cat’ can be converted into the second prompt (22) of ‘cat filling the photo’ based on the ratio of the UI element (31) occupying the screen.

[0127] According to another example, when a first prompt, "first cat and second cat," is input, the processor (130) can control the display (110) to display a UI element containing "first cat" and a UI element containing "second cat." Additionally, if the UI element containing "first cat" is adjusted to be larger than the UI element containing "second cat" through a user's pinch-out interaction (51), the artificial intelligence model can convert the first prompt into a second prompt, "first cat and second cat smaller than first cat." However, the expression of the second prompt is merely an example and is not limited to the embodiments of the present disclosure.

[0128] The processor (130) can receive a second prompt (22) from an external device (200) via a communication unit (140). Subsequently, the processor (130) can control the display (110) to display the second prompt (22). After checking the second prompt (22), the user can edit the second prompt (22) or a plurality of UI elements corresponding to the second prompt (22). If the user edits a plurality of UI elements corresponding to the second prompt (22), the processor (130) can transmit information about the edited plurality of UI elements to the external device (200) and receive a third prompt converted by the artificial intelligence model of the external device (200). Additionally, if the user edits the second prompt (22), the processor (130) can display the UI elements corresponding to the edited second prompt (22) on the display (110). Users can visually check the content edited in the prompt through UI elements, making it easy to predict the intended result.

[0129] The processor (130) can transmit a second prompt (22) to an external device (200) through a communication unit (140) and receive an image (40) generated through a generative AI model stored in the external device (200). The generative AI model stored in the external device (200) may be a model trained to generate an image (40) through a prompt. The image (40) can be output by inputting the second prompt (22) into the generative AI model from the external device (200).

[0130] The processor (130) can control the display (110) to display an image (40) generated through a generative AI model. The processor (130) can control the display (110) to display the second prompt (22) and the generated image (40) together. Referring to FIG. 5, the display (110) displays an image corresponding to a ‘cat filling the photo’ generated through a generative AI model.

[0131] At this time, if the image (40) intended by the user is not correct, the user can obtain a new result by modifying the second prompt (22) or a plurality of UI elements (31) corresponding to the second prompt (22).

[0132] FIG. 6 is a drawing showing a case where a UI element is dragged to move its position according to at least one embodiment of the present disclosure.

[0133] FIG. 6 shows a plurality of UI elements corresponding to a first prompt (21) of ‘a cat wearing sunglasses on its head’ received from a user, which is displayed on the display (110). The processor (130) can control the display (110) to identify ‘sunglasses’ and ‘cat’ as keywords in the first prompt (21) and display UI elements containing each keyword.

[0134] At this time, the processor (130) can control the display (110) to identify the relationship between multiple keywords and to display multiple UI elements corresponding to the identified relationship. Specifically, the processor (130) can control the display (110) to arrange and display multiple UI elements corresponding to the relationship between the identified multiple keywords. For example, the processor (130) can identify a connection relationship between ‘cat’ and ‘sunglasses’ in the first prompt (21) of ‘cat wearing sunglasses on its head’, such that ‘cat’ is ‘wearing sunglasses on its head’. The processor (130) can arrange and display UI elements corresponding to the connection relationship between the identified keywords. Referring to FIG. 6, a UI element (32) containing ‘sunglasses’ is displayed overlapping a UI element (31) containing ‘cat’.

[0135] When a drag interaction is input in which one of the multiple UI elements is touched and dragged, the processor (130) can control the display (110) to move and display the UI element according to the direction of the drag interaction.

[0136] Referring to FIG. 6, a drag interaction (52) can be input from the user to a UI element (32) including ‘sunglasses’.

[0137] When a drag interaction (52) is input from a user, the processor (130) can identify the location of the point touched by the user. The processor (130) can identify the direction in which the user dragged from the starting position by continuously tracking the coordinate information of the location of the moving touch point while the user is dragging. Afterward, when the drag interaction (52) ends, the processor (130) can identify the coordinates of the ending position. The processor (130) can set the path of movement of the UI element (32) by reflecting the delta value of the coordinates of the starting position and the location of the moving touch point to the center point of the UI element (32) and can control it to be displayed on the display (110).

[0138] The processor (130) can store information about the moved UI element (32) in memory (120). Information about the position of the moved UI element (32) may include coordinates of the moved position of the UI element (32) and relative positional relationships with other UI elements. For example, in FIG. 6, a UI element (32) containing 'sunglasses' moved to the left of a UI element (31) containing 'cat' through a drag interaction (52). The processor (130) can store information about the coordinates of the position of the UI element (32) containing 'sunglasses', the distance from the UI element (31) containing 'cat', and relative positional relationships in memory (120).

[0139] The processor (130) can transmit information about the location of the UI element and the first prompt (21) to an external device (200) where an artificial intelligence model is stored via the communication unit (140).

[0140] The artificial intelligence model may be a model trained to convert a first prompt into a second prompt by adjusting the relationship between keywords corresponding to the UI element and other UI elements based on the distance and positional relationship with at least one other UI element that changes according to the movement of the UI element.

[0141] The artificial intelligence model can receive information about the location of UI elements and convert the first prompt (21) into the second prompt (22). Specifically, the artificial intelligence model can receive information about the UI elements and adjust the relationship of keywords corresponding to each UI element to add or change the expression for the keywords in the prompt. For example, referring to FIG. 6, the artificial intelligence model can receive information about the positional relationship where a UI element (32) containing 'sunglasses' is located next to a UI element (31) containing 'cat' and convert the first prompt (21) 'cat wearing sunglasses on its head' into the second prompt (22) 'cat next to sunglasses'. However, the expression of the second prompt is merely an example and is not limited to the embodiments of the present disclosure.

[0142] The processor (130) can receive a second prompt (22) from an external device (200) through the communication unit (140) and display it on the display (110).

[0143] The processor (130) can transmit a second prompt (22) to an external device (200) through a communication unit (140) and receive an image (40) generated by a generative AI model stored in the external device (200). The generative AI model stored in the external device (200) may be a model trained to generate an image (40) through a prompt. The generative AI model can receive the second prompt (22) as input and output an image (40).

[0144] The processor (130) can control the display (110) to display an image (40) generated through a generative AI model. The processor (130) can control the display (110) to display the second prompt (22) and the generated image (40) together. Referring to FIG. 6, the display (110) displays an image corresponding to 'cat next to sunglasses' generated through a generative AI model.

[0145] At this time, if the image (40) intended by the user is not correct, the user can obtain a new result by modifying the second prompt (22) or a plurality of UI elements (31) corresponding to the second prompt (22).

[0146] FIG. 7 is a drawing showing a case where a UI element is dragged and overlapped with another UI element according to at least one embodiment of the present disclosure.

[0147] Figure 7 shows a plurality of UI elements corresponding to a first prompt (21) of ‘cat in front of the sea’ received from a user, displayed on a display (110). A processor (130) can control the display (110) to identify ‘sea’ and ‘cat’ as keywords in the first prompt (21) and display UI elements containing each keyword.

[0148] At this time, the processor (130) can identify the relationship between multiple keywords and control the display (110) to display UI elements corresponding to the identified relationship between multiple keywords. Specifically, if the relationship between multiple keywords is a front-and-back relationship between keywords, the processor (130) can control the display (110) to display UI elements in a nested manner corresponding to the front-and-back relationship between multiple keywords. Referring to FIG. 7, the processor (130) can identify a front-and-back relationship in the first prompt (21) of 'cat in front of the sea' where 'sea' is the background and 'cat' is positioned in front of it. Subsequently, the processor (130) can control the display (110) so that a UI element (31) containing 'cat' is positioned in a nested manner in front of a UI element (33) containing 'sea'.

[0149] When a drag interaction is input in which one of the multiple UI elements is touched and dragged to a position where at least a part of it overlaps with another UI element, the processor (130) can control the display (110) to display the UI element and at least a part of the other element overlapping according to the direction of the drag interaction.

[0150] Referring to FIG. 7, a drag interaction (52) can be input from the user to a UI element (31) containing a ‘cat’.

[0151] As described above, when a drag interaction (52) is input from a user, the processor (130) can identify the location of the point touched by the user, its path, and the end point. The processor (130) can set the path along which the UI element (31) moves by reflecting the delta value of the coordinates for the starting position and the moving touch point location to the center point of the UI element (31), and can control the display (110) to display this path.

[0152] At this time, when a user drags a UI element (31) to a position where it overlaps with another UI element (33), the processor (130) can control the display (110) to display at least a portion of the dragged UI element (31) and the other UI element (33) overlapping. Specifically, when the UI element (31) overlaps with a portion of the other UI element (33), the processor (130) can control the display (110) so that the UI element (31) dragged by the user is placed on top of the other UI element (33) in a state where it overlaps with a portion of the other UI element (33). Another example is that the processor (130) can control the display (110) so that a UI element selected by the user via touch input is placed on top of the other UI element.

[0153] When a UI element (31) completely overlaps with another UI element (33), the processor (130) can control the display (110) by comparing the sizes of the multiple overlapping UI elements so that the relatively smaller UI element (31) is displayed on top of the other UI element (33). In another embodiment, the processor (130) can control the display (110) to display the relatively smaller UI element (31) as a dotted line to indicate that it is located behind the other UI element (33).

[0154] The processor (130) can store information about the moved UI element (32) in memory (120). Information about the position of the moved UI element (32) may include coordinates of the moved position of the UI element (32), relative positional relationships with other UI elements, and nesting relationships with other UI elements. For example, in FIG. 7, the processor (130) can control the display (110) to move the UI element (31) containing 'cat' to a position that completely overlaps with the UI element (33) containing 'sea' via drag interaction (52) and display it. The processor (130) can store information about the nesting relationship between the UI element (31) containing 'cat' and the UI element (33) containing 'sea' in memory (120).

[0155] The processor (130) can transmit information about the location of the UI element and the first prompt (21) to an external device (200) where an artificial intelligence model is stored via the communication unit (140).

[0156] The artificial intelligence model may be a model trained to convert a first prompt (21) into a second prompt (22) by adjusting the relationship between keywords corresponding to UI elements and other UI elements according to the degree of overlap between multiple UI elements.

[0157] The artificial intelligence model can receive information about the location of UI elements and convert the first prompt (21) into the second prompt (22). Specifically, the artificial intelligence model can receive information about the UI elements and adjust the relationships of keywords corresponding to each UI element to add or change the expression for the keywords in the prompt. For example, referring to FIG. 7, the artificial intelligence model can convert the first prompt (21), "cat in front of the sea," into the second prompt (22), "cat in the sea," through a relationship in which the UI element (31), containing "cat," is completely nested within the UI element (33), containing "sea." However, the expression of the second prompt is merely an example and is not limited to the embodiments of the present disclosure.

[0158] The artificial intelligence model can transform the prompt to include an expression regarding the relationship between keywords corresponding to the UI element and the other UI element when the UI element is partially nested with at least one other UI element. For example, if a UI element containing 'person' is partially nested before a UI element containing 'object', the artificial intelligence model can transform the prompt to include the expression 'object before person'. Alternatively, the artificial intelligence model can transform the prompt to include the expression 'person located before environment' even if a UI element containing 'person' is partially nested before a UI element containing 'environment'.

[0159] The artificial intelligence model can transform the prompt to include an expression regarding the inclusion relationship of keywords corresponding to the UI element and the other UI element when the UI element is completely nested with at least one other UI element. For example, the artificial intelligence model can transform the prompt to include the expression "wearing" when a UI element containing "clothes" is completely nested inside a UI element containing "person". Alternatively, the artificial intelligence model can transform the prompt to include the expression "inside" when a UI element containing "person" is completely nested inside a UI element containing "environment". The artificial intelligence model can transform the prompt to include the expression "riding" when a UI element containing "person" is completely nested inside a UI element containing "means of transportation".

[0160] The processor (130) can receive a second prompt (22) from an external device (200) through a communication unit (140) and display it on a display (110). After checking the second prompt (22), the user can edit the second prompt (22) or a plurality of UI elements corresponding to the second prompt (22).

[0161] The processor (130) can transmit a second prompt (22) to an external device (200) through a communication unit (140) and receive an image (40) generated through a generative AI model stored in the external device (200). The generative AI model stored in the external device (200) may be a model trained to generate an image (40) through a prompt. The generative AI model can receive the second prompt (22) as input and output an image (40).

[0162] The processor (130) can control the display (110) to display an image (40) generated through a generative AI model. The processor (130) can control the display (110) to display the second prompt (22) and the generated image (40) together. Referring to FIG. 7, the display (110) displays an image corresponding to a ‘cat in the sea’ generated through a generative AI model.

[0163] At this time, if the image (40) intended by the user is not correct, the user can obtain a new result by modifying the second prompt (22) or a plurality of UI elements (31) corresponding to the second prompt (22).

[0164] FIG. 8 is a drawing showing a case where the color of a UI element is changed according to at least one embodiment of the present disclosure.

[0165] The processor (130) can control the display (110) to change the color of the UI element when an interaction to change the color of one of the multiple UI elements is input.

[0166] FIG. 8 illustrates a UI element (31) corresponding to a first prompt (21) named 'cat' received from a user, which is displayed on a display (110) with its color changed by the user's color change interaction (53).

[0167] According to FIG. 8, a first prompt (21) of ‘cat’ is received from the user, and a corresponding UI element (31) is displayed on the display (110). At this time, when a color change interaction (53) is received from the user, the processor (130) can control the display (110) to display the UI element (31) whose color has been changed according to the color change interaction (53). Specifically, when the processor (130) receives a touch input for a UI (16) for changing the color of a UI element from the user, it can control the display (110) to display a color palette UI (18) for selecting a color. When the processor (130) receives a touch input (53) for a color desired by the user from the user, it can control the display (110) to change the color of the UI element (31) containing ‘cat’ to the color selected by the user and display it. Figure 8 illustrates that when the user selects gray, the UI element containing 'cat' is changed to gray.

[0168] When a color change interaction (53) is input from a user, the processor (130) can identify information about the color selected by the user and store it in memory (120). The information about the color may include at least one of RGB values, HEX codes, HSL codes, and CMYK codes. Referring to FIG. 8, the processor (130) can store an RGB value corresponding to 'gray', which is the color of a UI element (33) containing 'cat', in memory (120).

[0169] The processor (130) can transmit information about the color of the UI element and the first prompt (21) to an external device (200) where an artificial intelligence model is stored via the communication unit (140).

[0170] The artificial intelligence model may be a model trained to convert a first prompt into a second prompt by reflecting information about a color-modified UI element.

[0171] The artificial intelligence model can receive information about the color of a UI element and convert a first prompt (21) into a second prompt (22). Specifically, the artificial intelligence model can receive information about a UI element and convert the prompt to include a color expression for a keyword corresponding to the UI element. For example, according to FIG. 8, the artificial intelligence model can convert a first prompt (21) of 'cat' into a second prompt (22) of 'gray cat' through information about the color of a UI element (31). However, the expression of the second prompt is merely an example and is not limited to the embodiments of the present disclosure.

[0172] The processor (130) can receive a second prompt (22) from an external device (200) via a communication unit (140) and display it on a display (110). After checking the second prompt (22), the user can edit the second prompt (22) or multiple UI elements corresponding to the second prompt (22). When the user edits multiple UI elements corresponding to the second prompt (22), the processor (130) can transmit information about the edited multiple UI elements to the external device (200) and receive a third prompt converted by an artificial intelligence model of the external device (200). Additionally, when the user edits the second prompt (22), the processor (130) can control the display (110) to display the UI elements corresponding to the edited second prompt (22). The user can visually check the content of the prompt edited through the UI elements, allowing the user to easily predict the intended result.

[0173] The processor (130) can transmit a second prompt (22) to an external device (200) through a communication unit (140) and receive an image (40) generated through a generative AI model stored in the external device (200). The generative AI model stored in the external device (200) may be a model trained to generate an image (40) through a prompt. The generative AI model can receive the second prompt (22) as input and output the generated image (40).

[0174] The processor (130) can control the display (110) to display an image (40) generated through a generative AI model. The processor (130) can control the display (110) to display the second prompt (22) and the generated image (40) together. Referring to FIG. 8, the display (110) displays an image corresponding to a 'gray cat' generated through a generative AI model.

[0175] At this time, if the image (40) intended by the user is not correct, the user can obtain a new result by modifying the second prompt (22) or a plurality of UI elements (31) corresponding to the second prompt (22).

[0176] FIG. 9 is a drawing showing a case where a UI element is rotated according to at least one embodiment of the present disclosure.

[0177] FIG. 9 illustrates a UI element (34) corresponding to a first prompt (21) named 'rose' received from a user, rotated by the user's rotation interaction (54) and displayed on a display (110).

[0178] When an interaction (54) that rotates one of the multiple UI elements is input to the processor (130), the processor (130) can control the display (110) to rotate and display the UI element.

[0179] According to FIG. 9, a first prompt (21) of ‘rose’ is received from the user, and a corresponding UI element (34) is displayed on the display (110). At this time, when a rotation interaction (54) is received from the user, the processor (130) can control the display (110) to display a UI element (34) rotated according to the rotation interaction (54).

[0180] According to one embodiment, when a rotation interaction (54) is input from a user, the processor (130) can identify the location of two points touched by two fingers. The processor (130) can measure the initial angle and the angle after the rotation interaction (54) is performed through coordinate information regarding the location of the two points. Subsequently, the processor (130) can measure the rotation angle through the difference between the initial angle and the angle after the interaction. At this time, the processor (130) can determine that the rotation is counterclockwise if the rotation angle is positive, and clockwise if it is negative. Subsequently, the processor (130) can control the display (110) to rotate and display the UI element based on the measured rotation angle and rotation direction. At this time, the processor (130) can control the display (110) so that the keyword included in the UI element rotates and is displayed together with the rotation angle and rotation direction. Through this, the user can recognize that the UI element has rotated. Referring to Fig. 9, a UI element (34) containing 'rose' is rotated counterclockwise so that the 'rose' keyword included in the UI element is displayed in a rotated state according to the rotation angle and rotation direction.

[0181] The processor (130) can store information about the rotated UI element (34) in memory (120). The information about the rotated UI element (34) may include information about the rotation angle and the rotation direction. For example, in FIG. 9, a UI element (34) containing a 'rose' is rotated counterclockwise through a rotation interaction (54). The processor (130) can store information about the rotation angle and the rotation direction of the UI element (34) containing a 'rose' in memory (120).

[0182] The processor (130) can transmit information about the rotation of the UI element and the first prompt (21) to an external device (200) where an artificial intelligence model is stored via the communication unit (140).

[0183] The artificial intelligence model may be a model trained to convert the first prompt (21) into the second prompt (22) by reflecting information about the rotated UI element.

[0184] The artificial intelligence model can receive information about the rotation of a UI element and convert a first prompt (21) into a second prompt (22). Specifically, the artificial intelligence model can convert the prompt to include an expression corresponding to the rotation angle and rotation direction of the UI element for a keyword corresponding to the UI element. Referring to FIG. 9, the artificial intelligence model can convert the first prompt (21) of "rose" into a second prompt (22) of "rose tilted to the left" through the rotation direction of the UI element (34). However, the expression of the second prompt is merely an example and is not limited to the embodiments of the present disclosure.

[0185] The processor (130) can receive a second prompt (22) from an external device (200) via a communication unit (140) and display it on a display (110). The processor (130) can receive an interaction from a user to edit the second prompt (22) or a plurality of UI elements corresponding to the second prompt (22). When the user edits a plurality of UI elements corresponding to the second prompt (22), the processor (130) can transmit information about the edited plurality of UI elements to the external device (200) and receive a third prompt converted by an artificial intelligence model of the external device (200). Additionally, when the user edits the second prompt (22), the processor (130) can control the display (110) to display the UI elements corresponding to the edited second prompt (22). The user can visually confirm the content of the prompt edited through the UI elements, allowing the user to easily predict the intended result.

[0186] The processor (130) can transmit a second prompt (22) to an external device (200) through a communication unit (140) and receive an image (40) generated through a generative AI model stored in the external device (200). The generative AI model stored in the external device (200) may be a model trained to generate an image (40) through a prompt. The image (40) can be output by inputting the second prompt (22) into the generative AI model from the external device (200).

[0187] The processor (130) can control the display (110) to display an image (40) generated through a generative AI model. The processor (130) can control the display (110) to display the second prompt (22) and the generated image (40) together. Referring to FIG. 9, the display (110) displays an image corresponding to a 'left-leaning rose' generated through a generative AI model.

[0188] At this time, if the image (40) intended by the user is not correct, the user can obtain a new result by modifying the second prompt (22) or multiple UI elements (34) corresponding to the second prompt (22).

[0189] FIG. 10 is a drawing showing a case where a new UI element is added according to at least one embodiment of the present disclosure.

[0190] FIG. 10 illustrates a state in which a new UI element (34) containing a ‘rose’ is added by the user’s UI element addition interaction while a UI element (31) corresponding to a first prompt (21) of ‘cat’ received from the user is displayed, and the new UI element (34) is displayed on the display (110).

[0191] The processor (130) can control the display (110) to add and display a new UI element to a plurality of UI elements when a user interaction to add a new UI element is input.

[0192] According to FIG. 10, the processor (130) can control the display (110) to receive a first prompt (21) of ‘cat’ from the user and display a corresponding UI element (31). Additionally, the processor (130) can control the display (110) to display a guide UI (13) that guides the addition of a new UI element. The guide UI (13) that guides the addition of a new UI element may be a circular UI containing a ‘+’ shape. However, this is merely one example and is not limited thereto. At this time, when the processor (130) receives touch input for the guide UI (13) from the user, it can control the display (110) to display an input window that can receive a keyword corresponding to the new UI element. When the processor (130) receives a keyword corresponding to the new UI element from the user, it can control the display (110) to display a new UI element containing the received keyword. Referring to FIG. 10, the display (110) receives the new keyword 'rose' from the user and displays a new UI element (34) containing 'rose'.

[0193] The processor (130) can receive an interaction for editing a new UI element (34). For example, the processor (130) can receive at least one of a pinch-out or pinch-in interaction, a drag interaction, a color change interaction, and a rotation interaction from the user. The processor (130) can control the display (110) to display the edited UI element according to the interaction.

[0194] The processor (130) can store information about a new UI element (34) in memory (120). The information about the new UI element (34) may include information about a new keyword. For example, information about a new keyword, 'rose,' corresponding to the new UI element (34) in FIG. 10, may be stored in memory (120). Additionally, when a user interaction to edit the new UI element (34) is input, the processor (130) can store information about the UI element (34) edited according to the user interaction in memory (120). For example, when a pinch-out or pinch-in interaction to adjust the size of the new UI element (34) is input, the processor (130) can store information about the UI element (34) whose size has been changed.

[0195] The processor (130) can transmit information about a new UI element (34) and a first prompt (21) to an external device (200) where an artificial intelligence model is stored via the communication unit (140). For example, the processor (130) can transmit information about a UI element (31) containing a 'cat' and a newly added UI element (34) containing a 'rose', and a first prompt (21) to an external device (200) where an artificial intelligence model is stored via the communication unit (140).

[0196] The artificial intelligence model may be a model trained to convert a first prompt into a second prompt by reflecting information about the new UI element.

[0197] The artificial intelligence model can convert a first prompt (21) into a second prompt (22) based on information about a new UI element. Specifically, the artificial intelligence model can convert the prompt to include a keyword corresponding to the new UI element and an expression modifying the keyword. Referring to FIG. 10, the artificial intelligence model can receive a new UI element (34) containing 'rose' and a UI element (31) containing 'cat' as input and convert the first prompt (21) 'cat' into a second prompt (22) 'rose next to cat'. However, the expression of the second prompt (22) is merely an example and is not limited to the embodiments of the present disclosure.

[0198] The processor (130) can receive a second prompt (22) from an external device (200) via a communication unit (140) and display it on a display (110). After checking the second prompt (22), the user can edit the second prompt (22) or a plurality of UI elements corresponding to the second prompt (22). If the user edits a plurality of UI elements corresponding to the second prompt (22), the processor (130) can transmit information about the edited plurality of UI elements to the external device (200) and receive a third prompt converted by an artificial intelligence model of the external device (200). Additionally, if the user edits the second prompt (22), the processor (130) can control the display (110) to display the UI elements corresponding to the edited second prompt (22).

[0199] The processor (130) can transmit a second prompt (22) to an external device (200) through a communication unit (140) and receive an image (40) generated through a generative AI model stored in the external device (200). The generative AI model stored in the external device (200) may be a model trained to generate an image (40) through a prompt. The image (40) can be output by inputting the second prompt (22) into the generative AI model from the external device (200).

[0200] The processor (130) can control the display (110) to display an image (40) generated through a generative AI model. The processor (130) can control the display (110) to display the second prompt (22) and the generated image (40) together. Referring to FIG. 10, the display (110) displays an image corresponding to 'a rose next to a cat' generated through a generative AI model.

[0201] At this time, if the image (40) intended by the user is not correct, the user can obtain a new result by modifying the second prompt (22) or multiple UI elements (34) corresponding to the second prompt (22).

[0202] FIG. 11 is a flowchart illustrating a control method of an electronic device (100) according to at least one embodiment of the present disclosure.

[0203] The electronic device (100) can display multiple UI elements corresponding to each of the multiple keywords included in the first prompt (21) received from the user (S1110). The prompt (20) may be a command to be input into an artificial intelligence model to output a result value. For example, the prompt (20) may be a sentence to be input into a generative AI model to generate an image (40).

[0204] Keywords may refer to words that can be generated as images (40) on the prompt (20). Keywords may include objects such as people, animals, or things, and may include backgrounds such as the space or environment where the objects are located. Keywords regarding objects may appear mainly as subjects on the prompt (20) and may be central elements that perform specific actions or have a presence. Additionally, keywords regarding objects may be elements connected to actions or described through modifiers for features such as appearance, size, or color. On the other hand, keywords regarding background may be spatial or environmental elements where the objects are located, identified mainly through words or prepositional phrases describing the location or environment.

[0205] Information regarding the attributes of multiple keywords may be information regarding the characteristics of each keyword or the relationships between keywords. The electronic device (100) can identify information regarding the attributes of multiple keywords through modifiers describing the keywords included in the prompt (20).

[0206] The electronic device (100) can display multiple UI elements (30) corresponding to multiple identified keywords. At this time, the electronic device (100) can display multiple UI elements (30) corresponding to information about the attributes of the multiple keywords.

[0207] When a user interaction for editing multiple UI elements is input to the electronic device (100), the device can obtain a second prompt (22) that edits the first prompt (21) based on the edited UI elements (S1120).

[0208] The electronic device (100) can receive user interactions for editing a plurality of UI elements (30). User interactions for editing a plurality of UI elements (30) may include at least one of pinch-in or pinch-out interactions, drag interactions, color change interactions, rotation interactions, and UI element addition interactions. When the electronic device (100) receives user interactions for editing UI elements (30), it can adjust and display the UI elements (30) to correspond to the interaction.

[0209] The electronic device (100) can convert the prompt (20) using information about the prompt (20) and the edited multiple UI elements (30) through an artificial intelligence model stored in the electronic device (100) or an external device (200). The artificial intelligence model may be a model trained to convert the first prompt (21) and the multiple UI elements (30) into a second prompt (22) when information about the first prompt (21) and the multiple UI elements (30) is input.

[0210] The electronic device (100) can display an image generated from a generative AI model based on a second prompt (22) (S1130).

[0211] The electronic device (100) can display a generated image (40) by inputting a second prompt (22) into a generative AI model stored in the electronic device (100) or an external device (200). After checking the generated image (40), the user can obtain a new image by modifying the second prompt (22) or the UI element (30) corresponding to the second prompt (22).

[0212] The various embodiments described above may be implemented individually, but are not necessarily limited thereto, and may be implemented together in combination with at least one other embodiment, either partially or wholly.

[0213] The methods according to the various embodiments of the present disclosure described above can be implemented by software upgrade or hardware upgrade alone for an existing electronic device (100).

[0214] Meanwhile, according to a specific example of the present disclosure, the control method according to the various embodiments described above may be implemented as software comprising instructions stored on a non-transitory machine-readable storage media that can be read by various machines (e.g., computers), such as an electronic device (100).

[0215] Specifically, a program for performing a control method may be provided in a state stored on a non-transient computer-readable recording medium, comprising the steps of: displaying a plurality of UI elements corresponding to each of a plurality of keywords included in a first prompt received from a user; when a user interaction for editing the plurality of UI elements is input, obtaining a second prompt in which the first prompt is edited based on the edited UI elements; and displaying an image generated by a generative AI model based on the second prompt.

[0216] When stored software or instructions are executed by the processor (130), the processor (130) may perform operations according to the various embodiments described above, either directly or by using other components. Instructions may include code generated or executed by a compiler or an interpreter. Here, 'non-transient' means only that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

[0217] Additionally, according to one or more embodiments of the present disclosure, the method according to the various embodiments described above may be provided as a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed online through an online store as well as the non-transient readable recording medium described above. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium such as the memory (120) of a manufacturer's server, an application store's server, or a relay server.

[0218] Additionally, each component (e.g., module or program) according to the various embodiments described above may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in the various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the functions performed by each of the respective components prior to integration in the same or similar manner. The operations performed by the module, program, or other components according to the various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added.

[0219] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In an electronic device, display; Memory in which at least one instruction is stored; and Includes a processor; The above processor, based on the execution of the above at least one instruction, Control the display to display multiple UI elements corresponding to each of the multiple keywords included in the first prompt received from the user, and When a user interaction for editing the above plurality of UI elements is input, a second prompt is obtained by editing the first prompt based on the edited plurality of UI elements, and Controlling the display to display an image generated by a generative AI model based on the second prompt above, Electronic device.

2. In Paragraph 1, It further includes a communication unit for performing communication with an external device storing an artificial intelligence model trained to edit a prompt based on the display state of the above UI element, and The above processor is, Information regarding the first prompt and the edited plurality of UI elements is transmitted to the external device through the communication unit, and Information regarding the first prompt and the edited plurality of UI elements is input into the artificial intelligence model from the above external device to obtain the second prompt, which is received through the communication unit, and The information of the plurality of UI elements above includes at least one of the size, position, color, arrangement, and relationship between UI elements of each UI element. Electronic device.

3. In Paragraph 2, The above processor is, When a pinch-in or pinch-out interaction is input to one of the plurality of UI elements, the display is controlled to decrease or increase the size of the UI element. The above artificial intelligence model is trained to convert the first prompt into the second prompt by adjusting the weights for keywords corresponding to each element based on the size change of each UI element. Electronic device.

4. In Paragraph 2, The above processor is, When a drag interaction is input by touching and dragging one of the plurality of UI elements, the display is controlled to move and display the UI element according to the direction of the drag interaction. The artificial intelligence model is trained to convert the first prompt into the second prompt by adjusting the relationship between the keywords corresponding to the UI element and the other UI element based on the distance and positional relationship with at least one other UI element that changes according to the movement of the UI element. Electronic device.

5. In Paragraph 2, The above processor is, When a drag interaction is input in which one of the plurality of UI elements is touched and dragged to a position where at least a part of it overlaps with another UI element, the display is controlled to display the UI element and at least a part of the other element in overlap according to the direction of the drag interaction. The artificial intelligence model is trained to convert the first prompt into the second prompt by adjusting the relationship between the keywords corresponding to the UI element and the other element according to the degree of overlap between the plurality of UI elements. Electronic device.

6. In Paragraph 2, The above processor is, When an interaction to change the color of one of the above plurality of UI elements is input, the display is controlled to change the color of the UI element, and The above artificial intelligence model is trained to convert the first prompt into the second prompt by reflecting information about the color-modified UI element, Electronic device.

7. In Paragraph 2, The processor, When an interaction is input to rotate one of the above plurality of UI elements, the display is controlled to rotate the UI element, and The above artificial intelligence model is trained to convert the first prompt into the second prompt by reflecting information about the rotated UI element, Electronic device.

8. In Paragraph 1, The above processor is, When a user interaction to add a new UI element is input, the display is controlled to add the new UI element to the plurality of UI elements and display it. The above artificial intelligence model is trained to convert the first prompt into the second prompt by reflecting information about the new UI element, Electronic device.

9. In Paragraph 1, The above processor is, The above second prompt is transmitted to the external device where the generative AI model is stored via the communication unit, and The generated image obtained by inputting the second prompt into the generative AI model from the above external device is received through the communication unit, and The above generative AI model is trained to output an image corresponding to the above second prompt, Electronic device.

10. In Paragraph 1, The above memory stores an artificial intelligence model trained to convert a prompt based on the display state of the above UI element, and The above processor is, Information regarding the first prompt and the edited UI element is input into the artificial intelligence model to obtain the second prompt, and Controlling the display to display an image generated by a generative AI model based on the second prompt above, Electronic device.

11. In a method for controlling an electronic device, A step of displaying multiple UI elements corresponding to each of the multiple keywords included in the first prompt received from the user; When a user interaction for editing the plurality of UI elements is input, a step of obtaining a second prompt that edits the first prompt based on the edited UI elements; and A control method comprising the step of displaying an image generated by a generative AI model based on the second prompt above.

12. In Paragraph 11, The step of converting the above-mentioned first prompt into a second prompt is, A step of transmitting information regarding the first prompt and the edited plurality of UI elements to an external device; and The method includes the step of receiving the second prompt obtained by inputting information about the first prompt and the edited plurality of UI elements into an artificial intelligence model from the external device; The above artificial intelligence model is trained to convert a prompt based on the display state of the above UI element, and The information of the above-mentioned edited plurality of UI elements includes at least one of the size, position, color, arrangement, and relationship between UI elements of each UI element. Control method.

13. In Paragraph 12, The method further includes the step of displaying the UI element by decreasing or increasing its size when a pinch-in or pinch-out interaction is input to one of the plurality of UI elements. The above artificial intelligence model is trained to convert the first prompt into the second prompt by adjusting the weights for keywords corresponding to each element based on the size change of each UI element. Control method.

14. In Paragraph 12, The method further includes the step of moving and displaying the UI element according to the direction of the drag interaction when a drag interaction is input in which one of the plurality of UI elements is touched and dragged. The artificial intelligence model is trained to convert the first prompt into the second prompt by adjusting the relationship between the keywords corresponding to the UI element and the other UI element based on the distance and positional relationship with at least one other UI element that changes according to the movement of the UI element. Control method.

15. In Paragraph 12, When a drag interaction is input in which one of the plurality of UI elements is touched and dragged to a position where at least a portion overlaps with another UI element, the method further includes the step of displaying an overlap of at least a portion of the UI element and the other element according to the direction of the drag interaction. The artificial intelligence model is trained to convert the first prompt into the second prompt by adjusting the relationship between the keywords corresponding to the UI element and the other element according to the degree of overlap between the plurality of UI elements. Control method.

Images