Electronic device and control method therefor

Abstract

Disclosed is an electronic device. One or more processors, when instructions are individually or collectively executed, cause the electronic device to: when a user voice is received, acquire, by means of an artificial intelligence model, a first response corresponding to the user voice and a second response in which the first response is summarized; identify a keyword and the type of the keyword included in the second response; acquire an image corresponding to the keyword on the basis of the type of the keyword; and output a voice corresponding to the first response by means of a speaker and output an image corresponding to the keyword by means of a display.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same.

[0002] Recently, technology for conversational AI (Artificial Intelligence) that converses with users using artificial intelligence models has been advancing.

[0003] Conversational AI is an artificial intelligence technology that understands the language corresponding to the user's voice and provides a response to the user's voice to enable natural conversation. Generally, when conversational AI receives a user's voice, it can generate a response corresponding to the voice and provide the response through a speaker.

[0004] However, when the amount of text included in the response to the user's voice was large, it took some time to provide the response, and it was difficult to check it intuitively.

[0005] An electronic device according to one or more embodiments of the present disclosure includes a display, a speaker, a memory for storing instructions, and one or more processors including processing circuitry.

[0006] According to one or more embodiments, when the instructions are executed individually or collectively, the electronic device, upon receiving a user voice, obtains a first response corresponding to the user voice and a second response summarizing the first response through an artificial intelligence model, identifies a keyword included in the second response and a type of the keyword, obtains an image corresponding to the keyword based on the type of the keyword, outputs a voice corresponding to the first response through the speaker, and outputs an image corresponding to the keyword through the display.

[0007] According to one or more embodiments, the image corresponding to the keyword is an image containing text corresponding to the keyword.

[0008] According to one or more embodiments, when the instructions are executed individually or collectively by the one or more processors, the electronic device identifies the type of the keyword based on attribute information of text corresponding to the keyword included in the second response, and the attribute information of the text includes at least one of object information, emotion information, time information, and weather information.

[0009] According to one or more embodiments, when the instructions are executed individually or collectively by the one or more processors, the electronic device identifies a keyword type corresponding to an attribute of the text among a plurality of preset keyword types, and the plurality of preset keyword types include a subjective keyword type and an objective keyword type.

[0010] According to one or more embodiments, when the instructions are executed individually or collectively by the one or more processors, the electronic device identifies at least one of the style information and effect information corresponding to the identified keyword type based on at least one of the style information and effect information corresponding to the plurality of keyword types stored in the memory, and acquires an image corresponding to the keyword based on at least one of the style information and effect information corresponding to the identified keyword type.

[0011] According to one or more embodiments, the style information includes at least one of the text color, text shape, and text material, and the effect information includes at least one of the text motion information and additional icon information related to the text.

[0012] According to one or more embodiments, when the instructions are executed individually or collectively by one or more processors, the electronic device, when the user is identified, obtains a third response and a fourth response summarizing the third response through the artificial intelligence model based on at least one of conversation history information and context information between the identified user and the electronic device, identifies keywords included in the fourth response and types of keywords included in the fourth response, obtains an image corresponding to keywords included in the fourth response based on the types of keywords included in the fourth response, outputs voice corresponding to the third response through the speaker, and outputs an image corresponding to keywords included in the fourth response through the display.

[0013] According to one or more embodiments, when the instructions are executed individually or collectively by the one or more processors, the electronic device displays the image corresponding to the keyword in one area of ​​the screen when the image corresponding to the keyword is acquired while content is being output on the screen of the display, and displays the image corresponding to the keyword in the entire area of ​​the screen when the image corresponding to the keyword is acquired while content is not being output on the screen of the display.

[0014] According to one or more embodiments, the artificial intelligence model is a model trained to generate a response corresponding to a user's voice and to summarize the generated response to obtain a summary response within a preset number of characters.

[0015] A control method for an electronic device according to one or more embodiments of the present disclosure includes, when a user voice is received, obtaining a first response corresponding to the user voice and a second response summarizing the first response through an artificial intelligence model; identifying a keyword included in the second response and a type of the keyword; obtaining an image corresponding to the keyword based on the type of the keyword; and outputting a voice corresponding to the first response and an image corresponding to the keyword.

[0016] A non-transient computer-readable storage medium storing computer instructions that cause the electronic device to perform an operation when executed by a processor of an electronic device according to one or more embodiments of the present disclosure, wherein the operation includes, when a user voice is received, obtaining a first response corresponding to the user voice and a second response summarizing the first response through an artificial intelligence model; identifying a keyword included in the second response and a type of the keyword; obtaining an image corresponding to the keyword based on the type of the keyword; and outputting a voice corresponding to the first response and an image corresponding to the keyword.

[0017] FIG. 1 is a drawing for explaining the operation of an electronic device according to one or more embodiments.

[0018] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one or more embodiments.

[0019] FIG. 3 is a diagram illustrating the process of obtaining a first response and a second response of an electronic device according to one or more embodiments.

[0020] FIG. 4 is a diagram illustrating the process of identifying keyword types of an electronic device according to one or more embodiments.

[0021] FIG. 5 is a drawing for explaining style information and effect information by keyword type of an electronic device according to one or more embodiments.

[0022] FIGS. 6 and FIGS. 7 are drawings for explaining the process of acquiring an image of an electronic device according to one or more embodiments.

[0023] FIG. 8 is a diagram illustrating a process of providing a response based on whether a user of an electronic device is identified according to one or more embodiments.

[0024] FIG. 9 is a drawing for explaining a method for displaying an acquired image of an electronic device according to one or more embodiments.

[0025] FIG. 10 is a drawing for explaining a method of operation of an electronic device according to one or more embodiments.

[0026] The terms used in the various embodiments of this Disclosure have been selected to be as widely used and general as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant description section of this disclosure. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0027] In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0028] The expression "at least one of A or / and B" should be understood as representing either "A" or "B" or "A and B".

[0029] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0030] Where it is stated that a component (e.g., Component 1) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., Component 2), it should be understood that the component may be directly connected to the other component or connected through the other component (e.g., Component 3).

[0031] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this disclosure, terms such as “comprising” or “consisting of” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0032] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor (not shown), except for a "module" or "part" that needs to be implemented in specific hardware.

[0033] In the present disclosure, the term "user" may refer to a person using an electronic device or a device used by such person.

[0034] An embodiment of the present disclosure will be described in more detail below with reference to the attached drawings.

[0035] FIG. 1 is a drawing for explaining the operation of an electronic device according to one or more embodiments.

[0036] According to one embodiment, the electronic device (100) receives a user voice, acquires an image corresponding to the user voice, and outputs the image corresponding to the user voice through a display (110). Here, the electronic device (100) can be implemented as various types of electronic devices such as a smart TV, digital signage, a monitor, a kiosk, a tablet PC, an electronic photo frame, a mobile phone, a large format display (LFD), a digital information display (DID), a video wall, a projector display, etc. However, depending on the case, it may be implemented as an image processing device (e.g., a set-top box, one connected box) that is connected to the electronic device and provides an image.

[0037] According to one embodiment, when a user voice is received, the electronic device (100) can obtain a first response corresponding to the user voice and a response summarizing the first response through an artificial intelligence model. The electronic device (100) can obtain a first response corresponding to the user voice through an artificial intelligence model. The electronic device (100) can obtain a second response summarizing the first response through an artificial intelligence model.

[0038] According to one embodiment, the electronic device (100) may identify a keyword included in the second response and obtain an image corresponding to the keyword. The image corresponding to the keyword may be an image containing text corresponding to the keyword. For example, if the keyword identified from the second response is “birthday,” the electronic device (100) may be an image containing the text “birthday.”

[0039] According to one embodiment, the electronic device (100) may output a voice corresponding to a first response through a speaker (120) and display an acquired image through a display (110). The order of outputting the voice corresponding to the first response and displaying the acquired image is not limited thereto and may be operated simultaneously or sequentially.

[0040] Referring to FIG. 1, an electronic device (100) can receive a user voice (30), such as “What is the weather like today?” from a user (20). The electronic device (100) can input the received user voice (30) into an artificial intelligence model to obtain a first response corresponding to the user voice (30). The electronic device (100) can input the received user voice (30) into an artificial intelligence model to obtain a second response that summarizes the first response.

[0041] The electronic device (100) can identify keywords from the second response. For example, if a second response such as “the weather is cold” is obtained from a user voice (30) such as “what is the weather like today?”, the electronic device (100) can identify keywords such as “cold” from the second response.

[0042] The electronic device (100) can obtain an image (10) corresponding to the keyword from the identified keyword. The electronic device (100) can output the image corresponding to the keyword through a display (110). The electronic device (100) can output a response corresponding to the user's voice (30) through a speaker (120).

[0043] Hereinafter, with reference to the drawings, a specific method will be described in which an electronic device (100) identifies a keyword type and generates an image corresponding to the keyword.

[0044] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to one or more embodiments.

[0045] According to FIG. 2, the electronic device (100) includes a display (110), a speaker (120), a memory (130), and one or more processors (140). However, it is not limited thereto, and the electronic device (100) may be implemented with some components excluded or with other components included.

[0046] The display (110) is configured to display the estimated location of content and preferred objects, including a plurality of frames. The display (110) may be implemented as a display including a self-emissive element or as a display including a non-emissive element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (110) may also include a driving circuit, a backlight unit, etc., which may be implemented in the form of an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc.

[0047] The speaker (120) can convert a digital audio signal processed by one or more processors (140) into an analog audio signal, amplify it, and output it. For example, the speaker (120) may include at least one speaker unit, a D / A converter, an audio amplifier, etc., capable of outputting at least one channel. For example, the speaker (120) may output a first response corresponding to a user's voice.

[0048] The memory (130) can store at least one instruction, data, program, etc. required for the operation of the electronic device (100). For example, the memory (130) can store style information and effect information by keyword type.

[0049] The memory (130) may be implemented in the form of a memory embedded in the electronic device (100) or in the form of a memory detachable from the electronic device (100), depending on the purpose of data storage. For example, data for operating the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for the expansion function of the electronic device (100) may be stored in a memory detachable from the electronic device (100).

[0050] In the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD).

[0051] The memory (130) may be implemented as a single memory that stores data generated in various operations according to the present disclosure, but is not limited thereto, and the memory (130) may be implemented to include a plurality of memories that each store different types of data or each store data generated in different stages.

[0052] One or more processors (140) control the overall operation of the electronic device (100). Specifically, one or more processors (140) may be connected to each component of the electronic device (100) to control the overall operation of the electronic device (100). For example, one or more processors (140) may be electrically connected to the display (110), speaker (120), and memory (130) to control the overall operation of the electronic device (100). One or more processors (140) may include processing circuits and may be composed of one or more processors.

[0053] One or more processors (140) can perform the operation of an electronic device (100) according to various embodiments by executing one or more instructions stored in memory (130).

[0054] One or more processors (130) may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator. One or more processors (140) may control one or any combination of other components of an electronic device and may perform operations or data processing related to communication. One or more processors (130) may execute one or more programs or instructions stored in memory. For example, one or more processors may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in memory.

[0055] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0056] One or more processors (130) may be implemented as a single-core processor including one core, or as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When one or more processors (130) are implemented as multicore processors, each of the multiple cores included in the multicore processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by multiple cores may be included in the multicore processor. Additionally, each of the multiple cores included in the multicore processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one or more embodiments of the present disclosure.

[0057] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0058] In the embodiments of the present disclosure, a processor may refer to a system-on-chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. For convenience of explanation, one or more processors (140) will be referred to as processors (140) below.

[0059] According to one embodiment, when a user voice is received, the processor (140) can obtain a first response corresponding to the user voice and a second response summarizing the first response through an artificial intelligence model.

[0060] The second response is a response that summarizes the first response corresponding to the user's voice, and may be a response containing text within a preset number of characters. For example, the second response may be a response that summarizes the first response into text within 10 characters. For example, the second response may be a response that summarizes the first response into one word.

[0061] According to one embodiment, the processor (140) can identify the keyword and the type of keyword included in the second response.

[0062] The keywords included in the second response may be key words that represent or summarize the second response. For example, the electronic device (100) can identify the keyword “cold” from “the weather is cold” and the keyword “Mom’s birthday” from “Today is Mom’s birthday”.

[0063] For example, keyword types may include subjective keyword types and objective keyword types. Subjective keyword types may include positive emotion keyword types and negative emotion keyword types. Objective keyword types may include time keyword types, weather keyword types, temperature keyword types, day of the week keyword types, and object keyword types. Keyword types are not limited to these and, of course, can be updated through user input.

[0064] According to one example, the processor (140) can identify the keyword type included in the second response based on a pre-set keyword type (e.g., subjective keyword type, objective keyword type).

[0065] According to one embodiment, the processor (140) can acquire an image corresponding to a keyword based on the type of keyword. The processor (140) identifies style information and effect information mapped to the keyword type, and can acquire an image corresponding to the keyword based on the identified style information and effect information. A detailed explanation of this will be provided later in FIG. 5.

[0066] According to one embodiment, the processor (140) can output a voice corresponding to the first response through the speaker (120) and output an image corresponding to the keyword through the display (110).

[0067] FIG. 3 is a diagram illustrating the process of obtaining a first response and a second response of an electronic device according to one or more embodiments.

[0068] According to one embodiment, the electronic device (100) can obtain a first response (330) and a second response (340) corresponding to a user voice (320) through an artificial intelligence model (310).

[0069] According to one embodiment, the artificial intelligence model (310) may be a model trained to generate a response corresponding to a user's voice and to summarize the generated response to obtain a summary response within a preset number of characters.

[0070]

[0071] Here, the term "training an artificial intelligence model" means that a basic artificial intelligence model (e.g., an artificial intelligence model containing arbitrary random parameters) is trained using multiple training data by a learning algorithm, thereby creating predefined behavioral rules or an artificial intelligence model configured to perform a desired characteristic (or objective). This learning may be performed via a separate server and / or system, but is not limited thereto, and may also be performed in a cooking device. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples mentioned above.

[0072] Here, the artificial intelligence model may be implemented as, for example, CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), BRDNN (Bidirectional Recurrent Deep Neural Network), or Deep Q-Networks, but is not limited thereto.

[0073] According to one example, the electronic device (100) can obtain a first response (330) corresponding to a user voice (320) through a Large Language Model (LLM). The electronic device (100) can obtain a second response (340) summarizing the first response (330) through a Large Language Model (LLM).

[0074] Referring to FIG. 3, the electronic device (100) can input a user voice (320) into an artificial intelligence model (310) to obtain a first response (330) corresponding to the user voice. For example, when the electronic device (100) receives a user voice (320) such as “How is the weather in London these days?”, it can obtain a first response (330) such as “The weather in London these days is getting really chilly. Especially in the mornings and evenings, the temperature drops sharply, so it is cold. So you need to make sure to wear thick clothes.”

[0075] The electronic device (100) can input user voice (320) into an artificial intelligence model (310) to obtain a second response (340) that summarizes the first response. For example, the electronic device (100) can obtain a second response (340) such as “It’s getting colder” that summarizes the first response (330).

[0076] FIG. 4 is a diagram illustrating the process of identifying keyword types of an electronic device according to one or more embodiments.

[0077] According to one embodiment, the electronic device (100) can identify the type of the keyword based on attribute information of the text corresponding to the keyword included in the second response.

[0078] According to one embodiment, the electronic device (100) can identify a keyword type corresponding to an attribute of text among a plurality of preset keyword types.

[0079] Text attribute information may be information including at least one of object information, sentiment information, time information, and weather information. Text attribute information may be information for identifying a keyword type based on the attributes of a keyword.

[0080] Multiple pre-configured keyword types may include subjective keyword types and objective keyword types. For example, subjective keyword types may include positive emotion keyword types containing positive emotion text and negative emotion keyword types containing negative emotion text.

[0081] For example, objective keyword types may include time keyword types containing text such as morning, lunch, and evening, weather keyword types containing text such as clear, cloudy, and rain, and date keyword types such as March, the 15th, and Christmas.

[0082] For example, if the keyword included in the second response is text regarding emotions such as “thank you” or “angry,” the electronic device (100) can identify attribute information corresponding to the keyword as emotion information. In this case, the electronic device (100) can identify the keyword type corresponding to the keyword as a subjective keyword type among the pre-set keyword types.

[0083] For example, if the keyword included in the second response is text related to weather, such as “clear” or “rain / snow,” the electronic device (100) can identify attribute information corresponding to the keyword as weather information. In this case, the electronic device (100) can identify the keyword type corresponding to the keyword as an objective keyword type among the pre-set keyword types.

[0084] Referring to FIG. 4, the electronic device (100) can identify a keyword (410) from a second response. The electronic device (100) can identify one of the pre-set keyword types (430) as a keyword type (440) based on text attribute information (420) corresponding to the keyword (410). For example, the electronic device (100) can identify "Christmas" as a keyword (410) from a second response such as "December 25 is Christmas." The electronic device (100) can identify the text attribute information corresponding to the keyword "Christmas" as time information. The electronic device (100) can identify it as an objective keyword type among the pre-set keyword types.

[0085] FIG. 5 is a drawing for explaining style information and effect information by keyword type of an electronic device according to one or more embodiments.

[0086] According to one embodiment, the electronic device (100) can identify at least one of the style information and effect information corresponding to the identified keyword type based on at least one of the style information and effect information for each of the plurality of keyword types stored in the memory (130).

[0087] According to one example, style information may include at least one of text color, text form, and text material. The text color may be red, blue, or green, the text form may be a serif font or a sans-serif font, and the text material may be a soft material or a rough material.

[0088] According to one example, effect information may include at least one of text motion information and additional icon information related to the text. The text motion information may be up-and-down movement information, left-and-right movement information, and zoom in / out information. The additional icon information related to the text may be a star icon, a snow falling motion icon, or a rain falling motion icon.

[0089] Style information and effect information for multiple keyword types may be style information and effect information mapped according to keyword types. For example, among subjective keyword types, the positive emotion keyword type may be mapped to the blue color and serif font in the style information. For example, among objective keyword types, the weather keyword type may be mapped to the red color and soft texture in the style information, and the left-right movement effect in the effect information.

[0090] According to one embodiment, the electronic device (100) may acquire an image corresponding to a keyword based on at least one of style information and effect information corresponding to an identified keyword type. The image corresponding to the keyword may be an image containing the text itself corresponding to the keyword. The image corresponding to the keyword may be an image of text to which style information and effect information are applied.

[0091] Referring to FIG. 5, the electronic device (100) can identify style information (510) and effect information (520) corresponding to the identified keyword type (430). The electronic device (100) can identify style information (510) and effect information (520) corresponding to the identified keyword type (430) based on the style information (510) and effect information (520) mapped by keyword type.

[0092] For example, when an electronic device (100) is identified as a weather keyword type, which is an objective keyword type, from the keyword “cold,” it can identify style information (510) (e.g., blue color, Gothic font) and effect information (e.g., motion effect of ice forming, additional icon of falling snow) mapped to the weather keyword type.

[0093] The electronic device (100) can obtain an image (530) corresponding to a keyword based on identified style information (510) and effect information (520).

[0094] For example, style information and effect information for multiple keyword types are not limited to those shown in FIG. 5, and may be mapped to different style information and effect information, and may be mapped differently depending on user input.

[0095] According to one example, the electronic device (100) can identify style information and effect information corresponding to a keyword type through an artificial intelligence model. For example, the electronic device (100) inputs a keyword and a keyword type into an artificial intelligence model to identify style information and effect information corresponding to the keyword, and can obtain an image corresponding to the keyword based on the identified style information and effect information.

[0096] FIGS. 6 and FIGS. 7 are drawings for explaining the process of acquiring an image of an electronic device according to one or more embodiments.

[0097] Referring to FIG. 6, the electronic device (100) can obtain a first response (610) such as “The weather in London is getting really chilly these days. Especially in the mornings and evenings, the temperature drops sharply, so it is cold. So you need to make sure to wear thick clothes.” The electronic device (100) can obtain a second response (620) such as “It is getting chillier and chillier” from the user’s voice.

[0098] The electronic device (100) can identify a keyword such as “chilly” from the second response (620), and can identify a weather keyword type, which is an objective keyword type among the pre-set keyword types, from the identified keyword.

[0099] The electronic device (100) can identify style information (e.g., blue color, serif font) corresponding to the weather keyword type. The electronic device (100) can identify effect information (e.g., ice formation motion effect, up-down-left-right motion effect) corresponding to the weather keyword type.

[0100] The electronic device (100) can obtain an image (630) corresponding to a keyword based on identified style information and effect information.

[0101] Referring to FIG. 7, the electronic device (100) can obtain a first response (710) such as “Vitamin intake is important for overall health and well-being. Don’t forget to prioritize self-care by taking necessary supplements” from a user voice such as “What are some habits for getting healthy?” The electronic device (100) can obtain a second response (720) such as “Take vitamins” from a user voice.

[0102] The electronic device (100) can identify a keyword such as “vitamin” from the second response (720), and can identify an object keyword type, which is an objective keyword type among the pre-set keyword types, from the identified keyword.

[0103] The electronic device (100) can identify style information (e.g., yellow color, handwriting font) corresponding to an object keyword type. The electronic device (100) can identify effect information (e.g., pill icon, glowing motion effect) corresponding to an object keyword type.

[0104] The electronic device (100) can obtain an image (730) corresponding to a keyword based on identified style information and effect information.

[0105] FIG. 8 is a diagram illustrating a process of providing a response based on whether a user of an electronic device is identified according to one or more embodiments.

[0106] According to one embodiment, when a user is identified, the electronic device (100) can obtain a third response and a fourth response summarized from the third response through an artificial intelligence model based on at least one of conversation history information and context information between the identified user and the electronic device (100).

[0107] Conversation history information may include data on the user voice of an identified user and response data corresponding to the user voice. Conversation history information may include information on log data generated through interaction between the user and the electronic device (100).

[0108] Context information may include user profile information, preference information, and lifestyle pattern information. Context information may include information about the current situation (e.g., time zone, weather, day of the week) and user event information.

[0109] Generally, when the electronic device (100) receives a user voice (e.g., a user query, a user request), it can provide a response corresponding to the user voice. However, even if the user is identified but the user voice is not received, the electronic device (100) can provide a voice corresponding to a specific topic to the identified user based on context information or conversation history information.

[0110] For example, the third response may be a response corresponding to the user's voice, or a voice corresponding to a specific topic.

[0111] According to one embodiment, the electronic device (100) can identify a keyword included in the fourth response and a type of keyword included in the fourth response, and acquire an image corresponding to a keyword included in the fourth response based on the type of keyword included in the fourth response.

[0112] According to one embodiment, the electronic device (100) may output a voice corresponding to the third response through a speaker (120) and output an image corresponding to a keyword included in the fourth response through a display (110). Since the specific method for this has been explained through the embodiments described above, it will be omitted.

[0113] Referring to FIG. 8, in operation 810, the electronic device (100) can identify whether user voice is received.

[0114] In operation 820 (operation 810, Y), when a user voice is received, the electronic device (100) can obtain a first response corresponding to the user voice through an artificial intelligence model.

[0115] In operation 830 (operation 810, Y), when a user voice is received, the electronic device (100) can obtain a second response in which the first response is summarized through an artificial intelligence model.

[0116] In operation 840, the electronic device (100) can identify a keyword from the second response and obtain an image corresponding to the keyword.

[0117] In operation 850, the electronic device (100) can provide the user with a voice and an acquired image corresponding to the first response.

[0118] In operation 860 (operation 810, N), if the electronic device (100) does not receive a user voice, it can identify whether the user is located in the same space as the electronic device (100).

[0119] In operation 870 (S860, Y), the electronic device (100) can obtain a response or voice corresponding to a specific topic based on at least one of conversation history information and context information between the identified user and the electronic device (100).

[0120] FIG. 9 is a drawing for explaining a method for displaying an acquired image of an electronic device according to one or more embodiments.

[0121] According to one embodiment, when an image corresponding to a keyword is acquired while content is being output on the screen of a display (110), the electronic device (100) can display the image corresponding to the keyword in one area of ​​the screen.

[0122] For example, when a user is watching content through an electronic device (100), if an image corresponding to a keyword is obtained, the electronic device (100) can display the image corresponding to the keyword at the bottom left of the screen.

[0123] According to one embodiment, when an image corresponding to a keyword is acquired while content is not being displayed on the screen of the display (110), the electronic device (100) can display the image corresponding to the keyword over the entire area of ​​the screen.

[0124] For example, unless the user is watching content through the electronic device (100), the electronic device (100) can display an image corresponding to the keyword in the center of the screen as a whole area.

[0125] Referring to FIG. 9, in operation 910, the electronic device (100) can identify whether to use the full screen of the display (110) based on whether to output content.

[0126] In operation 920 (S910, N), when the electronic device (100) receives a user voice while content is not being displayed on the screen of the display (110), it can obtain a first response and a second response corresponding to the user voice.

[0127] In operation 930, the electronic device (100) can identify a keyword from the second response and generate an image corresponding to the keyword.

[0128] In operation 940, the electronic device (100) can display an image corresponding to the keyword over the entire area of ​​the display (100) screen.

[0129] In operation 950 S910, Y), when a user voice is received while content is being displayed on the screen of the display (110), the electronic device (100) can obtain a first response corresponding to the user voice and a second response summarizing the first response in one word.

[0130] In operation 960, the electronic device (100) can obtain an image corresponding to a keyword corresponding to a second response summarized in one word.

[0131] In operation 970, the electronic device (100) can display an image corresponding to a keyword in one area of ​​the display (100) screen.

[0132] FIG. 10 is a drawing for explaining a method of operation of an electronic device according to one or more embodiments.

[0133] Referring to FIG. 10, in operation 1010, when a user voice is received, the electronic device (100) can obtain a first response corresponding to the user voice and a second response summarizing the first response through an artificial intelligence model.

[0134] In operation 1020, the electronic device (100) can identify the keyword and the type of keyword included in the second response.

[0135] In operation 1030, the electronic device (100) can acquire an image corresponding to the keyword based on the type of the keyword.

[0136] In operation 1040, the electronic device (100) can output a voice corresponding to the first response and an image corresponding to the keyword.

[0137] Since the method for identifying keyword types and obtaining images corresponding to keywords has been specifically explained through the embodiments described above, a description thereof will be omitted.

[0138] The control method described in FIG. 10 can be performed by an electronic device (100) having the configuration of FIG. 2 described above, but is not necessarily limited thereto and can be performed by an electronic device having various configurations.

[0139] The various embodiments described above may be implemented as individual embodiments, or at least one embodiment may be combined with one another, either wholly or partially, to be implemented together in a single device.

[0140] According to the various embodiments described above, the electronic device (100) can provide a response corresponding to the user's voice more intuitively by providing a response corresponding to the user's voice as voice and image.

[0141] Meanwhile, the various embodiments described above may be applied to a product as embodiments alone, but at least some of their contents may be combined with other embodiments of the present disclosure to be implemented together.

[0142] The various embodiments described above may be implemented as software containing instructions stored on a machine-readable storage medium (e.g., computer). The machine may include an electronic device (e.g., electronic device (100)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions. When instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or by using other components under the control of the processor. Instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory computer-readable storage medium. Here, "non-transitory" means only that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

[0143] In addition, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided by being included in a computer program product.

[0144] Specifically, a non-transient readable storage medium or computer program product may be provided that stores computer instructions for performing, upon receiving a user voice, an operation to obtain a first response corresponding to the user voice and a second response summarizing the first response through an artificial intelligence model, an operation to identify a keyword and a type of keyword included in the second response, an operation to obtain an image corresponding to the keyword based on the type of keyword, and an operation to output a voice corresponding to the first response and an image corresponding to the keyword.

[0145] Computer program products may be distributed in the form of device-readable storage media (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0146] In addition, computer instructions or programs for performing the control method of an electronic device according to the various embodiments described above may be stored on a non-transitory computer-readable medium. When computer instructions stored on such a non-transitory computer-readable medium are executed by the processor of a specific device, they cause the specific device to perform a processing operation according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of a non-transitory computer-readable medium may include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, ROMs, etc.

[0147] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

In electronic devices, display; speaker; Memory for storing instructions; and One or more processors including processing circuitry; and The above one or more processors, When the above instructions are executed individually or collectively, the electronic device, When a user voice is received, a first response corresponding to the user voice and a second response summarizing the first response are obtained through an artificial intelligence model, and Identifying the keywords included in the second response and the types of the keywords, Based on the type of the above keyword, an image corresponding to the above keyword is obtained, and An electronic device that outputs a voice corresponding to the first response through the speaker and outputs an image corresponding to the keyword through the display. In paragraph 1, The image corresponding to the above keyword is, An electronic device that is an image containing text corresponding to the above keyword. In paragraph 1, When the above instructions are executed individually or collectively by the one or more processors, the electronic device, Identify the type of the keyword based on the attribute information of the text corresponding to the keyword included in the second response above, and The attribute information of the above text is, An electronic device comprising at least one of object information, emotion information, time information and weather information. In paragraph 3, When the above instructions are executed individually or collectively by the one or more processors, the electronic device, Identify the keyword type corresponding to the attribute of the text among a plurality of previously set keyword types, and The above-mentioned multiple keyword types are, An electronic device including a subjective keyword type and an objective keyword type. In paragraph 4, When the above instructions are executed individually or collectively by the one or more processors, the electronic device, Identifying at least one of the style information and effect information corresponding to the identified keyword type based on at least one of the plurality of keyword type-specific style information and effect information stored in the memory, and An electronic device that obtains an image corresponding to the keyword based on at least one of style information and effect information corresponding to the identified keyword type. In paragraph 5, The above style information is, It includes at least one of the text color, text shape, and text material, and The above effect information is, An electronic device comprising at least one of motion information of text and additional icon information related to text. In paragraph 1, When the above instructions are executed individually or collectively by the one or more processors, the electronic device, When the user is identified, a third response and a fourth response summarizing the third response are obtained through the artificial intelligence model based on at least one of conversation history information and context information between the identified user and the electronic device, and Identifying the keywords included in the fourth response and the types of the keywords included in the fourth response, Based on the type of keyword included in the fourth response, an image corresponding to the keyword included in the fourth response is obtained, and An electronic device that outputs a voice corresponding to the third response through the speaker and outputs an image corresponding to a keyword included in the fourth response through the display. In paragraph 1, When the above instructions are executed individually or collectively by the one or more processors, the electronic device, When an image corresponding to the keyword is acquired while content is being displayed on the screen of the above display, the image corresponding to the keyword is displayed in one area of ​​the screen, and An electronic device that displays an image corresponding to a keyword over the entire area of ​​the screen when an image corresponding to the keyword is acquired while content is not being displayed on the screen of the display. In paragraph 1, The above artificial intelligence model is, An electronic device, which is a model trained to generate a response corresponding to a user's voice and to summarize the generated response to obtain a summary response within a preset number of characters. In a method for controlling an electronic device, When a user voice is received, an operation to obtain a first response corresponding to the user voice and a second response summarizing the first response through an artificial intelligence model; An operation to identify keywords included in the second response and types of the keywords; An operation to acquire an image corresponding to the keyword based on the type of the keyword; and A control method comprising: an operation of outputting a voice corresponding to the first response and an image corresponding to the keyword. In Paragraph 10, The image corresponding to the above keyword is, A control method, which is an image containing text corresponding to the above keyword. In Paragraph 10, The operation of identifying the type of the keyword based on attribute information of the text corresponding to the keyword included in the second response; The attribute information of the above text is, A control method comprising at least one of object information, emotion information, time information and weather information. In Paragraph 12, The operation of identifying a keyword type corresponding to the attribute of the text among a plurality of previously set keyword types; is included, The above-mentioned multiple keyword types are, A control method including subjective keyword types and objective keyword types. In Paragraph 13, An operation of identifying at least one of style information and effect information corresponding to the identified keyword type based on at least one of the plurality of keyword type-specific style information and effect information stored in the electronic device; and A control method comprising: an operation of acquiring an image corresponding to the keyword based on at least one of style information and effect information corresponding to the identified keyword type. In a non-transient computer-readable storage medium storing computer instructions that cause said electronic device to perform an operation when executed by a processor of said electronic device, said operation is, When a user voice is received, an operation to obtain a first response corresponding to the user voice and a second response summarizing the first response through an artificial intelligence model; An operation to identify keywords included in the second response and types of the keywords; An operation to acquire an image corresponding to the keyword based on the type of the keyword; and A non-transient computer-readable storage medium comprising: an operation of outputting a voice corresponding to the first response and an image corresponding to the keyword.

Images