Integrated search system and method capable of providing answer including multimedia data

Abstract

An integrated search system according to one embodiment disclosed in the present document comprises: an input device; an output device; and a processor including a control module, an AI module, and a search module, wherein the control module may: when identifying a user prompt inputted through the input device, input a first instruction prompt requesting a multimedia marker related to the user prompt into the AI module together with the user prompt; obtain a primary answer including the multimedia marker as a response corresponding to the user prompt from the AI module, wherein the multimedia marker is text related to multimedia data included in the response; generate a secondary answer in which the multimedia marker of the primary answer is replaced with at least a part of the multimedia data by using the search module; and output the generated secondary answer through the output device.

Description

Integrated search system and method capable of providing answers including multimedia materials

[0001] The various embodiments disclosed in this document relate to AI-based search services.

[0002] Recently, various forms of LLM-based conversational AI have been released and are being used by many people. For example, conversational AIs include ChatGPT4o, Claude3.5, and Gemini 1.5 pro.

[0003] However, since conventional conversational AI is a language model, questions and answers are fundamentally text-based. In other words, the user asks a question in text, and the AI ​​outputs an answer in text. Consequently, when conversational AI receives photos or videos as input, it undergoes a process of converting them into text; similarly, when generating images or videos at the user's request, the AI's text output is converted into images or videos.

[0004] Therefore, while multimedia materials may be included in the general questioning process using conversational AI, the answering process excludes materials such as photos, videos, documents, charts, and graphs created directly by the user, and does not include multimedia materials obtained through external database connections or portal searches. For example, if you ask a conversational AI to explain the differences between bird sounds, the response consists solely of text because the AI's generative response does not include multimedia materials. Furthermore, although conversational AI services that ask questions and provide answers via voice have recently been launched, if asked about the differences between bird sounds, they explain the distinctions only verbally, just like text-based AI. In other words, the reality is that it is inherently impossible for LLM and conversational AI to include and present diverse and extensive multimedia materials in their responses.

[0005] Multimedia materials can be of great help in enhancing user understanding. For example, literature describing or citing three-dimensional objects such as buildings, facilities, historical sites, artifacts, places, people, paintings, plants, animals, topography, drawings, structural diagrams, and blueprints should include photographs or illustrations of the objects to improve comprehension. As another example, in literature describing a specific event or accident, adding related news footage or social network videos (e.g., YouTube) can be advantageous for aiding user understanding. As yet another example, when describing bird calls or a musician's performance or singing, bird sounds, music sounds, or voices are effective in enhancing understanding.

[0006] The various embodiments disclosed in this document may provide an integrated search system and method capable of providing answers containing multimedia materials based on an AI module. Meanwhile, in this document, the term "search system" refers to a system that provides "answers" to "questions."

[0007] An integrated search system according to one embodiment disclosed in this document includes an input device; an output device; and a processor including a control module, an AI module, and a search module. When the control module checks a user prompt entered through the input device, it inputs a first instruction prompt requesting a multimedia sign related to the user prompt to the AI ​​module together with the user prompt, obtains a first response including a multimedia sign as a response corresponding to the user prompt from the AI ​​module, wherein the multimedia sign is text related to multimedia material included in the response, generates a second response by replacing the multimedia sign of the first response with at least a part of the multimedia material using the search module, and outputs the generated second response through the output device.

[0008] The control module may request the AI ​​module, through the first instruction prompt, to generate the multimedia sign including the type and the title at a location that enhances the understanding of the answer when each multimedia material is acquired.

[0009] The above multimedia material may include at least one type of multimedia material among still images, video, sound, or documents.

[0010] The control module above can generate the first instruction prompt requesting the multimedia label capable of distinguishing at least one detailed type among a photograph, map, chart, picture, drawing, or others when the multimedia material is a still image.

[0011] The control module above can generate the first instruction prompt requesting the multimedia cover that can distinguish at least one detailed type among articles, books, reports, manuals, design documents, laws, rules, ordinances, announcements, or notices when the multimedia material is a document.

[0012] The control module may generate a second instruction prompt, either together with or separately from the first instruction prompt, requesting the multimedia label that is to be included in the first answer or is included in the first answer in a language related to the multimedia material.

[0013] The control module can generate a keyword using at least some of the multimedia covers included in the first answer, search for multimedia materials by inputting the keyword into the search module, and generate the second answer by adding at least some of the searched multimedia materials to the location of the multimedia covers within the first answer.

[0014] The above search module includes a plurality of search modules, and the control module can search for multimedia materials using a search module among the plurality of search modules that corresponds to the type of multimedia materials related to the multimedia cover.

[0015] At least a portion of the above multimedia material may be at least one of the above multimedia material or a link containing the above multimedia material.

[0016] The above search module can be implemented as a DB storing the multimedia data or as an API (Application Program Interface) of the DB.

[0017] If the multimedia material is available for purchase, the control module can generate a third response including information related to the purchase of the multimedia material and output the third response through the output device.

[0018] The control module above can generate the third answer including at least one purchase-related information among the address of an offline store, the link address of an online store, and a purchase / subscription link.

[0019] When a word or string included in the third answer is selected through the input device, the control module may perform an additional search based on the selected word or string using at least one of the AI ​​module or the search module, and provide the result of the additional search through the output device.

[0020] When a word or string included in the second answer is selected through the input device, the control module may perform an additional search based on the selected word or string using at least one of the AI ​​module or the search module, and provide the result of the additional search through the output device.

[0021] The control module above can provide the results of the additional search to include the multimedia sign using the AI ​​module and the search module.

[0022] The control module inputs a fourth instruction prompt requesting the generation of a multimedia search term based on at least a portion of the multimedia signage to the AI ​​module, obtains at least one search term from the AI ​​module in response to the fourth instruction prompt, and can search the multimedia material using the obtained search term.

[0023] Additionally, an integrated search method by at least one processor according to one embodiment disclosed in this document may include: an operation of obtaining an input user prompt; an operation of providing a first instruction prompt requesting a multimedia sign related to the user prompt to an AI module together with the user prompt; an operation of obtaining a first answer including a multimedia sign as a response corresponding to the user prompt from the AI ​​module; an operation of generating a second answer in which the multimedia sign is text related to multimedia material included in the response and the multimedia sign is replaced with at least a part of the multimedia material through a search module; and an operation of outputting the generated second answer.

[0024] The operation provided above may include an operation of requesting the generation of the multimedia sign including type and title at a location to improve understanding of the first answer when each multimedia material is acquired by the AI ​​module through the first instruction prompt.

[0025] According to the various embodiments disclosed in this document, the present invention can provide answers including multimedia materials based on an AI module. In addition, various effects that can be identified directly or indirectly through this document may be provided.

[0026] FIG. 1 shows a configuration diagram of an integrated search system according to one embodiment.

[0027] Figure 2 shows the search results according to the conventional method.

[0028] FIG. 3 shows a search prompt created by adding an instruction prompt according to one embodiment to a first user prompt querying 'appearance' according to one embodiment, and a first response from an AI module to the same.

[0029] FIG. 4 shows a search prompt created by adding an instruction prompt according to one embodiment to a second user prompt querying 'sound' according to one embodiment, and a first response from an AI module to the same.

[0030] FIG. 5 shows a search prompt created by adding an instruction prompt according to one embodiment to a third user prompt that queries how to use an electronic product according to one embodiment, FIG. 6 shows a first answer from an AI module regarding this, and FIG. 1 shows a first answer according to the expansion of the instruction prompt.

[0031] FIG. 8 shows a flowchart of an integrated search method according to one embodiment.

[0032] FIGS. 9 and FIGS. 10 show modified embodiments of FIG. 3.

[0033] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0034] FIG. 1 shows a configuration diagram of an integrated search system according to one embodiment.

[0035] Referring to FIG. 1, an integrated search system (100) according to one embodiment may include an input device (110), an output device (120), a memory (130), and a processor (140). In one embodiment, some components of the integrated search system (100) may be omitted, or additional components may be included. Additionally, some of the components of the integrated search system (100) may be combined to form a single entity, while performing the same functions as the components prior to combination. In one embodiment, the integrated search system (100) may be or include at least one computing device among a mobile terminal, a smartphone, a smartpad, a laptop, a notebook, and a PC. The integrated search system (100) may be, for example, a user terminal using an integrated search service.

[0036] The input device (110) can receive user input using the integrated search system (100). The input device (110) may include at least one input detection circuit among, for example, a keypad, a button, a touchscreen, and a microphone.

[0037] The output device (120) can visually or audibly output at least one data of a symbol, a number, or a character under the control of the processor (140). The output device (120) may include, for example, at least one output device among a liquid crystal display, an OLED, a touchscreen display, and a speaker.

[0038] Memory (130) may include various forms of volatile or non-volatile memory. For example, memory (130) may include ROM (read only memory), RAM (random access memory), and SSD (Solid State Drive). In one embodiment, memory (130) may be located inside or outside the processor (140), and memory (130) may be connected to the processor (140) through various known means. Memory (130) may store various data used by at least one component of the integrated search system (100) (e.g., processor (140)). The data may include, for example, input data or output data for software and related commands. For example, memory (130) may store at least one instruction and data for providing an integrated search service. Memory (130) may store instructions and data for performing operations of, for example, the control module (141), the AI ​​module (143), and the search module (145). The memory (130) can store instructions and data for generating, for example, a first instruction prompt, a second instruction prompt, and a third instruction prompt.

[0039] The processor (140) can control at least one other component (e.g., a hardware or software component) of the integrated search system (100) and can perform various data processing or operations. The processor (140) may include, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, an application processor, an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), and may have multiple cores.

[0040] According to one embodiment, the processor (140) may include a control module (141), an AI module (143), and a search module (145). The control module (141), the AI ​​module (143), and the search module (145) may be software / hardware modules included in the processor (140) or executed by the processor (140). According to various embodiments, the processor (140) may further include a designated app corresponding to a dedicated app or a general-purpose app that is installed in the integrated search system (100) and provides integrated search functions. In this case, the control module (141), the AI ​​module (143), and the search module (145) may be included in the designated app or may each perform a designated function according to a request from the designated app.

[0041] According to one embodiment, the AI ​​module (143) may provide a response to a prompt input (or provided) by the control module (141). For example, the AI ​​module (143) may be at least one of the following: a large language model (LLM) itself, a conversational AI combining an LLM and a message management module (e.g., LangChain), or an AI platform with some functions added to an LLM (e.g., OpenAI’s AI Assistant). Additionally or alternatively, the AI ​​module (143) may be a conversational AI web service accessible via a Robotic Process Automation (RPA) module. The AI ​​module (143) may be an Application Program Interface (API) module for connecting to an AI service server connected via a communication network.

[0042] According to one embodiment, the search module (145) can perform a search corresponding to a keyword using a portal site or another designated site in accordance with the command of the control module (141). For example, the search module (145) may be implemented as a database storing multimedia materials or as an API (Application Program Interface) of the database.

[0043] According to one embodiment, the control module (141) can obtain a user prompt entered by a user through the input device (110). The user prompt may include, for example, words, strings or sentences, or multimedia materials that query content to be searched. If the multimedia material is a document, it may be included in the prompt using the RAG (Retrieval Augmented Generation) method. For example, the control module (141) can obtain a user prompt entered by a user in text or voice through the input page of a designated app.

[0044] According to one embodiment, when the control module (141) obtains a user prompt, it may input (or provide) a query including the user prompt and a first instruction prompt to the AI ​​module (143). For example, the control module (141) may input to the AI ​​module (143) a first instruction prompt requesting a multimedia sign related to the user prompt, by adding it to the user prompt. The first instruction prompt may include at least one definition of the insertion target, insertion location, or format of the multimedia sign within the response corresponding to the user prompt. The multimedia sign may include at least one of a still image sign, a video sign, a sound sign, and a document sign.

[0045] According to one embodiment, the control module (141) may generate an instruction prompt to instruct the AI ​​module to include multimedia material in the answer corresponding to the user prompt. For example, the control module (141) may generate a first instruction prompt that specifies the location where the multimedia material is to be inserted (e.g., inserting it in a part that would be helpful for understanding if the multimedia material is included) and requests a multimedia label so that the type and title (or name) of each multimedia material can be distinguished. The multimedia material may be, for example, at least one type of material among still images, video, sound, or documents.

[0046] For example, the first instruction prompt for a video label may be [If the answer contains a video, insert a video label in a part that aids understanding, and the video label should be in the format [[Video Type: Video Name]]. For example, the first instruction prompt for a sound label may be [If the answer contains sound, insert a sound label in a part that aids understanding, and the sound label should be in the format [[Sound Type: Sound Name]]. For example, the first instruction prompt for a still image label may be [If the answer contains a still image, insert a still image label in a part that aids understanding, and the still image label should be in the format [[Still Image Type: Still Image Name]].

[0047] In one embodiment, the type of still image may be one of, for example, a photograph, a map, a chart, a graph, a picture, or a drawing. In this case, the photograph may be a picture of a map, a picture, a drawing, a chart, or a graph. The control module (141) may request the creation of the multimedia sign indicating at least one detailed type among a photograph, a map, a chart, a picture, a drawing, or others for the still image sign via the first instruction prompt as follows.

[0048]

[0049] If the above multimedia material is a document, the control module (141) may request the multimedia cover indicating at least one detailed type among an article, book, report, manual, design document, law, rule, ordinance, notice, or guide for the document cover through the first instruction prompt as follows.

[0050]

[0051] According to one embodiment, the control module (141) may request the multimedia label, which can distinguish each multimedia type and title, at a location related to each multimedia material through the first instruction prompt. For example, the control module (141) may specify the format of the multimedia label to be included in the first response as [[Multimedia Material Type: Multimedia Material Name]] through the first instruction prompt.

[0052] According to one embodiment, the control module (141) may obtain a primary answer including a multimedia label as a response to a user prompt and a first instruction prompt from the AI ​​module (143). For example, the control module (141) may input a user prompt and a first instruction prompt into the AI ​​module (143) and obtain a primary answer including a multimedia label as a response from the AI ​​module (143). The primary answer may include, for example, an answer corresponding to the user prompt, but if the answer includes multimedia material, the answer may include a multimedia label created to distinguish the type and title of each multimedia material at a relevant location for each multimedia material.

[0053] According to one embodiment, the control module (141) can generate a second instruction prompt for search language expansion that requests the addition of multimedia markers (e.g., types and titles of multimedia materials) included in the first answer in another language, and input the second instruction prompt into the AI ​​module (143).

[0054]

[0055] Accordingly, the control module (141) can convert (expand or replace) the multimedia markers of the first language within the first answer from the AI ​​module (143) into the second language, and convert the first answer to include the multimedia markers of the first language and the multimedia markers of the second language within the first answer.

[0056] According to one embodiment, the control module (141) can generate a secondary answer by using the search module (145) to replace each multimedia sign in the primary answer with at least a part of the multimedia material associated therewith. At least a part of the multimedia material may be the multimedia material itself or at least one piece of information among the links containing (e.g., posted) the multimedia material.

[0057] According to one embodiment, the control module (141) can separate multimedia markers from the obtained primary answer and generate keywords using at least a portion of the multimedia markers (e.g., title). The control module (141) can input the keywords into the search module (145) to search for multimedia materials through the search module (145). At least a portion of the multimedia markers used as keywords may be, for example, the title of the multimedia materials.

[0058] In addition, information on the multimedia cover can be input into the AI ​​module (143) along with a prompt for generating search terms as shown below to generate a list of search terms, which can then be used for portal search or DB search.

[0059] [Multimedia Cover Information = … … .]

[0060] [Instruction Prompt 100: Please generate a list of up to 5 search terms so that the above multimedia cover information can be used for portal or database searches.]

[0061] Additionally or generally, the control module (141) can generate a second answer by using the AI ​​module (143) to replace (substitute) the multimedia marker included in the first answer with at least a part of the retrieved multimedia material. At least a part of the multimedia material may be, for example, at least one piece of information such as multimedia material or a link containing multimedia material.

[0062] For example, the control module (141) may generate a third instruction prompt requesting that the retrieved multimedia material be inserted at the multimedia cover location of the first answer. The control module (141) inputs the first answer, the multimedia material, and the third instruction prompt into the AI ​​module (143) and obtains a second answer from the AI ​​module (143) in which the multimedia material is appropriately, at least partially, inserted at the multimedia cover location of the first answer.

[0063] Meanwhile, according to the example, the AI ​​module (143) may be an LLM / LMM that is trained or fine-tuned to answer a user's question following the first instruction prompt and the second instruction prompt. Additionally, the LLM may be an LLM / LMM configured to call the plurality of search modules. Furthermore, it may be an LLM / LMM trained to perform a task according to the third instruction prompt.

[0064] According to one embodiment, when the search module (145) includes a plurality of search modules, the control module (141) can search for multimedia materials using the search module (145) corresponding to the type of multimedia material designated on each multimedia label among the plurality of search modules.

[0065] For example, if the multimedia cover is a photograph, the control module (141) can search for the photograph from an image / photo database such as Getty Images Bank or an image / photo database of a newspaper or broadcasting company. For example, the control module (141) can obtain multimedia materials corresponding to search keywords (including search conditions) from the multimedia cover by using a search module (145) implemented as an API (Application Program Interface) of Getty Images Bank.

[0066] For another example, if the multimedia cover is a book, the control module (141) can search for book-related information (e.g., still images, links) through a book sales site using a portal search means. Alternatively, if the multimedia cover is a video, the control module (141) can search for the video on a video provider site such as YouTube. In contrast, if the multimedia cover is a newspaper article or a report, the control module (141) can search for related newspaper articles or reports on a newspaper company site or a sales site.

[0067] As described above, the control module (141) may generate a secondary answer by replacing the multimedia markers with multimedia materials obtained as search results. For example, the control module (141) may generate a secondary answer such as Figure 9 or Figure 10 by replacing the multimedia markers in the primary answer of Figure 3 with multimedia materials. For example, as in Figure 9, a secondary answer may be generated in which all multimedia markers are replaced with multimedia materials. Alternatively, as in Figure 10, a secondary answer may be generated in which some of the multimedia markers are replaced with multimedia materials and other parts include links related to the multimedia materials.

[0068] According to one embodiment, if the multimedia material included in the second answer is available for purchase, the control module (141) can generate a third answer containing information related to the purchase of the multimedia material and output the third answer through the output device (120).

[0069] For example, the control module (141) can generate a third answer that includes at least one purchase-related information among the address of an offline store related to multimedia material, a link address of an online store, or a purchase / subscription link.

[0070] As another example, if multimedia material photos are obtained using the API (Application Program Interface) of Getty Images Bank, the control module (141) may provide a third response that additionally includes relevant information or links where the photos can be purchased.

[0071] As another example, if the multimedia cover is a book, the control module (141) may generate a third answer by activating a shopping tab found on the portal based on the first answer and presenting a link to purchase the book.

[0072] Alternatively, if the multimedia cover is a video, the control module (141) may present a third answer that includes information related to paid subscriptions found based on the first answer. Alternatively, if the multimedia cover is a newspaper article or a report, the control module (141) may present a third answer that includes information related to purchases on the seller's webpage found based on the first answer.

[0073] According to various embodiments, when text of a word or string included in a second answer or a third answer is selected through the input device (110), the control module (141) may perform an additional search based on the selected text using at least one of the AI ​​module (143) or the search module (145) and provide a fourth answer including the results of the additional search. Additionally or generally, the control module (141) may provide an additional search including multimedia signs using the AI ​​module (143) and the search module (145). The results of the additional search may be provided in a separate window. The results of the additional search may perform a re-search using search keywords on the selected text and multimedia signs and present the re-search results as an answer including the multimedia signs described above.

[0074] According to various embodiments, the integrated search system (100) may be a single computing device or may be composed of a plurality of devices. For example, some components of the integrated search system (100) may be included in one device, and the remaining components may be included in another device. In this case, since the one device and the other device are connected through various communication channels, the one device and the other device may further include a communication module that supports the establishment of a communication channel and the performance of communication through the established communication channel. The communication channel may include, for example, at least one communication channel among LAN, FTTH, xDSL, Wibro, Wireless LAN, Wi-Fi, Bluetooth, Zigbee, Wi-Fi Direct (WFD), Ultrawideband (UWB), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), 3G, 4G, or 5G.

[0075] According to the above-described embodiment, the control module (141) can create a multimedia sign in a language that is judged to be more advantageous for content search depending on the type of multimedia material included in the first answer through the second instruction prompt. For example, most users tend to create user prompts in a familiar language they frequently use, such as Korean for Koreans and Japanese for Japanese. However, the artificial intelligence model (e.g., AI module (143) or search module (145)) can provide different answers to prompts depending on the language used during learning, and can provide more accurate or more search results in the learned language keywords. More specifically, when 'Apollo 11' is included in the multimedia sign, the search module (145) can provide much richer and more accurate search results when searching for English content using the English 'Apollo 11' as a keyword, rather than the Korean 'Apollo 11' or the Japanese 'アポロ11'. Taking into account the characteristics of this AI model, the control module (141) can generate a richer and more accurate second answer based on the first answer through search language expansion.

[0076] According to the above-described embodiment, the control module (141) can create a multimedia sign in a language that is judged to be more advantageous for content search depending on the type of multimedia material included in the first answer through the second instruction prompt. For example, most users tend to create user prompts in a familiar language they frequently use, such as Korean for Koreans and Japanese for Japanese. However, the artificial intelligence model (e.g., AI module (143) or search module (145)) can provide different answers to prompts depending on the language used during learning, and can provide more accurate or more search results in the learned language keywords. More specifically, when 'Apollo 11' is included in the multimedia sign, the search module (145) can provide much richer and more accurate search results when searching for English content using the English 'Apollo 11' as a keyword, rather than the Korean 'Apollo 11' or the Japanese 'アポロ11'. Taking into account the characteristics of this AI model, the control module (141) can generate a richer and more accurate second answer based on the first answer through search language expansion.

[0077] In this way, the integrated search system (100) according to one embodiment can improve user understanding by attaching an image or video related to the answer content to the text answer result of the AI ​​module (143).

[0078] In addition, the integrated search system (100) according to one embodiment is combined with the previously implemented AI module (143) to provide an enhanced service, so that even though the user uses the AI ​​service in a familiar way using their terminal, it can provide answers that are richer in content than the conventional AI module which includes various multimedia materials and purchase links.

[0079] Hereinafter, the effects of an integrated search system according to one embodiment will be described with reference to FIGS. 2 to 7.

[0080] FIGS. 2 and FIGS. 3 show conventional and integrated search results for a first user prompt querying 'appearance'. FIG. 2 shows conventional AI-based search results, and FIG. 3 shows integrated search results according to one embodiment.

[0081] On the other hand, as shown in FIG. 3, when the control module (141) checks the input first user prompt (210), it can input a first instruction prompt (310) to the AI ​​module (143) in addition to the first user prompt (210). The first instruction prompt (310) may include first instruction prompts A through E. The first instruction prompt A may define the insertion conditions and insertion location of a multimedia sign, and the first instruction prompt B may define the type of multimedia sign. The first instruction prompt C may define the classification conditions and format of a still image sign, the first instruction prompt D may define the classification conditions and format of a video sign, and the first instruction prompt E may define the classification conditions and format of a sound sign.

[0082] Accordingly, the AI ​​module (143) can generate a primary answer (320) to which multiple multimedia markers (321, 322, 323, 324, 325) have been added, and provide the generated primary answer (320) to the control module (141). For example, the multiple multimedia markers (321, 322, 323, 324, 325) can each be configured in the format [Type of multimedia material (e.g., 'photo', 'video'): Name of each multimedia material].

[0083] As shown in FIG. 3, in the case of a first user prompt (210) requesting an explanation of an appearance (gait), the search results of the AI ​​module (143) related to the first user prompt (210) may include photos and videos. In this case, the AI ​​module (143) may generate a multimedia cover in the format [Photo / Video: Title of Photo / Video] that relates the type of the searched photo and video and a title describing the content thereof in response to the first instruction prompt. Additionally, the AI ​​module (143) generates a primary answer (320) related to multimedia materials by adding the multimedia cover as a response corresponding to the first user prompt (210), and the control module (141) may obtain the primary answer (320) generated by the AI ​​module (143) from the AI ​​module (143).

[0084] FIG. 4 shows an example of an integrated search result for a second user prompt querying 'sound' according to one embodiment.

[0085] Referring to FIG. 4, when the control module (141) detects a second user prompt (410) requesting an explanation of a sound, it can input (or transmit) a query including the second user prompt (410) and the first instruction prompt (420) to the AI ​​module (143).

[0086] The AI ​​module (143) can perform a search based on the second user prompt (410) and search for multimedia materials corresponding to the second user prompt (410), such as the sound of a cuckoo and the sound of a crow, as part of the search results. The AI ​​module (143) can generate a multimedia label in the format [Sound: Title describing the sound] that associates the searched sound with the title of each sound, and generate a first answer (430) by inserting the generated multimedia label into the answer corresponding to the second user prompt (410).

[0087] Accordingly, the control module (141) can obtain a first response (430) corresponding to the response of the second user prompt (410) with two sound markers [[Sound: Cuckoo call]] (431) and [[Sound: Crow call]] (432) inserted from the AI ​​module (143).

[0088] FIGS. 5 to 7 show example diagrams of an integrated search service related to the use of an electronic product according to one embodiment.

[0089] Referring to FIG. 5, the control module (141) can obtain a third user prompt (510) that inquires about how to use the Marantz amplifier model 30.

[0090] When the control module (141) obtains the third user prompt (510), it can input the third user prompt (510) and the first instruction prompt (520) into the AI ​​module (143) as shown in FIG. 5. The first instruction prompt (520) may be the first instruction prompt (310, 420) with the addition of the first instruction prompt F related to the format definition of the document cover. Since the first instruction prompt (520) is natural language, the format or sentence may change slightly as the first instruction prompt F is added. However, if a similar meaning is included, the control module (141) can obtain the same or similar primary answer from the first instruction prompt (520).

[0091] Referring to FIG. 6, the control module (141) can obtain a first response (530) from the AI ​​module (143) in response to the third user prompt (510), which includes photo covers (531, 532, 533, 535, 536, 537), a video cover (534), and document covers (538, 539). It can be seen that the first response (530) further includes a document cover such as [[Document: Marantz Model 30 User Manual]] according to the first instruction prompt F.

[0092] The AI ​​module (143) indicated the type of multimedia cover as 'document' and the document name as '… manual' as the cover of the searched Marantz Model 30 user manual, such as [[Document: Marantz Model 30 User Manual]] (538). Accordingly, the control module (141) according to one embodiment can easily search for 'Marantz Model 30 user manual' by setting the document name as a search keyword.

[0093] The control module (141) can input a second instruction prompt for language expansion to the AI ​​module (143) for expressing multimedia signs included in the first answer in another language. For example, the control module (141) can input (provide) the second instruction prompt to the AI ​​module (143) after the first answer according to the first instruction prompt has been obtained.

[0094]

[0095] According to one embodiment, the AI ​​module (143) may insert multimedia signs in multiple languages ​​as multimedia signs in response to the second instruction prompt. For example, since the 'Marantz amplifier' is an American product, the AI ​​module (143) may further insert English multimedia signs (541, 542, 543, 544, 545, 546, 547, 548, 549) after the multimedia signs (531, 532, 533, 534, 535, 526, 537, 538, 539) included in the first answer of FIG. 6 in response to the aforementioned second instruction prompt.

[0096] For example, the AI ​​module (143) can convert the still image label [[Photo: Amplifier Power Button Location]] (531) into English and add it as [[photo: Amplifier Power Button Location]] (541), and convert the still image label [[Photo: Speaker Terminal Connection Method]] (532) into English and add it as [[photo: Speaker Terminal Connection]] (542). Additionally, it can be seen that the AI ​​module (143) converts the video label [[Video: How to Turn On Marantz Amplifier]] (534) into English and adds it as [[video: How to Turn On Marantz Amplifier]] (544). Furthermore, it can be seen that the AI ​​module (143) converts the document label [[Document: Marantz Model 30 User Manual]] (538) into English and adds it as [[document: Marantz Model 30 User Manual]] (548).

[0097] According to various embodiments, the control module (141) may input a second instruction prompt to the AI ​​module (143) simultaneously with the first instruction prompt. In this case, the AI ​​module (143) may provide a corrected version of the first answer according to FIG. 7 immediately without providing the first answer according to FIG. 6.

[0098] In this way, a control module (141) according to one embodiment can provide a primary answer including multilingual multimedia markers according to the language associated with the keyword in the input first user prompt. FIG. 8 shows a flowchart of an integrated search method according to one embodiment.

[0099] Referring to FIG. 8, in operation 610, the integrated search system (100) can obtain an input user prompt.

[0100] In operation 620, the integrated search system (100) may provide a first instruction prompt requesting a multimedia sign related to the user prompt to the AI ​​module (143) along with the user prompt. The first instruction prompt may include at least one definition of the insertion target, insertion location, or format of the multimedia sign within the response to the user prompt.

[0101] In operation 630, the integrated search system (100) can obtain a primary answer including a multimedia label as a response corresponding to a user prompt from the AI ​​module (143). For example, the integrated search system (100) can perform a search based on a user prompt through the AI ​​module (143) and organize the search results to construct a response. When the integrated search system (100) identifies multimedia material in the search results, it can generate and obtain a primary answer by inserting a multimedia label related to the identified multimedia material into the answer being constructed. The multimedia label is configured, for example, in the format [[Type of multimedia material: Multimedia material name]], and the multimedia material name may include text describing each multimedia material.

[0102] In operation 640, the integrated search system (100) can generate a second answer by replacing the multimedia cover of the first answer with at least a part of the multimedia material through the search module (145). For example, the integrated search system (100) can separate the title of the multimedia cover and perform a search using the search module (145) with the title of the multimedia cover as a keyword. The integrated search system (100) can generate a second answer by obtaining multimedia material through the keyword-based search results and replacing the multimedia cover of the first answer with at least a part of the obtained multimedia material (e.g., a link or a photo).

[0103] In operation 650, the integrated search system (100) can output the generated secondary answer through the output device (120). Subsequently, when a user selects at least one text among the words, strings, or sentences included in the output secondary answer, the control module (141) can provide a tertiary answer corresponding to the selected text through at least one module among the AI ​​module (143) or the search module (145). For example, the control module (141) can generate and provide a tertiary answer through a process similar to the process of generating the secondary answer described above, using the selected text as a keyword.

[0104] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another corresponding component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0105] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0106] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., internal memory or external memory) readable by a machine (e.g., an integrated search system). For example, a processor (e.g., processor (140)) of a device (e.g., integrated search system (100)) may call at least one of one or more instructions stored from a storage medium and execute it. This enables the device to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium readable by the device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0107] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0108] Components according to various embodiments of this document may be implemented in software or in hardware form, such as a digital signal processor (DSP), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC), and may perform specific roles. The term "components" is not limited to software or hardware, and each component may be configured to reside in an addressable storage medium or configured to run one or more processors. As an example, components may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0109] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the components of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to the integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In the integrated search system, Input device; output device; and It includes a processor comprising a control module, an AI module, and a search module, and The above control module is, When a user prompt input through the above input device is checked, a first instruction prompt requesting a multimedia sign related to the user prompt is input into the AI ​​module together with the user prompt, and Obtaining a primary response including a multimedia label as a response corresponding to the user prompt from the above AI module, and Using the above search module, multimedia data corresponding to the above multimedia cover of the above first answer is searched and obtained from a DB or the Internet, and a second answer is generated that includes or replaces at least a part of the obtained multimedia data, and The above-generated secondary answer is output through the above-mentioned output device, and An integrated search system in which the above multimedia cover is composed of text related to multimedia materials included in the above response.

2. In claim 1, the control module is, An integrated search system that requests the AI ​​module to generate the multimedia label in a specified format capable of distinguishing the type and title of each multimedia material through the first instruction prompt.

3. In claim 2, the control module is, An integrated search system that requests the generation of a multimedia sign including the type and title at a location to enhance understanding of the answer when each multimedia material is acquired by the AI ​​module through the first instruction prompt.

4. In Claim 1, the multimedia material is, An integrated search system comprising at least one type of multimedia material among still images, video, sound, or documents.

5. In claim 4, the control module is, An integrated search system that generates the first instruction prompt requesting the multimedia label capable of distinguishing at least one detailed type among photographs, maps, charts, drawings, diagrams, or others when the multimedia material is a still image.

6. In claim 4, the control module is, An integrated search system that generates the first instruction prompt requesting the multimedia label capable of distinguishing at least one detailed type among articles, books, reports, manuals, design documents, laws, rules, ordinances, announcements, or notices, when the multimedia material is a document.

7. In claim 2, the control module is, An integrated search system that generates a second instruction prompt, together with or separately from the first instruction prompt, requesting the multimedia label to be included in or included in the first answer in the language related to the multimedia material.

8. In claim 1, the control module is, Generate keywords using at least some of the multimedia covers included in the above first answer, input the keywords into the search module to search for multimedia materials, and An integrated search system that generates a second answer by adding at least a portion of the searched multimedia material to the location of the multimedia cover in the first answer.

9. In Claim 8, The above search module includes a plurality of search modules, and The above control module is an integrated search system that searches for multimedia materials using a search module corresponding to the type of multimedia materials related to the multimedia sign among the plurality of search modules.

10. In claim 1, at least a portion of the multimedia material is, An integrated search system that is at least one of the above multimedia material or a link containing the above multimedia material.

11. In claim 1, the search module is, An integrated search system that is implemented as a DB storing the above multimedia data or as an API (Application Program Interface) of the above DB.

12. In claim 1, the control module is, An integrated search system that, if the above multimedia material is available for purchase, generates a third answer including information related to the purchase of the above multimedia material and outputs the third answer through the above output device.

13. In claim 12, the control module is, An integrated search system that generates the above third answer including at least one purchase-related information among the address of an offline store, the link address of an online store, and a purchase / subscription link.

14. In claim 13, the control module is, An integrated search system that, when a word or string included in the third answer is selected through the input device, performs an additional search based on the selected word or string using at least one of the AI ​​module or the search module, and provides the result of the additional search through the output device.

15. In claim 1, the control module is, An integrated search system that, when a word or string included in the second answer is selected through the input device, performs an additional search based on the selected word or string using at least one of the AI ​​module or the search module, and provides the result of the additional search through the output device.

16. In claim 15, the control module is, An integrated search system that provides the results of the additional search to include the multimedia sign using the AI ​​module and the search module.

17. In claim 15, the control module is, An integrated search system that inputs a fourth instruction prompt requesting the generation of a multimedia search term based on at least a portion of the above-mentioned multimedia cover into the AI ​​module, obtains at least one search term from the AI ​​module in response to the fourth instruction prompt, and searches the multimedia material using the obtained search term.

18. As an integrated search method by at least one processor, An operation of confirming a user prompt input through an input device, and providing a first instruction prompt requesting a multimedia sign related to the user prompt to the AI ​​module together with the user prompt; An operation of obtaining a primary response including a multimedia marker as a response corresponding to the user prompt from the AI ​​module; the multimedia marker is composed of text related to multimedia material included in the response, and An operation to search for and obtain multimedia data corresponding to the multimedia cover of the first answer in a DB or the Internet through a search module, and to generate a second answer that includes or replaces at least a part of the obtained multimedia data; and An integrated search method including the operation of outputting the generated secondary answer through an output device.

19. In claim 18, the operation provided is, An integrated search method comprising, when each multimedia material is acquired by the AI ​​module through the first instruction prompt, an operation to request the generation of the multimedia sign including a type and a title at a location to enhance the understanding of the first answer.

Images