Information processing system, information processing method, and information processing program

Abstract

An information processing system includes a data acquisition unit configured to acquire data of any modal related to a user, a search unit configured to search for external data based on the data, a query acquisition unit configured to acquire a query from the user, and an output control unit configured to output, based on the external data and the query, response data.

Description

INFORMATION PROCESSING SYSTEM, INFORMATION PROCESSING METHOD, AND INFORMATION PROCESSING PROGRAM

[0001] The present disclosure relates to an information processing system, an information processing method, and an information processing program.

[0002] Patent Literature 1 describes acquiring a context of an interaction between a user and an artificial intelligence (AI) character in a system that prefetches a result from a large-scale language model (LLM), and predicting an utterance content by the user based on the context. Furthermore, Patent Literature 1 describes generating a query to the LLM based on a predicted utterance content, and generating a predicted response of an AI character model with respect to the predicted utterance contents by the user based on a response acquired from the LLM. In addition, Patent Literature 1 describes providing a predicted response to a user in a case where a level of inconsistency between a word uttered by the user and a predicted word is below a predetermined threshold value.

[0003] Patent Literature 1: US 11,960,983

[0004] However, in order to improve accuracy of a content of a response, in a case where a response is generated using a method (retrieval augmented generation (RAG)) of generating the response using searched external knowledge for a context, the search may take time and the response may be delayed.

[0005] Therefore, the present disclosure proposes an information processing system, an information processing method, and an information processing program capable of quickly providing a high-quality response.

[0006] According to the present disclosure, an information processing system includes: a data acquisition unit configured to acquire data of any modal related to a user; a search unit configured to search for external data based on the data; a query acquisition unit configured to acquire a query from the user; and an output control unit configured to output, based on the external data and the query, response data.

[0007] Fig. 1 is a diagram illustrating an example of a configuration of an information processing system according to a first embodiment of the present disclosure.Fig. 2 is a block diagram illustrating an example of a functional configuration of a terminal device according to the first embodiment of the present disclosure.Fig. 3 is a diagram illustrating an example of a flow of data according to the first embodiment of the present disclosure.Fig. 4 is a flowchart illustrating an example of response processing according to the first embodiment of the present disclosure.Fig. 5 is a block diagram illustrating an example of a functional configuration of a terminal device according to a second embodiment of the present disclosure.Fig. 6 is a diagram illustrating an example of a flow of data according to the second embodiment of the present disclosure.Fig. 7 is a flowchart illustrating an example of response processing according to the second embodiment of the present disclosure.Fig. 8 is a diagram illustrating an example of a flow of data in a use case UC1.Fig. 9 is a diagram illustrating an example of interaction in the use case UC1.Fig. 10 is a diagram illustrating an example of a flow of data in a use case UC2.Fig. 11 is a diagram illustrating an example of interaction in the use case UC2.Fig. 12 is a diagram illustrating an example of a flow of data in a use case UC3.Fig. 13 is a diagram illustrating an example of interaction in the use case UC3.Fig. 14 is a diagram illustrating an example of a flow of data in a use case UC4.Fig. 15 is a diagram illustrating an example of interaction in the use case UC4.Fig. 16 is a diagram illustrating an example of interaction in a use case UC5.Fig. 17 is a diagram illustrating an example of interaction in a use case UC6.Fig. 18 is a hardware configuration diagram illustrating an example of a computer configured to implement a function of an information processing apparatus.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. It is noted that, in each of the following embodiments, the same parts are denoted by the same reference numerals, and a redundant description will be omitted.

[0009] The present disclosure will be described according to the following order of items. 1. Configuration of system according to first embodiment 2. Configuration of terminal device according to first embodiment 3. Flow of data of response processing according to first embodiment 4. Procedure of information processing according to first embodiment 5. Configuration of terminal device according to second embodiment 6. Flow of data of response processing according to second embodiment 7. Procedure of information processing according to second embodiment 8. Use case 9. About generation model 10. Modification of embodiment 11. Hardware configuration

[0010] (Embodiment) <<1. Configuration of system according to first embodiment>> Fig. 1 is a diagram illustrating an example of a configuration of an information processing system according to a first embodiment of the present disclosure. As illustrated in Fig. 1, an information processing system 1 includes a terminal device 100, a general server 200, and an artificial intelligence (AI) server 300. The terminal device 100, the general server 200, and the AI server 300 are communicably connected to each other in a wired or wireless manner via, for example, a network N. It is noted that, in the following description, the general server 200 and the AI server 300 are separated from each other, but may be configured to be the same server.

[0011] The terminal device 100 is an information processing apparatus operated by a user who uses interaction AI (hereinafter, also referred to as an AI agent). The terminal device 100 receives a voice input by a user's utterance or an input by a user's operation. Furthermore, the terminal device 100 receives multimodal data input such as input of an image captured by a camera. The terminal device 100 outputs a response sentence generated by the AI server 300 by voice, screen display, or the like.

[0012] The general server 200 is an information processing apparatus that searches for sentences and images based on the multimodal data input. The general server 200 includes, for example, an image group, a web search engine, various databases, and the like, and provides information on various searches from the terminal device 100.

[0013] The AI server 300 is an information processing apparatus including a generation model that generates a response sentence based on a prompt input from the terminal device 100. The AI server 300 generates a response sentence by LLM, for example, and provides an interaction with the AI agent to the user of the terminal device 100.

[0014] That is, the information processing system 1 performs a method (RAG: Retrieval Augmented Generation) of generating a response using searched external knowledge for a context. While using the RAG, the information processing system 1 performs recognition and search asynchronously with the interaction by the AI agent and stores the latest context so that a waiting time due to the search for the external knowledge does not occur until a query from the user is received and a response is generated. The information processing system 1 can minimize the time loss of the response by generating the response using the stored context at the time of the utterance by the AI agent.

[0015] <<2. Configuration of terminal device according to first embodiment>> Fig. 2 is a block diagram illustrating an example of a functional configuration of the terminal device according to the first embodiment of the present disclosure. As illustrated in Fig. 2, the terminal device 100 includes a display unit 101, an operation unit 102, a camera 103, a microphone 104, a speaker 105, a communication unit 110, a storage unit 120, and a control unit 130. Examples of the terminal device 100 include a personal computer, a smartphone, a vehicle-mounted terminal, and the like.

[0016] The display unit 101 is a display device for displaying various types of information. The display unit 101 is implemented by, for example, a liquid crystal display, an organic electro luminescence (EL) display, or the like as a display device. The display unit 101 displays various screens such as conversation sentences between the user and the interaction AI, search results, and the like.

[0017] The operation unit 102 is an input device that receives various operations from a user who operates the terminal device 100. The operation unit 102 is implemented by, for example, a keyboard, a mouse, a touch panel, or the like as an input device. It is noted that the display device of the display unit 101 and the input device of the operation unit 102 may be integrated like a display with a touch panel.

[0018] The camera 103 captures, for example, an image of the user or the like who operates the terminal device 100. The camera 103 captures an image using, for example, a complementary metal oxide semiconductor (CMOS) image sensor, a charge coupled device (CCD) image sensor, or the like as an imaging element. The camera 103 photoelectrically converts light received by the imaging element and performs A / D conversion to generate an image. The camera 103 outputs the captured image to the control unit 130.

[0019] For example, the microphone 104 acquires a voice of a user or the like who operates the terminal device 100. As the microphone 104, for example, various microphones such as an electret condenser microphone can be used. The microphone 104 outputs a voice signal of the acquired voice to the control unit 130.

[0020] The speaker 105 outputs, for example, an utterance content of the AI agent. As the speaker 105, for example, various speakers such as a dynamic type and a condenser type can be used. The speaker 105 outputs sound based on the voice signal input from the control unit 130.

[0021] The communication unit 110 is implemented by, for example, a network interface card (NIC), a wireless local area network (LAN) such as Wi-Fi (registered trademark), or the like. The communication unit 110 is a communication interface that is connected to the general server 200 and the AI server 300 via the network N in a wired or wireless manner and manages communication of information with the general server 200 and the AI server 300. For example, the communication unit 110 transmits an image or the like captured by the camera 103 to the general server 200. Furthermore, the communication unit 110 receives, for example, a search result of image search from the general server 200. Furthermore, the communication unit 110 transmits, for example, a prompt or the like including a context to the AI server 300. Furthermore, the communication unit 110 receives, for example, response data (response sentence (generation statement) of interaction) generated by a generation model from the AI server 300.

[0022] The storage unit 120 is implemented by, for example, a random access memory (RAM), a semiconductor memory element such as a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 includes a context storage unit 121, an interaction history storage unit 122, and a summary storage unit 123. In addition, the storage unit 120 stores information (programs and data) used for processing in the control unit 130.

[0023] The context storage unit 121 stores a context extracted from the search result acquired from the general server 200. For example, the context storage unit 121 may sequentially store a plurality of contexts such as a first in first out (FIFO) format, and sequentially discard old contexts after a certain period of time has elapsed.

[0024] The interaction history storage unit 122 stores a history of interaction between the user and the AI agent. The interaction history storage unit 122 may store the history of the interaction separately into a portion from which a superficial memory in units of episodes can be extracted from the entire history of the interaction and a portion from which archived detailed contents of individual episodes can be extracted. It is noted that the interaction history storage unit 122 may be omitted in a case where the interaction history is not used for processing.

[0025] The summary storage unit 123 stores a summary of the interaction generated from the history of the interaction between the user and the AI agent. The summary storage unit 123 is referred to, for example, in a case where the interaction between the user and the AI agent becomes long and the topic at the beginning of the interaction is mentioned, so that the AI agent continues a conversation without forgetting a flow of the conversation. The summary storage unit 123 may be omitted in a case where the summary of the interaction is not used for processing.

[0026] The control unit 130 is implemented by, for example, a central processing unit (CPU), a micro processing unit (MPU), or the like configured to execute a program stored in an internal storage device using a RAM as a work area. Furthermore, the control unit 130 may be implemented by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0027] The control unit 130 includes a data acquisition unit 131, a search unit 132, a query acquisition unit 133, and an output control unit 134, and implements or executes a function and an action of information processing to be described below. It is noted that the internal configuration of the control unit 130 is not limited to the configuration illustrated in Fig. 2, and may be another configuration as long as information processing to be described later is performed.

[0028] The data acquisition unit 131 acquires data of any modal related to the user via input devices of the terminal device 100, such as the operation unit 102, the camera 103, and the microphone 104. It is noted that, in the present embodiment, the modal indicates a type of data such as an image, a voice, and a text. The data acquisition unit 131 acquires, for example, a text by key input from the user via the operation unit 102. Furthermore, the data acquisition unit 131 acquires, for example, a captured image equivalent to the user's field of view via the camera 103. Furthermore, the data acquisition unit 131 acquires, for example, the voice of the user, the surrounding environmental sound, and the like via the microphone 104. It is noted that the voice of the user may be a conversation between the users. The data acquisition unit 131 outputs data of any modal such as the acquired text, captured image, and voice to the search unit 132. It is noted that the data of any modal may also be simply referred to as input data.

[0029] When input data is input from the data acquisition unit 131, the search unit 132 transmits the input data to the general server 200 via the communication unit 110 and the network N. In the general server 200, data corresponding to the input data is searched for based on the input data. The searched data may be data of various modalities. For example, the searched data may be data such as a sentence and an image. That is, the input data is data including at least one of image data, voice data, and text data. The search unit 132 receives a search result corresponding to the input data from the general server 200 via the network N and the communication unit 110. The search unit 132 extracts a context from the received search result and stores the context in the context storage unit 121. Here, in the search based on the input data in the search unit 132, when the input of the input data from the data acquisition unit 131 is started, the search is executed prior to a query from the user in the interaction between the user and the AI agent. That is, the search based on the input data in the search unit 132 is executed without synchronizing with the query from the user.

[0030] The query acquisition unit 133 acquires, for example, a text by key input from the user via the operation unit 102. Furthermore, the query acquisition unit 133 acquires, for example, a voice or the like of the user via the microphone 104. When acquiring the voice of the user, the query acquisition unit 133 performs voice recognition and acquires recognized texts. The query acquisition unit 133 acquires these texts as a query. For example, the query acquisition unit 133 executes morphological analysis or the like on the input text, and acquires the text as a query in a case where it is determined as an inquiry sentence with respect to the AI agent. The query acquisition unit 133 outputs the acquired query to the output control unit 134.

[0031] When a query is input from the query acquisition unit 133, the output control unit 134 refers to the context storage unit 121 and acquires, for example, the latest context. It is noted that the output control unit 134 may refer to the context storage unit 121 and, for example, acquire a plurality of contexts from the latest one. The output control unit 134 transmits a prompt including the query and the context to the AI server 300 via the communication unit 110 and the network N. In the AI server 300, the prompt is used as an input of a generation model, and response data is generated by the generation model. The output control unit 134 receives response data corresponding to the transmitted prompt from the AI server 300 via the network N and the communication unit 110. The output control unit 134 displays a generation statement based on the received response data on the display unit 101, and outputs the voice of the generation statement from the speaker 105. That is, the modal of the data acquired by the data acquisition unit 131 and the modal of the response data output by the output control unit 134 may be different types of modals.

[0032] <<3. Flow of data of response processing according to first embodiment>> Next, a flow of data of the response processing according to the first embodiment will be described. Fig. 3 is a diagram illustrating an example of a flow of data according to the first embodiment of the present disclosure. In Fig. 3, an example of an AI agent that can comment on an image captured by the camera 103 will be described. It is noted that, in Fig. 3, the operation unit 102, the camera 103, and the microphone 104 are collectively referred to as an input device 111, and the display unit 101 and the speaker 105 are collectively referred to as an output device 112.

[0033] The terminal device 100 starts acquisition of an image and a voice from the input device 111. For example, the data acquisition unit 131 starts acquisition of an image from the camera 103, and the query acquisition unit 133 starts acquisition of a voice of the user from the microphone 104. That is, input of input data from the input device 111 to the data acquisition unit 131 is started. It is noted that the image in the input data may be, for example, a video that is seen by the user in a metaverse. Furthermore, the image in the input data may be, for example, an image captured by a vehicle-mounted camera that captures an image of the outside of the vehicle or an in-vehicle camera that captures an image of the inside of the vehicle in a vehicle-mounted AI. Furthermore, the image in the input data may be, for example, an image captured by an in-camera installed on a storefront signage and configured to capture an image of a person looking at a signage.

[0034] The data acquisition unit 131 outputs the acquired image to the search unit 132. The search unit 132 transmits the input image to the general server 200. It is noted that the search unit 132 may convert the input image into a sentence and may transmit the sentence to the general server 200. In addition, it is assumed that the search using the general server 200 by the search unit 132 is executed at a first timing. Furthermore, the search using the general server 200 by the search unit 132 may be executed only when the user is detected in combination with input data, for example, detection of the user or the like by the camera 103 or the like. In this case, the camera 103 or the like is an example of a detection unit that detects the motion of the user, and the search unit 132 executes the search in a case where the motion of the user is detected. Furthermore, the detection unit such as the camera 103 detects, for example, one or more pieces of information of the line of sight and gesture of the user.

[0035] For example, on a non player character (NPC) on the screen, the search by the search unit 132 may be executed only in a case where a character of the NPC is shown in a visible range. Furthermore, for example, in a virtual reality (VR) metaverse, the search by the search unit 132 may be executed only in a case where a user avatar is looking at a character. Furthermore, for example, in the storefront signage, the search by the search unit 132 may be executed only in a case where the user stands in front of the storefront signage. Furthermore, for example, the search may be started with the start of the conversation between the users as a trigger. In these cases, it is possible to suppress continuous search by the search unit 132 and to suppress costs.

[0036] Upon receiving an image, a sentence, and the like from the terminal device 100, the general server 200 executes a search using an image search engine, a web search engine, and the like. The general server 200 transmits a search result corresponding to the received image, sentence, and the like to the terminal device 100.

[0037] Upon receiving the search result from the general server 200, the search unit 132 of the terminal device 100 extracts a context from the received search result and stores the context in the context storage unit 121. For example, assuming that "〇〇 tower" is shown in an image captured by the camera 103, the search unit 132 receives a sentence including "〇〇 tower" as a search result from the general server 200. The search unit 132 extracts "〇〇 tower" as a context from the sentence and stores the context in the context storage unit 121. Here, the contexts sequentially stored in the context storage unit 121 by the search unit 132 are a back ground (BG) context.

[0038] In a state in which the BG context is stored in the context storage unit 121, the query acquisition unit 133 of the terminal device 100 acquires, for example, the voice of the user from the microphone 104. The query acquisition unit 133 performs voice recognition on the voice of the user and acquires a recognized text as a query. For example, the query acquisition unit 133 acquires a query such as "What is that building?" as an input text by voice recognition. The query acquisition unit 133 outputs the acquired query to the output control unit 134.

[0039] When the query is input from the query acquisition unit 133, the output control unit 134 refers to the context storage unit 121 and acquires, for example, the latest context "〇〇 tower". The output control unit 134 transmits a prompt including the query and the latest context to the AI server 300.

[0040] Upon receiving the prompt from the terminal device 100, the AI server 300 generates response data (generation statement) using a generation model. The AI server 300 transmits generated response data corresponding to the prompt to the terminal device 100.

[0041] Upon receiving the response data from the AI server 300, the output control unit 134 of the terminal device 100 outputs the received response data (generation statement), for example, "That is 〇〇 tower" from the output device 112. The generation statement may be displayed on the display unit 101 or may be output as a voice from the speaker 105. In addition, when the received response data is a plurality of generation statements, the output control unit 134 may enumerate and display the response data on the display screen of the display unit 101, as illustrated in search results SR1 to SR3. It is noted that the output of the response data (generation statement) by the output control unit 134 is executed at a second timing after the first timing of the search using the general server 200 by the search unit 132. As described above, the information processing system 1 separates the search processing of the search unit 132 from the response processing of the output control unit 134, thereby enabling an interaction using the latest context without search time loss at a necessary timing. As described later, for example, it is effective in processing in which a search can be performed prior to a question, such as image recognition, or processing in which a time lag is allowed, such as searching for miscellaneous knowledge related to conversation to make a topic of conversation.

[0042] <<4. Procedure of information processing according to first embodiment>> Next, information processing according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart illustrating an example of response processing according to the first embodiment of the present disclosure.

[0043] The data acquisition unit 131 of the terminal device 100 starts acquisition of an image from the camera 103 as an example of input data (step S1). The data acquisition unit 131 outputs the acquired image to the search unit 132. When the image acquired from the data acquisition unit 131 is input, the search unit 132 transmits the image to the general server 200 and executes a search based on the acquired image (step S2). The search unit 132 receives a search result corresponding to the image from the general server 200. The search unit 132 extracts a context from the received search result (step S3). The search unit 132 stores the extracted context in the context storage unit 121 (step S4).

[0044] The query acquisition unit 133 acquires, for example, a voice or the like of a user via the microphone 104. The query acquisition unit 133 performs voice recognition on the voice of the user and acquires the recognized text as a query (step S5). The query acquisition unit 133 outputs the acquired query to the output control unit 134.

[0045] When the query is input from the query acquisition unit 133, the output control unit 134 refers to the context storage unit 121 and acquires the latest context. The output control unit 134 transmits a prompt including the latest context to the AI server 300 and inputs the prompt to a generation model (step S6). The output control unit 134 receives response data from the AI server 300. The output control unit 134 outputs a generation statement, which is the received response data, from the output device 112 (step S7). As a result, the information processing system 1 can quickly make a high-quality response. Furthermore, the information processing system 1 can enhance user experience by quickly starting and continuing a response each time while providing a high-quality response using multi-modal recognition that takes time for processing and highly accurate knowledge.

[0046] That is, the information processing system 1 includes the data acquisition unit 131, the search unit 132, the query acquisition unit 133, and the output control unit 134. The data acquisition unit 131 acquires data of any modal related to the user. The search unit 132 searches for external data (for example, a search destination document of the general server 200) based on the data. The query acquisition unit 133 acquires a query from the user. The output control unit 134 outputs response data (for example, a generation statement generated by the AI server 300) based on the external data and the query. As a result, the information processing system 1 can quickly make a high-quality response.

[0047] Furthermore, the search by the search unit 132 is executed at a first timing before the query is acquired, and the output by the output control unit 134 is executed at a second timing after the first timing. As a result, since the information processing system 1 can perform the search prior to the acquisition of the query by interaction, the information processing system 1 can make a response using the latest context without causing the user to feel search time loss at a desired timing.

[0048] In addition, the search unit 132 extracts a context from the search result of the external data. In addition, the output control unit 134 outputs response data based on the extracted context and the query. As a result, the information processing system 1 can quickly make a response using the latest context.

[0049] In addition, the output control unit 134 uses a prompt including the context as an input of the generation model, and outputs response data generated by the generation model as response data. As a result, the information processing system 1 can provide an interaction by the AI agent.

[0050] In addition, the search unit 132 stores the plurality of extracted contexts in the storage unit (the context storage unit 121). As a result, the information processing system 1 can quickly make a response using the latest context.

[0051] Furthermore, an information processing program causes the terminal device 100 (computer) to acquire data of any modal related to the user. Furthermore, the information processing program transmits the external data based on the data to the general server 200, and causes the terminal device 100 to search for the external data at the first timing before the query is acquired. Furthermore, the information processing program causes the terminal device 100 to receive a search result of the external data from the general server 200 and extract a context from the search result. Furthermore, the information processing program causes the terminal device 100 to acquire a query from the user. Furthermore, the information processing program causes the terminal device 100 to transmit a prompt including the extracted context and query to the AI server 300. Furthermore, the information processing program causes the terminal device 100 to receive response data generated by the generation model of the AI server 300 and output the received response data as response data based on the query. Here, the output of the response data is executed at a second timing after the first timing. As a result, the information processing program can quickly make a high-quality response.

[0052] <<5. Configuration of terminal device according to second embodiment>> In the first embodiment described above, the latest context is acquired from the context storage unit 121 in response to the acquisition of the query and is input to the generation model. However, search may be performed in real time according to the utterance of the user, and an embodiment in this case will be described as a second embodiment. It is noted that, in the second embodiment, the same components as those of the terminal device 100 of the first embodiment are denoted by the same reference numerals, and description of the overlapping components and operations will be omitted.

[0053] Fig. 5 is a block diagram illustrating an example of a functional configuration of a terminal device according to the second embodiment of the present disclosure. A terminal device 100a illustrated in Fig. 5 includes a control unit 130a instead of the control unit 130 as compared with the terminal device 100 of the first embodiment. Compared with the control unit 130, the control unit 130a includes an output control unit 134a instead of the output control unit 134. Furthermore, the control unit 130a further includes a real-time search unit 135. It is noted that, in the control unit 130a, the query acquisition unit 133 outputs an acquired query to the real-time search unit 135.

[0054] The real-time search unit 135 sequentially generates response data based on an intermediate result of the search in a synchronous search for receiving and searching for a user's question. The real-time search unit 135 performs, for example, three-stage sequential processing such as pre-processing, search, and post-processing. The real-time search unit 135 performs, for example, query conversion as pre-processing. For example, when a query is input from the query acquisition unit 133, the real-time search unit 135 refers to the context storage unit 121 and acquires, for example, the latest context (BG context). It is noted that the real-time search unit 135 may refer to the context storage unit 121 and may acquire, for example, a plurality of BG contexts from the latest one. The real-time search unit 135 performs query conversion based on the acquired BG context. The real-time search unit 135 outputs the context subjected to the query conversion to the output control unit 134a as a real time (RT) context. Here, the RT context is an example of an intermediate result of the real-time search.

[0055] The real-time search unit 135 transmits the RT context to the general server 200 via the communication unit 110 and the network N. In the general server 200, for example, a vector search is performed based on the RT context. The real-time search unit 135 receives a search result corresponding to the RT context from the general server 200 via the network N and the communication unit 110. For example, it is assumed that the real-time search unit 135 has received four search results. The real-time search unit 135 outputs the received search results (four search results) to the output control unit 134a as intermediate results of the real-time search.

[0056] The real-time search unit 135 performs, for example, re-ranking as post-processing based on the received search result. The real-time search unit 135 outputs, for example, search results higher than half of the search results after re-ranking to the output control unit 134a. For example, when receiving four search results, the real-time search unit 135 outputs the top two search results to the output control unit 134a as search results after re-ranking.

[0057] That is, the real-time search unit 135 sequentially searches for the general server 200 (external data) based on the BG context and the query.

[0058] When the RT context is input from the real-time search unit 135, the output control unit 134a transmits a prompt including the RT context to the AI server 300 via the communication unit 110 and the network N. In the AI server 300, the prompt is used as an input of a generation model, and response data is generated by the generation model. The output control unit 134a receives response data corresponding to the transmitted prompt from the AI server 300 via the network N and the communication unit 110. The output control unit 134a displays a generation statement based on the received response data on the display unit 101, and outputs the voice of the generation statement from the speaker 105.

[0059] When an intermediate result of the real-time search is input from the real-time search unit 135, the output control unit 134a transmits a prompt including the intermediate result of the real-time search to the AI server 300 via the communication unit 110 and the network N, and similarly, response data is generated. As in the case of the RT context, the output control unit 134a displays a generation statement based on the response data on the display unit 101, and outputs the voice of the generation statement from the speaker 105.

[0060] When a search result after re-ranking is input from the real-time search unit 135, the output control unit 134a transmits a prompt including the search result after re-ranking to the AI server 300 via the communication unit 110 and the network N, and similarly, response data is generated. As in the case of the RT context, the output control unit 134a displays a generation statement based on the response data on the display unit 101, and outputs the voice of the generation statement from the speaker 105. In this manner, the output control unit 134a can sequentially output the generation statement of the response data in response to the intermediate result of the real-time search.

[0061] <<6. Flow of data of response processing according to second embodiment>> Next, a flow of data of the response processing according to the second embodiment will be described. Fig. 6 is a diagram illustrating an example of a flow of data according to the second embodiment of the present disclosure. In Fig. 6, an example of an AI agent capable of describing an image captured by the camera 103 will be described. It is noted that, in Fig. 6, the operation unit 102, the camera 103, and the microphone 104 are collectively referred to as an input device 111, and the display unit 101 and the speaker 105 are collectively referred to as an output device 112.

[0062] The terminal device 100a starts acquisition of an image and a voice from the input device 111. For example, the data acquisition unit 131 starts acquisition of an image from the camera 103, and the query acquisition unit 133 starts acquisition of a voice of the user from the microphone 104. That is, input of input data from the input device 111 to the data acquisition unit 131 is started. The data acquisition unit 131 outputs the acquired image to the search unit 132. The search unit 132 transmits the input image to the general server 200. It is noted that the search unit 132 may convert the input image into a sentence and may transmit the sentence to the general server 200. In addition, it is assumed that the search using the general server 200 by the search unit 132 is executed at the first timing before the query is acquired.

[0063] Upon receiving an image, a sentence, and the like from the terminal device 100a, the general server 200 executes a search using an image search engine, a web search engine, and the like. The general server 200 transmits a search result corresponding to the received image, sentence, and the like to the terminal device 100a.

[0064] Upon receiving the search result from the general server 200, the search unit 132 of the terminal device 100a extracts a context from the received search result and stores the context in the context storage unit 121. For example, assuming that "〇〇 tower" is shown in an image captured by the camera 103, the search unit 132 receives a sentence including "〇〇 tower" as a search result from the general server 200. The search unit 132 extracts "〇〇 tower" as a context from the sentence and stores the context in the context storage unit 121. Here, the contexts sequentially stored in the context storage unit 121 by the search unit 132 are assumed to be a BG context.

[0065] In a state in which the BG context is stored in the context storage unit 121, the query acquisition unit 133 of the terminal device 100a acquires, for example, the voice of the user from the microphone 104. The query acquisition unit 133 performs voice recognition on the voice of the user and acquires a recognized text as a query. For example, the query acquisition unit 133 acquires a query such as "How big do you think that is?" as an input text by voice recognition. The query acquisition unit 133 outputs the acquired query to the real-time search unit 135.

[0066] When the query is input from the query acquisition unit 133, the real-time search unit 135 performs query conversion as pre-processing. The real-time search unit 135 refers to the context storage unit 121 and acquires the latest context (BG context). For example, the real-time search unit 135 acquires "〇〇 tower" as the BG context from the context storage unit 121, and converts the input query into a RT context "height of 〇〇 tower". The real-time search unit 135 outputs the RT context "height of 〇〇 tower" to the output control unit 134a as an intermediate result of the real-time search.

[0067] When the RT context "height of 〇〇 tower" is input from the real-time search unit 135, the output control unit 134a transmits a prompt including the RT context "height of 〇〇 tower" to the AI server 300.

[0068] Upon receiving the prompt from the terminal device 100a, the AI server 300 generates response data (generation statement) using a generation model. The AI server 300 transmits generated response data corresponding to the prompt to the terminal device 100a.

[0069] Upon receiving the response data from the AI server 300, the output control unit 134a of the terminal device 100a outputs the received response data (generation statement), for example, "I will check the height of 〇〇 tower" from the output device 112. The generation statement may be displayed on the display unit 101 or may be output as a voice from the speaker 105. It is noted that the output of the response data (generation statement) by the output control unit 134a is executed at the second timing after the first timing of the search using the general server 200 by the search unit 132.

[0070] On the other hand, the real-time search unit 135 searches for a RT context "height of 〇〇 tower" while the output control unit 134a is processing response data (generation statement) corresponding to the RT context "height of 〇〇 tower". The real-time search unit 135 transmits the RT context "height of 〇〇 tower" to the general server 200.

[0071] When receiving the RT context "height of 〇〇 tower" from the terminal device 100a, the general server 200 executes a vector search. The general server 200 transmits a search result corresponding to the received RT context "height of 〇〇 tower" to the terminal device 100a.

[0072] The real-time search unit 135 receives the search result corresponding to the RT context "height of 〇〇 tower" from the general server 200. For example, it is assumed that the real-time search unit 135 has received four search results such as "use", "size", "structure", and "year of construction". The real-time search unit 135 outputs the received four search results to the output control unit 134a as intermediate results of the real-time search.

[0073] When the intermediate results of the real-time search are input from the real-time search unit 135, the output control unit 134a transmits a prompt including the intermediate results of the real-time search (four search results) to the AI server 300.

[0074] Upon receiving the prompt from the terminal device 100a, the AI server 300 generates response data (generation statement) using a generation model. The AI server 300 transmits generated response data corresponding to the prompt to the terminal device 100a.

[0075] Upon receiving the response data from the AI server 300, the output control unit 134a of the terminal device 100a outputs the received response data (generation statement), for example, "The 〇〇 tower is...,..." from the output device 112. The generation statement may be displayed on the display unit 101 or may be output as a voice from the speaker 105.

[0076] The real-time search unit 135 performs re-ranking as post-processing while the output control unit 134a is processing the response data (generation statement) corresponding to four search results. For example, when the top two search results after the re-ranking are "size" and "structure", the real-time search unit 135 outputs "size" and "structure" to the output control unit 134a as the search results after re-ranking.

[0077] When the search result after the re-ranking is input from the real-time search unit 135, the output control unit 134a transmits a prompt including "size" and "structure", which are the search results after the re-ranking, to the AI server 300.

[0078] Upon receiving the prompt from the terminal device 100a, the AI server 300 generates response data (generation statement) using a generation model. The AI server 300 transmits generated response data corresponding to the prompt to the terminal device 100a.

[0079] Upon receiving the response data from the AI server 300, the output control unit 134a of the terminal device 100a outputs the received response data (generation statement), for example, "The height is 〇〇〇 m" and "This is a self-supporting tower and is the 〇-th tallest in the world" from the output device 112. The generation statement may be displayed on the display unit 101 or may be output as a voice from the speaker 105. In addition, the output control unit 134a may enumerate and display the contents of sequential responses on the display screen of the display unit 101, as illustrated in the search results SR1 to SR3. As described above, in a case where a search is performed in response to a user's question, the information processing system 1 is required to perform a synchronous search in real time. However, by sequentially generating responses based on the intermediate results of the search, it is possible to quickly start a response and produce a natural interaction without waiting for a time to obtain a highly accurate search result.

[0080] <<7. Procedure of information processing according to second embodiment>> Next, information processing according to the second embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart illustrating an example of response processing according to the second embodiment of the present disclosure. In the following description, since the processing of steps S1 to S5 of the response processing is similar to that of the first embodiment, the description thereof will be omitted.

[0081] The real-time search unit 135 of the terminal device 100a executes the following processing subsequent to the processing in step S5. When the query is input from the query acquisition unit 133, the real-time search unit 135 performs query conversion as pre-processing. The real-time search unit 135 refers to the context storage unit 121 and acquires the latest context (BG context). The real-time search unit 135 converts the input query into a RT context based on the BG context. The real-time search unit 135 outputs the RT context to the output control unit 134a as an intermediate result of the real-time search. The real-time search unit 135 continues to transmit the RT context to the general server 200 and receive a search result. The real-time search unit 135 outputs the search result to the output control unit 134a as an intermediate result of the real-time search. The real-time search unit 135 continues to perform re-ranking as post-processing. The real-time search unit 135 outputs the search result after the re-ranking to the output control unit 134a. That is, the real-time search unit 135 sequentially outputs the context of the intermediate result of the real-time search to the output control unit 134a.

[0082] The RT context, the search result, and the search result after the re-ranking are sequentially input from the real-time search unit 135 to the output control unit 134a. The output control unit 134a sequentially transmits prompts including each of these to the AI server 300, and sequentially inputs the prompts to generation model. That is, the output control unit 134a sequentially inputs the prompts including the context based on the intermediate result of the real-time search to the generation model (step S11).

[0083] The output control unit 134a sequentially receives the response data from the AI server 300. The output control unit 134a sequentially outputs generation statements as the received response data from the output device 112 (step S12). As a result, the information processing system 1 can quickly start a response and produce a natural interaction without waiting for a highly accurate search result.

[0084] That is, the information processing system 1 includes the real-time search unit 135 that sequentially searches for external data based on the context and the query. In addition, the output control unit 134a outputs the response data based on the query and the response data based on the search result of the real-time search unit 135 as response data. As a result, the information processing system 1 can quickly start a response and produce a natural interaction.

[0085] <<8. Use case>> Next, a use case using the information processing system according to each embodiment of the present disclosure will be described with reference to Figs. 8 to 17. It is noted that, in the description of Figs. 8 to 15, an example of the terminal device 100 of the first embodiment will be described, but the present invention can also be applied to the terminal device 100a of the second embodiment. In addition, in Figs. 8, 10, 12, and 14, the operation unit 102, the camera 103, and the microphone 104 are collectively referred to as an input device 111, and the display unit 101 and the speaker 105 are collectively referred to as an output device 112. Furthermore, in the interaction examples of Figs. 9, 11, 13, and 15 to 17, the AI agent is simply represented as an agent. It is noted that, in each interaction example, an operation of each unit of the information processing system 1 is represented as an AI agent that interacts with the user.

[0086] <Use case UC1> Fig. 8 is a diagram illustrating an example of a flow of data in a use case UC1. Fig. 9 is a diagram illustrating an example of an interaction in the use case UC1. The use case UC1 illustrated in Figs. 8 and 9 is an example of performing storage related to a talk theme and a search of miscellaneous knowledge.

[0087] The data acquisition unit 131 of the terminal device 100 starts acquisition of an image and a voice from the input device 111. For example, the data acquisition unit 131 starts acquisition of the voice of the user from the microphone 104. That is, input of input data from the input device 111 to the data acquisition unit 131 is started.

[0088] The data acquisition unit 131 performs voice recognition on the voice of the user and outputs a recognized text (sentence) to the search unit 132. The search unit 132 transmits the input text to the general server 200. Furthermore, the search unit 132 refers to the interaction history storage unit 122 and searches for the input text. It is noted that it is assumed that the search using the general server 200 by the search unit 132 is executed at a first timing.

[0089] Upon receiving the text from the terminal device 100, the general server 200 executes a search using a past interaction database, a web search engine, or the like. The general server 200 transmits a search result corresponding to the received text to the terminal device 100.

[0090] Upon receiving the search result from the general server 200, the search unit 132 of the terminal device 100 extracts a BG context from the received search result and stores the BG context in the context storage unit 121. Furthermore, the search unit 132 extracts a BG context from the search result of the interaction history storage unit 122 and stores the BG context in the context storage unit 121. For example, the search unit 132 extracts BG contexts such as "episode of travel in Hokkaido", "episode in which camera is forgotten during travel in Hokkaido", and "sightseeing boat event", and stores the BG contexts in the context storage unit 121.

[0091] In a state in which the BG contexts are stored in the context storage unit 121, the query acquisition unit 133 of the terminal device 100 acquires the voice of the user from the microphone 104. The query acquisition unit 133 performs voice recognition on the voice of the user and acquires a recognized text as a query. For example, the query acquisition unit 133 acquires a query such as "I am going on a trip to Hakone next time" as an input text by voice recognition. The query acquisition unit 133 outputs the acquired query to the output control unit 134. It is noted that the voice of the user is similarly recognized in the data acquisition unit 131, and the recognized text is output to the search unit 132. Furthermore, the voice recognition may be shared between the data acquisition unit 131 and the query acquisition unit 133.

[0092] When a query is input from the query acquisition unit 133, the output control unit 134 refers to the context storage unit 121, and acquires, for example, contexts "episode of travel in Hokkaido", "episode in which camera is forgotten during travel in Hokkaido", and "sightseeing boat event". The output control unit 134 transmits a prompt including the query and the context to the AI server 300, and receives response data (generation statement). The output control unit 134 outputs the received response data (generation statement) via the output device 112. In addition, the output control unit 134 outputs an interaction history to the search unit 132 and stores the interaction history in the interaction history storage unit 122 via the search unit 132. It is noted that the output control unit 134 may store the interaction history in the interaction history storage unit 122 without passing through the search unit 132.

[0093] In an interaction example C1 illustrated in Fig. 9, when the user says "I am going on a trip to Hakone next time", the AI agent first responds with "GOOD!". The terminal device 100 further searches for a portion from which a superficial memory of the interaction history storage unit 122 can be extracted, and the AI agent responds with "You went to Hokkaido the other day". In response to this, the user responds with "That’s right". When the user makes a response, the terminal device 100 decompresses a portion of the interaction history storage unit 122, from which archived detailed contents can be extracted, and further, the AI agent makes a response of "At that time, you said {I forgot the camera}. Please be careful". Here, a portion surrounded by curly brackets is a response based on the decompressed archive. Furthermore, the terminal device 100 further searches for the interaction history storage unit 122 for a memory with another user, and the AI agent responds with "In Hakone, {there are many people who talk about the sightseeing boat}". Here, a portion surrounded by curly brackets is a response based on the memory with another user. As described above, in the use case UC1, the memory of the past interaction and the memory with another user can be searched asynchronously with the interaction with the user, and can be used and responded in the case of being used in the interaction. That is, although the time required for a search may be different between a portion from which a superficial memory in units of episodes can be extracted from the entire interaction history storage unit 122 and a portion from which archived detailed contents of each individual episode can be extracted, the memory can be contextualized from the interaction history up to the last moment.

[0094] An interaction example C2 illustrated in Fig. 9 is a scene following the interaction example C1. In the interaction example C2, the terminal device 100 searches for miscellaneous knowledge related to Hakone. Next, when the user says "Yes, I'm going to get on a sightseeing boat", the AI agent responds with "It feels good. It seems like there is {an event in which dogs can board a sightseeing boat together} now". Here, a portion surrounded by curly brackets is a response based on a Web search of the general server 200 by the search unit 132. As described above, in the use case UC1, it is possible to search for related miscellaneous knowledge in the interaction and to respond by using the related miscellaneous knowledge in a case where the related miscellaneous knowledge can be used in the interaction.

[0095] That is, the output control unit 134 stores a query and response data in the storage unit (the interaction history storage unit 122) as an interaction history with the user. In addition, the search unit 132 further searches for the interaction history stored in the storage unit. As a result, the information processing system 1 can respond by using a memory of a past interaction or a memory with another user in a case where the memory can be used in an interaction.

[0096] <Use case UC2> Fig. 10 is a diagram illustrating an example of a flow of data in a use case UC2. Fig. 11 is a diagram illustrating an example of an interaction in the use case UC2. The use case UC2 illustrated in Figs. 10 and 11 is an example in which an AI agent talks with a user while performing investigation in the background. In the use case UC2, an AI server 200a having a generation model is used instead of the general server 200.

[0097] The data acquisition unit 131 of the terminal device 100 starts acquisition of an image and a voice from the input device 111. For example, the data acquisition unit 131 starts acquisition of the voice of the user from the microphone 104. That is, input of input data from the input device 111 to the data acquisition unit 131 is started.

[0098] The data acquisition unit 131 performs voice recognition on the voice of the user and outputs a recognized text and an interaction history to the search unit 132. The search unit 132 stores the interaction history in the interaction history storage unit 122. In addition, the search unit 132 transmits the interaction history to the AI server 200a. It is noted that, in a case where the recognized text is not in the interaction history, for example, the search unit 132 may add the recognized text to the interaction history as a statement of another user.

[0099] The AI server 200a generates a summary from the interaction history using a generation model. Upon receiving the interaction history from the terminal device 100, the AI server 200a generates a summary using the generation model. The AI server 200a transmits the generated summary to the terminal device 100.

[0100] Upon receiving the summary from the AI server 200a, the search unit 132 of the terminal device 100 stores the received summary in the summary storage unit 123. In addition, the search unit 132 extracts a BG context from the summary and stores the BG context in the context storage unit 121. The search unit 132 may extract the BG context from a search result of the interaction history storage unit 122 and store the BG context in the context storage unit 121. Furthermore, in a case where there is a preset purpose, the search unit 132 may generate the next action from the purpose and the summary. The next action is, for example, a topic to talk next, an item to be determined within a predetermined time, or the like. The search unit 132 extracts the BG context from the generated next action and stores the BG context in the context storage unit 121. It is noted that generation of the summary and the next action by the search unit 132 using the AI server 200a is executed at the first timing.

[0101] In a state in which the BG contexts are stored in the context storage unit 121, the query acquisition unit 133 of the terminal device 100 acquires the voice of the user from the microphone 104. The query acquisition unit 133 performs voice recognition on the voice of the user and acquires a recognized text as a query. For example, the query acquisition unit 133 acquires a query such as "As for the second spot,..." as an input text by voice recognition. The query acquisition unit 133 outputs the acquired query to the output control unit 134. It is noted that the voice of the user is similarly recognized in the data acquisition unit 131, and the recognized text is output to the search unit 132. Furthermore, the voice recognition may be shared between the data acquisition unit 131 and the query acquisition unit 133.

[0102] When the query is input from the query acquisition unit 133, the output control unit 134 refers to the context storage unit 121, and acquires, for example, a summary of the interaction, a topic to be discussed next, and the like as the context. The output control unit 134 transmits a prompt including the query and the context to the AI server 300, and receives response data (generation statement). The output control unit 134 outputs the received response data (generation statement) via the output device 112. In addition, the output control unit 134 outputs an interaction history to the search unit 132 and stores the interaction history in the interaction history storage unit 122 via the search unit 132. It is noted that the output control unit 134 may store the interaction history in the interaction history storage unit 122 without passing through the search unit 132.

[0103] In an interaction example C3 illustrated in Fig. 11, it is assumed that the AI agent is interacting with a user while proposing recommended spots to the user one after another. At this time, the terminal device 100 generates a summary of spots proposed so far and reactions behind the interaction. Here, when the user says "As for the second spot,...", the terminal device 100 refers to the summary of the summary storage unit 123 and the AI agent responds with "It is 〇〇 Park. That is...". As described above, in the use case UC2, by generating the summary of the conversation, the interaction can be continued without forgetting the flow of the conversation so far.

[0104] Furthermore, in a case where the AI agent has a purpose of interaction, for example, in an one-hour meeting, in a case where it is desired to decide decision items on four agendas, the AI agent, while interacting with the user, extracts an agenda, the decision item of which has not been decided and sets time allocation. Furthermore, for example, in a case where it is desired to make a contract in sales, the AI agent predicts what proposal is effective next to this customer (user) from the past contract pattern while interacting with the user. As described above, in the use case UC2, the interaction can be continued so as to guide the user by making an interaction strategy. That is, the search unit 132 may be caused to perform inference or logical thinking by a generation model. The generation model in this case may use the AI server 200a or may be provided in the terminal device 100. In addition, although it takes time to perform summarization and strategy planning by the generation model each time, it is possible to create a context in advance from the interaction history up to immediately before.

[0105] That is, the output control unit 134 stores a query and response data in the storage unit (the interaction history storage unit 122) as an interaction history with the user. In addition, the search unit 132 generates the summary of the interaction with the user from the interaction history stored in the storage unit, and generates the context based on the preset purpose and the generated summary instead of the search of the external data. As a result, the information processing system 1 can continue the interaction without forgetting the flow of the conversation so far. Furthermore, the information processing system 1 can continue the interaction so as to guide the user.

[0106] <Use case UC3> Fig. 12 is a diagram illustrating an example of a flow of data in a use case UC3. Fig. 13 is a diagram illustrating an example of an interaction in the use case UC3. The use case UC3 illustrated in Figs. 12 and 13 is an example of collecting related information from a conversation between users.

[0107] The terminal device 100 starts acquisition of an image and a voice from the input device 111. For example, the data acquisition unit 131 starts acquisition of the voice of the user from the microphone 104. That is, input of input data from the input device 111 to the data acquisition unit 131 is started.

[0108] The data acquisition unit 131 performs voice recognition on the voice of the user and outputs a recognized text to the search unit 132. The search unit 132 transmits the input text to the general server 200. It is noted that it is assumed that the search using the general server 200 by the search unit 132 is executed at a first timing.

[0109] Upon receiving the text or the like from the terminal device 100, the general server 200 executes a search using a web search engine or the like. The general server 200 transmits a search result corresponding to the received text or the like to the terminal device 100.

[0110] Upon receiving the search result from the general server 200, the search unit 132 of the terminal device 100 extracts a context from the received search result and stores the context in the context storage unit 121. For example, when the title of a movie is included in the conversation between the users acquired by the microphone 104, the search unit 132 receives related information on the movie from the general server 200 as the search result. The search unit 132 extracts movie presentation information as a BG context from the related information, and stores the BG context in the context storage unit 121 as the related information on the conversation.

[0111] In a state in which the BG context is stored in the context storage unit 121, the query acquisition unit 133 of the terminal device 100 acquires, for example, the voice of the user from the microphone 104. The query acquisition unit 133 performs voice recognition on the voice of the user and acquires a recognized text as a query. For example, the query acquisition unit 133 acquires a query such as "Where is a nearby movie theater?" as an input text by voice recognition. The query acquisition unit 133 outputs the acquired query to the output control unit 134.

[0112] The output control unit 134 transmits a prompt including the query and the context to the AI server 300, and receives response data (generation statement). The output control unit 134 outputs the received response data (generation statement) via the output device 112.

[0113] In an interaction example C4 illustrated in Fig. 13, it is assumed that users A and B are excited in conversation about a movie that is currently popular, such as "△△△ seems to be interesting". At this time, the AI agent uses the general server 200 to search for, for example, movie presentation information of △△△ using the recognized text from the conversation between the users, and stores the information as the BG context. Here, when the user A says "Hey, where is a nearby movie theater?", the AI agent responds with "From here, 〇× cinema is close. {△△△ seems to be shown at 13:30 next time}" based on the query and the latest BG context. Here, a portion surrounded by curly brackets is a response based on related information of a conversation collected during the conversation between the users.

[0114] Furthermore, in an interaction example C5 illustrated in Fig. 13, in the automobile traveling on the highway, when a child in the vehicle says "I'm hungry", the AI agent searches for parking area (PA) information on the route using the general server 200, and stores the information as the BG context. Here, when a driver says "How far is the next PA?", the AI agent responds with "{Next one is △〇 PA, but the restaurant is crowded, so the one after that is good}" based on the query and the latest BG context. Here, a portion surrounded by curly brackets is a response based on related information of a conversation collected during the conversation between the child and the driver. As described above, in the use case UC3, in a case where the users have a conversation with each other before talking to the AI agent, information is collected in advance from the conversation, so that the AI agent can quickly answer when spoken to.

[0115] <Use case UC4> Fig. 14 is a diagram illustrating an example of a flow of data in a use case UC4. Fig. 15 is a diagram illustrating an example of an interaction in the use case UC4. The use case UC4 illustrated in Figs. 14 and 15 is an example of narrowing down a search target in the search unit 132 by what the user is looking at, pointing, or the like.

[0116] The terminal device 100 starts acquisition of an image and a voice from the input device 111. For example, the data acquisition unit 131 starts acquisition of an image from the camera 103 and starts acquisition of a voice of the user from the microphone 104. Here, the camera 103 includes, for example, an in-camera (in-vehicle camera) that captures an image of the inside of the vehicle and an out-camera (vehicle-mounted camera) that captures an image of the outside of the vehicle. That is, input of input data from the input device 111 to the data acquisition unit 131 is started.

[0117] The data acquisition unit 131 outputs the acquired image to the search unit 132. The search unit 132 transmits the input image to the general server 200. It is noted that the search unit 132 may convert the input image into a sentence and may transmit the sentence to the general server 200. In addition, it is assumed that the search using the general server 200 by the search unit 132 is executed at a first timing.

[0118] Upon receiving an image, a sentence, and the like from the terminal device 100, the general server 200 executes a search using an image search engine, a web search engine, and the like. The general server 200 transmits a search result corresponding to the received image, sentence, and the like to the terminal device 100.

[0119] Upon receiving the search result from the general server 200, the search unit 132 of the terminal device 100 extracts a context from the received search result and stores the context in the context storage unit 121. For example, when symbolic landmarks are included on the front left and right of the out-camera image, the search unit 132 recognizes the front left and right landmarks by processing asynchronously to the AI agent. When recognizing that the user specifies a landmark on the left front side by gesture recognition with respect to the in-camera image, the search unit 132 receives information on the landmark on the left front side from the general server 200 as a search result. The search unit 132 extracts the name and the like as a BG context from the information and stores the BG context in the context storage unit 121. It is noted that the search unit 132 may recognize what the user is looking at by line-of-sight recognition with respect to the in-camera image.

[0120] In a state in which the BG context is stored in the context storage unit 121, the query acquisition unit 133 acquires, for example, the voice of the user from the microphone 104. The query acquisition unit 133 performs voice recognition on the voice of the user and acquires a recognized text as a query. For example, the query acquisition unit 133 acquires a query such as "What is that building?" as an input text by voice recognition. The query acquisition unit 133 outputs the acquired query to the output control unit 134.

[0121] The output control unit 134 transmits a prompt including the query and the context to the AI server 300, and receives response data (generation statement). The output control unit 134 outputs the received response data (generation statement) via the output device 112.

[0122] In an interaction example C6 illustrated in Fig. 15, it is assumed that what is captured by the out-camera image while the vehicle is traveling is recognized. The AI agent recognizes that a symbolic landmark appears on the front left and right by asynchronous processing. At this time, when the user {points to the left and front side with the finger} while saying "Um~", the AI agent executes an asynchronous search for a landmark on the left front side by gesture recognition. Here, a portion surrounded by curly brackets indicates gesture recognition. In the asynchronous search here, the AI agent performs a search using the out-camera image and using the general server 200, and stores the search result as the BG context. Subsequently, when the user says "What is that building?", the AI agent responds with "That is the Yokohama 〇△ Hotel" based on the query and the latest BG context.

[0123] Furthermore, an interaction example C7 illustrated in Fig. 15 is an example in which narrowing is performed by a conversation between the users in a case where what is captured as an out-camera image is recognized during traveling of the vehicle. The AI agent recognizes that a symbolic landmark appears on the front left and right by asynchronous processing. At this time, when the user A says "{What is the oddly shaped building on the left}", the AI agent executes the asynchronous search for the landmark on the left front side. Here, a portion surrounded by curly brackets indicates the start of the conversation between the users. In the asynchronous search here, the AI agent performs a search using the general server 200 using the out-camera image and the text subjected to voice recognition, and stores the search result as the BG context. In response to the inquiry of the user A, the user B says "I don't know. □〇 what is that?". It is noted that □〇 is a nominal name of the AI agent. The AI agent responds with "That is the Yokohama 〇△ Hotel" based on the query and the latest BG context. As described above, in the use case UC4, by combining the multi-modal sub-information, it is possible to efficiently perform the asynchronous search which tends to be costly. In other words, the input data may be voice data related to a conversation between a plurality of users, or may be voice data related to a conversation between the user and the AI agent that can have a conversation with the user using the response data.

[0124] <Use case UC5> Fig. 16 is a diagram illustrating an example of an interaction in a use case UC5. The use case UC5 illustrated in Fig. 16 is an example of improving a real-time search by looking at the reaction of the user in the second embodiment. It is noted that, in the use cases UC5 and UC6, a flow of data will be omitted, and a description will be given using an interaction example.

[0125] An interaction example C8 illustrated in Fig. 16 is a case in which certainty of a response of the AI agent is determined while sensing reaction such as expression, nodding, and negative utterance of the user, and if the user is not satisfied, information is additionally provided with high accuracy. When the user says "How big is that?", the AI agent responds with "I will check the height of the 〇〇 tower". Here, when the user tilts his / her head while frowning, the AI agent determines that the response content is not what the user desires. This determination is represented by "..." in the interaction example C8. Here, the AI agent determines accuracy of the response, and performs another response based on a search result in a case where the accuracy is equal to or less than a threshold value. That is, the AI agent responds again with "{The height of the hotel behind you}" based on the determination. Here, a portion surrounded by curly brackets indicates a response reflecting (feeding back) the reaction of the user. Subsequently, when the user makes a positive utterance "Yes", the AI agent responds with "The height of the hotel is..." to provide additional information. As described above, in the interaction example C8, the reaction of the user can be elicited by sequential responses using an intermediate result, and the search accuracy can be improved according to the reaction.

[0126] Furthermore, an interaction example C9 illustrated in Fig. 16 is a case in which the user is asked a confirmation question before starting heavy processing such as a search, and the heavy processing is advanced as far as possible on the background side. When the user says "How big is that?", the AI agent starts investigation in the background while responding with "Height of the 〇〇 tower?". When the user says "Yes, {I also want to know the year of construction}", the AI agent starts to answer with the investigated information that "The height of the 〇〇 tower is... ", and performs investigation again in the background with "the year of construction". That is, when the response of the user is a predictable response such as "Yes", the AI agent quickly replies with information collected in the background. In addition, even in a case where the response is slightly different, heavy processing such as a search is executed again with the adjusted contents in the background while starting the response. Subsequently, the AI agent responds with "Further, the construction year of the tower is..." with information investigated again in the background As described above, in the interaction example C9, it is possible to speed up the speed up to the start of the response in the user experience by confirming an inquiry or the like for gaining time or performing heavy processing such as a search while sequential responses are interposed.

[0127] That is, the output control unit 134a determines accuracy of response data based on the reaction of the user based on the data, and one or more pieces of information of the query, and outputs other response data based on a search result as the response data in a case where the accuracy is equal to or less than a threshold value. As a result, the information processing system 1 can improve the search accuracy according to the reaction of the user.

[0128] Furthermore, the real-time search unit 135 sequentially searches for the external data based on the context and the query while outputting the response data based on the query. Furthermore, in a case where the response of the user is a response assumed based on the search result of the real-time search unit 135, the output control unit 134a outputs response data based on the search result of the real-time search unit 135 as response data. As a result, the information processing system 1 can speed up the start of the response in the user experience.

[0129] <Use case UC6> Fig. 17 is a diagram illustrating an example of an interaction in a use case UC6. The use case UC6 illustrated in Fig. 17 is an example of a variation of sequential responses in the second embodiment.

[0130] An interaction example C10 illustrated in Fig. 17 is a case in which an already uttered content is corrected with information that comes later by a search in the subsequent stage. In the interaction example C10, as in the second embodiment, when the user says "How big do you think that is?", the AI agent first responds with a rough search result that "The height of the 〇〇 tower is 〇□△". Thereafter, when obtaining a more accurate search result, the AI agent responds with "That was wrong, to be exact, that was 〇□×. Sorry".

[0131] An interaction example C11 illustrated in Fig. 17 is a case in which sequential responses are firstly started using the first search result at the time of continuous search. When the user says "What is the height of the 〇〇 tower?", the AI agent searches for "height of the 〇〇 tower", and responds with "The height of the 〇〇 tower is 〇□△" using the search result. When the user says "When was it built?", the AI agent responds with "〇〇 tower was built in □× year" based on the current search result of "height of the 〇〇 tower". In addition, the AI agent performs a search with "When the 〇〇 tower was built" in the background. If there is additional information as a result of the search, the AI agent additionally responds. As described above, in the use case UC6, a correction response can be made for an error in the preceding information that can be a problem of sequential response. In addition, in the use case UC6, since continuous searches are often associated with each other, a delay in response can be further reduced by using the first (previous) search result. Furthermore, in the use case UC6, in a case where the first (previous) search result is insufficient, it is possible to respond based on a more appropriate search result by performing a search again in the background while responding.

[0132] That is, the output control unit 134a outputs, as response data, response data for correcting the previous search result using the subsequent search result among the search results of the real-time search unit 135. As a result, the information processing system 1 can make a correction response for the error in the preceding information.

[0133] In addition, the output control unit 134a outputs response data based on the previous search result of the real-time search unit 135 as response data. As a result, the information processing system 1 can further reduce the delay of the response.

[0134] <<9. Regarding Generation model>> It is noted that each of the generation models used in each of the above-described processing is not limited to the example described in each portion, and any internal structure can be adopted as long as desired information can be output in response to an input. As for the input, output, and internal structure of the generation model, any combination can be adopted as long as desired information can be output.

[0135] The input of the generation model may be a text, an image, a voice, or the like, or a combination thereof. Furthermore, the output of the generation model may be a text, an image, a voice, or the like. It is noted that the above-described input and output are merely examples, and the input and output of the above-described generation model may be any input and output.

[0136] In addition, any structure can be adopted as the internal structure of the generation model according to a combination of input and output. That is, the internal structure of the generation model may be any structure as long as desired output can be performed for the input.

[0137] For example, the generation model may have a structure related to a transformer. For example, the generation model has a structure related to a transformer, and may perform processing in consideration of context such as context within data, such as text and time series data. For example, the generation model may have a self-attention mechanism. For example, the generation model may have any attention mechanism such as single-head attention and multi-head attention. It is noted that the generation model may not include an attention mechanism.

[0138] The generation model may have a mechanism for extracting features from the input. For example, the generation model may have an encoder. The generation model may have a mechanism for generating information based on the extracted features. For example, the generation model may have a decoder.

[0139] The generation model may have a structure related to a convolutional neural network (CNN). For example, the generation model may have a structure related to the CNN in a case of performing processing for an image. For example, the generation model may have at least one of a convolution layer, a pooling layer, a fully connected layer, and the like.

[0140] It is noted that the above-described internal structure is merely an example, and the above-described generation model may have any internal structure. For example, the generation model may have a skip connection. In addition, the generation model may have a structure related to a diffusion model.

[0141] Furthermore, the above-described generation model may be generated (learned) by any learning processing. The generation model may be a machine learning model learned using any machine learning method. For example, the generation model may be a model generated by fine-tuning based on a so-called foundation model so as to apply the foundation model to a specific task (for example, scenario data generation, code generation, and the like). For example, the generation model such as the LLM described above may be a model generated by fine-tuning the foundation model so as to be applied to a specific task.

[0142] The foundation model herein is a model learned so as to be applicable to various tasks, for example, so as to be able to execute various tasks. For example, the foundation model is a neural network pre-trained with a large number of unlabeled data sets. It is noted that the foundation model may have any structure such as a Transformer-based architecture. For example, the foundation model is generated by self-supervised learning using data without a correct answer label. As noted above, the foundation model is fine-tuned so as to be adaptable to a wide range of downstream tasks.

[0143] For example, in a case of being applied to a task of an interaction, the foundation model is fine-tuned so as to be adaptable to the task of the interaction, and the generation model (the generation model of the AI server 300 or the like) applied to the task of the interaction is generated.

[0144] The above-described generation model is learned by any learning processing according to the input, output, and internal structure of the generation model. For example, the generation model may be learned by an unsupervised learning method such as a generative adversarial network (GAN). In addition, the generation model may be learned in a distributed state without aggregating data, such as federated learning. In this case, in a device (server or the like) of each information processing service, a local model collected by the service may be generated, and a server (an aggregation server) that aggregates information (parameter and the like) of the local model generated by the device (server or the like) of each information processing service may generate a global model using the information of the local model. In this case, the information processing system 1 may receive the global model generated by the aggregation server from the aggregation server, and may use the received global model for processing as the generation model.

[0145] As described above, the above-described generation model may be generated (learned) by any computer. That is, the learning processing of generating the generation model may be performed by any device (computer or the like) of the information processing system 1 or may be performed by a device outside the information processing system 1. For example, in a case where a device outside the information processing system 1 generates at least one of the above-described generation models, the information processing system 1 acquires the generation model from the device outside the information processing system 1 and performs processing using the acquired generation model.

[0146] <<10. Modification of embodiment>> The processing according to each embodiment described above may be performed in various different modes other than each embodiment described above.

[0147] In each of the above embodiments, in the generation model of the AI server 300, a generation statement based on a query from a user is generated, and the terminal device 100 outputs the generated generation statement as response data, but the present invention is not limited thereto. For example, instead of the AI server 300, the terminal device 100 may include a generation model (small language model (SLM)) and generate a generation statement with the generation model.

[0148] In addition, information including processing procedure, specific name, and various types of data and parameters illustrated in the document and the drawings can be freely and selectively changed unless otherwise specified. For example, the various types of information illustrated in each drawing are not limited to the illustrated information.

[0149] In addition, each component of each device illustrated in the drawings is functionally conceptual, and is not necessarily physically configured as illustrated in the drawings. That is, a specific form of distribution and integration of each device is not limited to the illustrated form, and all or a part thereof can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, and the like. For example, the search unit 132 and the context storage unit 121 may be provided in the general server 200, and each function may be distributed. In addition, other functional units and storage units may be distributed to the general server 200 and the AI server 300. That is, the terminal device 100 may have a functional unit that serves as an interface with the user.

[0150] That is, in each of the above embodiments, the data acquisition unit 131, the search unit 132, the query acquisition unit 133, the output control unit 134 and 134a, and the real-time search unit 135 are provided in the terminal device 100, but the present invention is not limited thereto. For example, the data acquisition unit 131, the search unit 132, the query acquisition unit 133, the output control unit 134 and 134a, and the real-time search unit 135 may be arranged on a cloud server. In this case, the terminal device 100 may have a functional unit that serves as an interface with the user. In addition, some functions of the data acquisition unit 131, the search unit 132, the query acquisition unit 133, the output control unit 134 and 134a, and the real-time search unit 135 may be arranged on a cloud server.

[0151] In addition, the above-described embodiments and modifications can be appropriately combined within a range that does not contradict processing contents.

[0152] <<11. Hardware configuration>> The information processing apparatus (the terminal device 100 or the like) of the information processing system 1 according to each embodiment described above is implemented by, for example, a computer 1000 having a configuration as illustrated in Fig. 18. Hereinafter, the terminal device 100 will be described as an example. Fig. 18 is a hardware configuration diagram illustrating an example of a computer that implements the functions of the information processing apparatus. The computer 1000 includes a processing circuit 1100, a RAM 1200, a ROM 1300, a secondary storage device 1400, a communication interface 1500, an input / output interface 1600, a display unit 1700, a camera unit 1800, a microphone 1900, and a speaker 2000. The respective units of the computer 1000 are connected to each other by a bus 1050.

[0153] The processing circuit 1100 operates based on a program stored in the ROM 1300 or the secondary storage device 1400, and controls each unit. For example, the processing circuit 1100 loads the program stored in the ROM 1300 or the secondary storage device 1400 in the RAM 1200, and executes processing corresponding to various programs.

[0154] The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the processing circuit 1100 when the computer 1000 is started, a program dependent on hardware of the computer 1000, and the like.

[0155] The secondary storage device 1400 is a computer-readable recording medium that non-transiently records a program executed by the processing circuit 1100, data used by the program, and the like. Specifically, the secondary storage device 1400 is a recording medium that records a program of each processing of the terminal device 100 according to each embodiment which is an example of program data 1450.

[0156] The communication interface 1500 is an interface configured to connect the computer 1000 to an external network 1550. The communication interface 1500 corresponds to the communication unit 110 included in the terminal device 100. For example, the processing circuit 1100 receives data from another device or transmits data generated by the processing circuit 1100 to another device via the communication interface 1500.

[0157] The input / output interface 1600 is an interface configured to connect an input / output device 1650 to the computer 1000. For example, the processing circuit 1100 receives data from an input device such as a microphone 1900 or a touch panel via the input / output interface 1600. In addition, the processing circuit 1100 transmits data to an output device such as the display unit 1700 and the speaker 2000 via the input / output interface 1600. Furthermore, the input / output interface 1600 may function as a media interface configured to read a program or the like recorded in a predetermined recording medium (medium). The medium is, for example, an optical recording medium such as a digital versatile disc (DVD) or a phase change rewritable disk (PD), a magneto-optical recording medium such as a magneto-optical disk (MO), a tape medium, a magnetic recording medium, a semiconductor memory, or the like.

[0158] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electro luminescence (EL) display. Furthermore, the display unit 1700 may be a touch panel type display device or a video projection device.

[0159] The camera unit 1800 is an interface for the computer 1000 to capture an image. The microphone 1900 is an interface for the computer 1000 to capture a voice. The speaker 2000 is an interface for outputting a voice processed by the computer 1000. The respective units of the computer 1000 are connected to each other by the bus 1050. Each interface is not necessarily provided inside the computer 1000, and may be provided outside the computer 1000 through a network or the like. Furthermore, the respective units constituting the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100 but by a circuit dedicated to display processing included in the display unit 1700.

[0160] For example, in a case where the computer 1000 functions as the terminal device 100 according to each embodiment, the processing circuit 1100 of the computer 1000 functions as the control unit 130 by executing the program loaded in the RAM 1200. In addition, the secondary storage device 1400 stores an information processing program according to the present disclosure and various data stored in the storage unit 120. It is noted that the processing circuit 1100 reads the program data 1450 from the secondary storage device 1400 and executes the program data, but as another example, these programs may be acquired from another device via the external network 1550. That is, the secondary storage device 1400 is not limited to the inside of the computer 1000, and may be disposed outside the computer 1000. It is noted that the processing circuit 1100 is an example of an integrated circuit, and any of the CPU, the MPU, the GPU, the APU, the ASIC, and the FPGA can be regarded as an integrated circuit.

[0161] It is noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

[0162] It is noted that the present disclosure can also have the following configurations. (1) 　An information processing system comprising: 　a data acquisition unit configured to acquire data of any modal related to a user; 　a search unit configured to search for external data based on the data; 　a query acquisition unit configured to acquire a query from the user; and 　an output control unit configured to output, based on the external data and the query, response data. (2) 　The information processing system according to (1), wherein a search performed by the search unit is executed at a first timing before acquisition of the query, and 　an output by the output control unit is executed at a second timing after the first timing. (3) 　The information processing system according to (2), wherein 　the search unit extracts a context from a search result of the external data, and 　the output control unit outputs, based on the extracted context and the query, the response data. (4) 　The information processing system according to (3), wherein the output control unit uses a prompt including the context as an input of a generation model, and outputs response data generated by the generation model as the response data. (5) 　The information processing system according to (3) or (4), wherein the search unit stores a plurality of the extracted contexts in a storage unit. (6) 　The information processing system according to any one of (3) to (5), comprising a real-time search unit configured to, based on the context and the query, sequentially search for the external data, 　wherein the output control unit outputs, as the response data, response data based on the query and response data based on a search result of the real-time search unit. (7) 　The information processing system according to (6), wherein the output control unit is configured to: 　determine, based on a reaction of the user based on the data, and based on one or more pieces of information of the query, accuracy of the response data, and 　output, as the response data, another response data based on the search result when the accuracy is equal to or less than a threshold value. (8) 　The information processing system according to (6) or (7), wherein 　the real-time search unit sequentially searches for, based on the context and the query, the external data while outputting the response data based on the query, and 　the output control unit outputs, as the response data, response data based on the search result of the real-time search unit when a response of the user is a response assumed based on the search result of the real-time search unit. (9) 　The information processing system according to any one of (6) to (8), wherein the output control unit outputs, as the response data, response data for correcting a previous search result by using a subsequent search result among the search results of the real-time search unit. (10) 　The information processing system according to any one of (6) to (9), wherein the output control unit outputs, as the response data, response data based on a previous search result of the real-time search unit. (11) 　The information processing system according to any one of (3) to (10), wherein 　the output control unit stores the query and the response data in a storage unit as an interaction history with the user, and 　the search unit further searches for the interaction history stored in the storage unit. (12) 　The information processing system according to (3), wherein 　the output control unit stores, as an interaction history with the user, the query and the response data in a storage unit, and 　the search unit generates a summary of an interaction with the user from the interaction history stored in the storage unit and generates, based on a preset purpose and the generated summary, the context instead of the search for the external data. (13) 　The information processing system according to any one of (1) to (12), wherein a modal of the data acquired by the data acquisition unit and a modal of the response data output by the output control unit are different types of modalities. (14) 　The information processing system according to any one of (1) to (13), in which the data is data including at least one of image data, voice data, and text data. (15) 　The information processing system according to any one of (1) to (14), wherein the data is voice data related to a conversation between a plurality of the users. (16) 　The information processing system according to any one of (1) to (14), wherein the data is voice data related to a conversation between the user and an agent capable of having a conversation with the user using the response data. (17) 　The information processing system according to (15) or (16), wherein the search unit starts a search with start of the conversation as a trigger. (18) 　The information processing system according to any one of (1) to (17), further comprising a detection unit configured to detect motion of the user, 　wherein the search unit executes a search when the motion of the user is detected. (19) 　The information processing system according to (18), wherein the detection unit detects one or more pieces of information of a line of sight and a gesture of the user. (20) 　An information processing method comprising: 　by a computer, 　acquiring data of any modal related to a user; 　searching for external data based on the data; 　acquiring a query from the user; and 　outputting, based on the external data and the query, response data. (21) 　An information processing program causing a computer to execute: 　acquiring data of any modal related to a user; 　searching for external data based on the data; 　acquiring a query from the user; and 　outputting, based on the external data and the query, response data. (22) 　An information processing system comprising: 　circuitry configured to: 　　acquire input data of a first modality related to a user; 　　based on the input data, asynchronously search an external data source to obtain search results, the asynchronous search being performed prior to receiving a query from the user; 　　receive the query from the user; and 　　generate response data based on the search results and the query. (23) 　The information processing system according to (22), wherein 　the asynchronous search is executed at a first timing before acquisition of the query, and 　an output of the response data is executed at a second timing after the first timing. (24) 　The information processing system according to (22), wherein the circuitry is configured to: 　use a prompt including the context information as an input to a generation model; and 　output response data generated by the generation model as the response data. (25) 　The information processing system according to (24), wherein the generation model comprises a large language model. (26) 　The information processing system according to (22), wherein the circuitry is configured to store a plurality of contexts extracted from a plurality of search results in the memory. (27) 　The information processing system according to (22), wherein the circuitry is configured to: 　based on the stored context information and the query, perform a real-time search of the external data source to obtain real-time search results; and 　output the response data based on the real-time search results. (28) 　The information processing system according to (27), wherein the circuitry is further configured to: 　sequentially obtain a plurality of intermediate search results from the real-time search; and 　sequentially output a plurality of response data portions, each response data portion being generated based on a respective intermediate search result of the plurality of intermediate search results. (29) 　The information processing system according to (28), wherein the circuitry is configured to: 　determine, based on a reaction of the user and the query, accuracy of the response data; and 　output, as the response data, different response data based on a subsequent search result when the accuracy is equal to or less than a threshold value. (30) 　The information processing system according to (28), wherein the circuitry is configured to: 　sequentially search for external data based on the context information and the query while outputting the response data based on the query; and 　output, as the response data, response data based on a search result of the real-time search when a response of the user is a response assumed based on the search result. (31) 　The information processing system according to (28), wherein the circuitry is configured to output, as one of the plurality of response data portions, correction data for correcting a previous response data portion based on a subsequent intermediate search result. (32) 　The information processing system according to (28), wherein the circuitry is configured to output response data based on a previous search result of the real-time search as an initial response data portion. (33) 　The information processing system according to (22), wherein the circuitry is configured to: 　store the query and the response data in an interaction history storage as an interaction history with the user; and 　search the interaction history storage based on subsequent input data. (34) 　The information processing system of claim (33), wherein the circuitry is configured to: 　generate a summary of interaction with the user from the interaction history stored in the interaction history storage; and 　generate the context information based on a preset purpose and the generated summary. (35) 　The information processing system according to (22), wherein the first modality of the input data acquired by the circuitry and a second modality of the response data output by the circuitry are different types of modalities. (36) 　The information processing system according to (22), wherein the input data comprises voice data related to a conversation between a plurality of users. (37) 　The information processing system according to (22), wherein the input data comprises voice data related to a conversation between the user and an agent configured to converse with the user using the response data. (38) 　The information processing system according to (26), wherein the circuitry is configured to initiate the asynchronous search in response to detecting a start of the conversation. (39) 　The information processing system according to (22), wherein the circuitry is configured to: 　detect motion of the user from the input data; and 　execute the asynchronous search when the motion of the user is detected. (40) 　The information processing system according to (39), wherein the motion of the user comprises at least one of a gaze direction and a gesture of the user. (41) 　The information processing system according to (22), wherein 　the input data comprises image data captured by a camera, and 　the circuitry is configured to convert the image data into text data and search the external data source based on the text data. (42) 　The information processing system according to (41), wherein the image data comprises at least one of video visible to the user in a metaverse environment, an image captured by a vehicle-mounted camera, or an image captured by a camera of a storefront signage device. (43) 　The information processing system according to (22), wherein the circuitry is configured to execute the asynchronous search only when the user is detected within a detection range. (44) 　The information processing system according to (22), wherein the circuitry is configured to execute the asynchronous search only when a non-player character is visible within a display range of a screen. (45) 　The information processing system according to (22), wherein the circuitry is configured to execute the asynchronous search only when the user stands in front of a storefront signage device. (46) 　The information processing system according to (22), wherein the circuitry is configured to: 　extract context information from the search results and store the context information in memory; and 　generate the response data based on the stored context information and the query. (47) 　An information processing method comprising: 　acquiring input data of a first modality related to a user; 　based on the input data, asynchronously searching an external data source to obtain search results, the asynchronous search being performed prior to receiving a query from the user; 　receiving the query from the user; and 　generating response data based on the search results and the query. (48) 　One or more non-transitory computer-readable media including computer-program instructions, which when executed by circuitry, cause the circuitry to: 　acquire input data of a first modality related to a user; 　based on the input data, asynchronously search an external data source to obtain search results, the asynchronous search being performed prior to receiving a query from the user; 　receive the query from the user; and 　generate response data based on the search results and the query (49) 　An information processing system comprising: 　circuitry configured to: 　　acquire voice data from a user; 　　search an interaction history storage based on Cthe voice data to obtain prior interaction data, the interaction history storage storing interaction histories between the user and an artificial intelligence agent; 　　generate context information based on the prior interaction data and store the context information in memory; 　　receive a query from the user; and 　　generate response data based on the stored context information and the query. (50) 　An information processing system comprising: 　circuitry configured to: 　　acquire voice data related to a conversation between a plurality of users; 　　based on the voice data, search an external data source to obtain search results related to a subject of the conversation; 　　extract context information from the search results and store the context information in memory; 　　receive a query from one of the plurality of users; and 　　generate response data based on the stored context information and the query. (51) 　An information processing system comprising: 　circuitry configured to: 　　acquire input data of a first modality from a sensor associated with a user; 　　based on the input data, asynchronously search an external data source to obtain search results, the asynchronous search being performed prior to receiving a query from the user; 　　receive the query from the user; and 　　generate response data based on the search results and the query.

[0163] 1 INFORMATION PROCESSING SYSTEM 100, 100a TERMINAL DEVICE 101 DISPLAY UNIT 102 OPERATION UNIT 103 CAMERA 104 MICROPHONE 105 SPEAKER 110 COMMUNICATION UNIT 120 STORAGE UNIT 121 CONTEXT STORAGE UNIT 122 INTERACTION HISTORY STORAGE UNIT 123 SUMMARY STORAGE UNIT 130, 130a CONTROL UNIT 131 DATA ACQUISITION UNIT 132 SEARCH UNIT 133 QUERY ACQUISITION UNIT 134, 134a OUTPUT CONTROL UNIT 135 REAL-TIME SEARCH UNIT 200 GENERAL SERVER 200a, 300 AI SERVER N NETWORK

Claims

1. An information processing system comprising: 　circuitry configured to: 　　acquire input data of a first modality related to a user; 　　based on the input data, asynchronously search an external data source to obtain search results, the asynchronous search being performed prior to receiving a query from the user; 　　receive the query from the user; and 　　generate response data based on the search results and the query.

2. The information processing system of claim 1, wherein the circuitry is configured to: 　use a prompt including the context information as an input to a generation model; and 　output response data generated by the generation model as the response data.

3. The information processing system of claim 1, wherein the circuitry is configured to store a plurality of contexts extracted from a plurality of search results in the memory.

4. The information processing system of claim 1, wherein the circuitry is configured to: 　based on the stored context information and the query, perform a real-time search of the external data source to obtain real-time search results; and 　output the response data based on the real-time search results.

5. The information processing system of claim 4, wherein the circuitry is further configured to: 　sequentially obtain a plurality of intermediate search results from the real-time search; and 　sequentially output a plurality of response data portions, each response data portion being generated based on a respective intermediate search result of the plurality of intermediate search results.

6. The information processing system of claim 5, wherein the circuitry is configured to: 　determine, based on a reaction of the user and the query, accuracy of the response data; and 　output, as the response data, different response data based on a subsequent search result when the accuracy is equal to or less than a threshold value.

7. The information processing system of claim 5, wherein the circuitry is configured to: 　sequentially search for external data based on the context information and the query while outputting the response data based on the query; and 　output, as the response data, response data based on a search result of the real-time search when a response of the user is a response assumed based on the search result.

8. The information processing system of claim 5, wherein the circuitry is configured to output, as one of the plurality of response data portions, correction data for correcting a previous response data portion based on a subsequent intermediate search result.

9. The information processing system of claim 5, wherein the circuitry is configured to output response data based on a previous search result of the real-time search as an initial response data portion.

10. The information processing system of claim 1, wherein the circuitry is configured to: 　store the query and the response data in an interaction history storage as an interaction history with the user; and 　search the interaction history storage based on subsequent input data.

11. The information processing system of claim 10, wherein the circuitry is configured to: 　generate a summary of interaction with the user from the interaction history stored in the interaction history storage; and 　generate the context information based on a preset purpose and the generated summary.

12. The information processing system of claim 1, wherein the first modality of the input data acquired by the circuitry and a second modality of the response data output by the circuitry are different types of modalities.

13. The information processing system of claim 1, wherein the input data comprises voice data related to a conversation between a plurality of users.

14. The information processing system of claim 1, wherein the input data comprises voice data related to a conversation between the user and an agent configured to converse with the user using the response data.

15. The information processing system of claim 13, wherein the circuitry is configured to initiate the asynchronous search in response to detecting a start of the conversation.

16. The information processing system of claim 1, wherein the circuitry is configured to: 　detect motion of the user from the input data; and 　execute the asynchronous search when the motion of the user is detected.

17. The information processing system of claim 16, wherein the motion of the user comprises at least one of a gaze direction and a gesture of the user.

18. The information processing system of claim 1, wherein the circuitry is configured to: 　extract context information from the search results and store the context information in memory; and 　generate the response data based on the stored context information and the query.

19. An information processing method comprising: 　acquiring input data of a first modality related to a user; 　based on the input data, asynchronously searching an external data source to obtain search results, the asynchronous search being performed prior to receiving a query from the user; 　receiving the query from the user; and 　generating response data based on the search results and the query.

20. One or more non-transitory computer-readable media including computer-program instructions, which when executed by circuitry, cause the circuitry to: 　acquire input data of a first modality related to a user; 　based on the input data, asynchronously search an external data source to obtain search results, the asynchronous search being performed prior to receiving a query from the user; 　receive the query from the user; and 　generate response data based on the search results and the query

Images