Device and method

The apparatus and method improve AI response determination by acquiring multiple responses and selecting the best answer strategy, addressing the inconsistency in existing generative AI systems by enhancing response reliability and efficiency.

WO2026234801A1PCT designated stage Publication Date: 2026-11-12NTT DOCOMO INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2025/017045
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-05-09
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing systems using generative AI fail to determine an effective response method for input information, particularly questions, leading to inconsistent and inefficient output.

Method used

An apparatus and method that acquire multiple responses to the same input information from generative AI and determine a response method based on these responses, utilizing a decision unit to select the appropriate answer strategy, which may involve external information retrieval or model switching.

Benefits of technology

Enhances the consistency and accuracy of responses by determining the optimal response method, reducing the need for redundant external searches and improving the reliability of AI-generated answers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2025017045_12112026_PF_FP_ABST
    Figure JP2025017045_12112026_PF_FP_ABST
Patent Text Reader

Abstract

This device (1) is provided with: an acquisition unit (12) that acquires a plurality of responses obtained by inputting the same input information a plurality times into an LLM (40); and a determination unit (14) that determines a method for responding to the input information on the basis of the plurality of responses. The plurality of responses may comprise responses that are output on each occasion as a result of inputting the same input information into the LLM (40) a plurality of times. The plurality of responses may be output by the LLM (40) in which the randomness of output has been set to be higher than a normal setting. The determination by the determination unit (14) may be based on input, to the LLM, of a prompt that includes an instruction to determine the method for responding to the input information on the basis of the plurality of responses.
Need to check novelty before this filing date? Find Prior Art

Description

Apparatus and Method

[0001] One aspect of the present disclosure relates to an apparatus and a method for determining a response method to input information.

[0002] The following Patent Document 1 discloses a system in which generative AI generates responses to questions.

[0003] Japanese Unexamined Patent Application Publication No. 2025-43741

[0004] However, the above system cannot determine a response method for input information including a question. Accordingly, it is desired to determine a response method for input information.

[0005] An apparatus according to one aspect of the present disclosure includes an acquisition unit that acquires a plurality of responses obtained by inputting the same input information to generative AI a plurality of times, and a determination unit that determines a response method to the input information based on the plurality of responses.

[0006] In such an aspect, the response method to the input information is determined based on the plurality of responses obtained by inputting the same input information to the generative AI a plurality of times. That is, the response method to the input information can be determined.

[0007] According to one aspect of the present disclosure, the response method to input information can be determined.

[0008] This figure shows an example of the system configuration of a system including the device according to the embodiment. This figure shows an example of the relationship between the device according to the embodiment and the generated AI. This figure shows a modified version of the relationship in Figure 2. This figure shows an example of the functional configuration of the device according to the embodiment. This flowchart shows an example of the processing performed by the device according to the embodiment. This flowchart shows an example of the specific processing performed by the device according to the embodiment. This is a sequence diagram (1) showing an example of the specific processing without sampling responses performed by the device according to the embodiment. This figure shows an example of knowledge judgment prompt 1. This figure shows an example of output based on knowledge judgment prompt 1. This figure shows an example of the first search result of external information. This is a sequence diagram (2) showing an example of the specific processing without sampling responses performed by the device according to the embodiment. This is a diagram (1) showing an example of knowledge judgment prompt 2. This is a diagram (2) showing an example of knowledge judgment prompt 2. This is a diagram showing an example of the first output based on knowledge judgment prompt 2. This figure shows an example of the second search result of external information. This is a diagram showing an example of the second output based on knowledge judgment prompt 2. This is a sequence diagram (1) showing an example of the response processing performed by the device according to the embodiment. This figure shows an example of final response acquisition prompt 1. This figure shows an example of output based on final response acquisition prompt 1. This is a sequence diagram (2) showing an example of the response processing performed by the device according to the embodiment. This figure shows an example of the final response acquisition prompt 2. This figure shows an example of the output based on the final response acquisition prompt 2. This is a sequence diagram (1) showing an example of a specific process with sampling responses executed by the apparatus according to the embodiment. This figure shows an example of the sampling response acquisition prompt 1. This figure shows an example of the output based on the sampling response acquisition prompt 1. This is a sequence diagram (2) showing an example of a specific process with sampling responses executed by the apparatus according to the embodiment. This is a figure (1) showing an example of the knowledge judgment prompt 3. This is a figure (2) showing an example of the knowledge judgment prompt 3. This figure shows an example of the output based on the knowledge judgment prompt 3. This is a sequence diagram (3) showing an example of a specific process with sampling responses executed by the apparatus according to the embodiment. This figure shows an example of the sampling response acquisition prompt 2.This figure shows an example of output based on the sampling response acquisition prompt 2. This is a sequence diagram (4) showing an example of specific processing with sampling responses performed by the apparatus according to the embodiment. This is a diagram (1) showing an example of the knowledge judgment prompt 4. This is a diagram (2) showing an example of the knowledge judgment prompt 4. This figure shows an example of output based on the knowledge judgment prompt 4. This is a diagram (1) showing an example of the hallucination judgment prompt. This is a diagram (2) showing an example of the hallucination judgment prompt. This is a diagram (3) showing an example of the hallucination judgment prompt. This figure shows an example of the training data. This figure shows an example of the hardware configuration of the computer used in the apparatus according to the embodiment.

[0009] The embodiments of this disclosure will be described in detail below with reference to the drawings. In the description of the drawings, the same elements will be denoted by the same reference numerals, and redundant descriptions will be omitted. Furthermore, the embodiments of this disclosure described below are specific examples of the present invention and are not limited to these embodiments unless otherwise stated to limit the present invention.

[0010] Figure 1 is a diagram showing an example of the system configuration of a system 3 including a device 1 according to an embodiment. As shown in Figure 1, system 3 is composed of a device 1 and a terminal 2. The device 1 and the terminal 2 are connected to each other by a network such as a mobile communication network, and can send and receive information to each other. In system 3, there may be multiple terminals 2 (each used by a different user), in which case the device 1 and each terminal 2 are similarly connected to each other and can send and receive information to each other.

[0011] Device 1 is a computer device (server) that determines how to answer a question (input information). In this embodiment, "question" may be replaced with "input information". The input information includes information about any input (user input, or input to Device 1 or the generating AI), such as questions, commands, or queries. In this embodiment, the explanation will mainly use "question," but it is not limited to this, and as mentioned above, it may be replaced with "input information". Device 1 may output an answer to the question based on the determined answer method. Device 1 may also have the function of RAG (Retrieval-Augmented Generation). Device 1 may perform various processes using the generating AI, LLM (Large Language Model), and prompts.

[0012] Generative AI is a technology that, in response to a prompt containing input information, generates content according to one or a combination of the instructions, context, questions, and output formats indicated by the prompt, and returns that content as response information. The prompt can also contain input information, in which case the generative AI generates response information based on the input information. The generative AI may be a conversational AI that includes, for example, a large-scale language model (LLM) and a user interface (UI) for interaction with the user, enabling text chat or voice chat with the user. Examples of such generative AI include tsuzumi®, ChatGPT, GPT®-3.5, GPT-4V, and PaLM2.

[0013] The generated AI may be stored in another device connected to device 1 via a network, and configured to allow for information exchange. Figure 2 shows an example of the relationship between device 1 and the generated AI. In the relationship shown in Figure 2, the generated AI, LLM 40, is stored in LLM server 4, which is connected to device 1 via a network. When needed, device 1 has LLM 40 perform processing via the network, receives the execution results, and uses them.

[0014] The generated AI may be stored within device 1. Figure 3 shows a modified example of the relationship shown in Figure 2. In the modified example shown in Figure 3, the generated AI, LLM40, is stored within device 1. When necessary, device 1 executes processing on the LLM40 stored within device 1 and uses the execution result.

[0015] A prompt is information indicating instructions or questions to be entered by a user in an interactive system such as a conversation with a generating AI or a command-line interface (CLI). For example, a prompt may express, in text form, the command to be executed by the generating AI, the task to be performed by the generating AI, the background / context that the generating AI should consider (e.g., role, conditions), the question that the generating AI should answer, and the output format of the response information from the generating AI. In addition, information about the input that will be the target of the command / task to be executed by the generating AI may be added to the prompt. Such input information includes data files with filenames containing predetermined extensions, such as text data, image data, application-related data, audio data, video data, and still image data. Application-related data refers to data such as document data, tabular data, and graph data that can be processed by a default application program.

[0016] Details of device 1 will be described later.

[0017] Terminal 2 is a mobile communication terminal or a computer device such as a laptop computer that performs mobile communication. In this embodiment, terminal 2 is assumed to be a smartphone, but is not limited to this. Terminal 2 is carried by the user of terminal 2. Terminal 2 may be equipped with other sensors or functions that are typical of a smartphone. Terminal 2 may transmit the user's question to device 1, obtain the answer output by device 1 as a response, and display the obtained answer to the user.

[0018] Figure 4 is a diagram showing an example of the functional configuration of the apparatus 1 according to the embodiment. As shown in Figure 4, the apparatus 1 is composed of a storage unit 10, a setting unit 11, an acquisition unit 12 (acquisition unit), a generation unit 13 (acquisition unit, determination unit), a determination unit 14 (determination unit, answer unit), and an answer unit 15 (answer unit).

[0019] Each functional block of Device 1 is intended to function within Device 1, but is not limited to this. For example, some of the functional blocks of Device 1 may function within a computer device separate from Device 1, connected to Device 1 via a network, while appropriately sending and receiving information with Device 1. Furthermore, some functional blocks of Device 1 may be omitted, multiple functional blocks may be integrated into a single functional block, or a single functional block may be broken down into multiple functional blocks.

[0020] The following describes the functions of the device 1 shown in Figure 4.

[0021] The storage unit 10 stores arbitrary information used in calculations in the device 1, as well as the results of calculations in the device 1. The information stored in the storage unit 10 may be referenced as appropriate by each function of the device 1. The information stored in the storage unit 10 may be pre-set information, information purchased in advance, or information acquired at any time by the acquisition unit 12 described later. The storage unit 10 may also store the LLM 40 described above.

[0022] The setting unit 11 performs arbitrary settings related to the device 1 or LLM 40.

[0023] The setting unit 11 may set the randomness of the LLM40's output. The setting unit 11 may set the randomness of the LLM40's output lower than the normal setting (default setting, general setting) or higher than the normal setting. The lower the randomness setting of the LLM40 is set, the more consistent and predictable the LLM40 will produce (similar output). On the other hand, the higher the randomness setting of the LLM40 is set, the more random and unpredictable the LLM40 will produce (diverse output).

[0024] The setting unit 11 may also set Temperature, a parameter that controls the randomness of the output of the LLM 40. Temperature is represented by a real value between "0.0" and "1.0", for example, and the usual setting is "0.5". When Temperature is set to a value between "0.0" and "0.5", the LLM 40 produces a more consistent and predictable output, as described above. On the other hand, when Temperature is set to a value greater than "0.5" and less than or equal to "1.0", the LLM 40 produces a more random and unpredictable output, as described above.

[0025] The acquisition unit 12 acquires arbitrary information. The acquisition unit 12 may acquire arbitrary information used in calculations in the device 1 from other functional blocks of the device 1, including the storage unit 10, or it may acquire (receive) it from other devices via a network or the like. The acquisition unit 12 may acquire arbitrary information in advance prior to any calculations in the device 1, or it may acquire the information when it becomes necessary, in accordance with the requests of each functional block of the device 1. The acquisition unit 12 may store the acquired information in the storage unit 10, or it may output it to other functional blocks of the device 1.

[0026] The acquisition unit 12 acquires multiple answers obtained by inputting the same question multiple times to the LLM 40. More specifically, device 1 or another device may input the same question multiple times to the LLM 40, and the acquisition unit 12 may acquire the multiple answers output by the LLM 40 as responses. Alternatively, the acquisition unit 12 may input the same question multiple times to the LLM 40, and the acquisition unit 12 may acquire the multiple answers output by the LLM 40 as responses.

[0027] The question may be text or a prompt composed of text. Upon receiving a question, the LLM40 outputs an answer to that question based on the knowledge it stores internally. As described above, the randomness setting of the LLM40's output or the temperature setting of the LLM40 will determine whether the output is a consistent and predictable output or a random and unpredictable output.

[0028] Multiple answers may consist of the answers output each time the same question is input to the LLM 40 multiple times. That is, the acquisition unit 12 may acquire multiple answers consisting of the answers output each time the same question is input to the LLM 40 multiple times. More specifically, device 1 or another device may input the same question to the LLM 40 multiple times, and the acquisition unit 12 may acquire multiple answers (for those multiple inputs) consisting of the answers output by the LLM 40 as a response each time. Alternatively, the acquisition unit 12 may input the same question to the LLM 40 multiple times, and the acquisition unit 12 may acquire multiple answers (for those multiple inputs) consisting of the answers output by the LLM 40 as a response each time.

[0029] Multiple responses may be output by an LLM40 with a higher randomness setting than the normal setting. That is, the acquisition unit 12 may acquire multiple responses output by an LLM40 with a higher randomness setting than the normal setting (for example, a Temperature setting of "0.8") than the normal setting (for example, a Temperature setting of "0.5").

[0030] The acquisition unit 12 may acquire questions from the user (terminal 2) or other devices. The acquisition unit 12 may search for external information based on queries and acquire external information as search results. The acquisition unit 12 may perform a search based on questions and acquire search results. The acquisition unit 12 may acquire various prompts. The acquisition unit 12 may cause the LLM 40 to execute various prompts and acquire the execution results (queries, outputs, answers, sampled answers, etc.). The acquisition unit 12 may acquire answers or sampled answers. Details will be explained in the specific examples below.

[0031] The generation unit 13 generates various prompts. In this embodiment, "prompt generation" may be replaced with "prompt preparation" or "inputting some input data to pre-prepared prompts." The generation unit 13 may cause the LLM 40 to execute the various prompts and obtain the execution results (queries, outputs, answers, sampled answers, etc.). Details will be explained in the specific examples described later.

[0032] The generation unit 13 may store the generated prompts or acquired execution results in the storage unit 10, output them to other functional blocks of the device 1, or output (transmit) them to other devices via the network.

[0033] The decision unit 14 determines how to answer the question based on multiple responses. More specifically, the decision unit 14 may determine how to answer the question (the question entered into the LLM 40) based on multiple responses obtained by the acquisition unit 12 through multiple inputs of the same question into the LLM 40.

[0034] The determination of the answer method by the determination unit 14 may be based on the input of a prompt to the LLM (LLM 40 or an LLM other than LLM 40) that includes an instruction to determine the answer method to the question based on multiple answers. More specifically, the determination unit 14 may use the determined answer method, which is the output result of the LLM output based on the input of a prompt to the LLM that includes an instruction to determine the answer method to the question based on multiple answers, as the answer method determined by the determination unit 14.

[0035] The decision unit 14 may decide to use the LLM 40 as the method of answering. For example, the decision unit 14 may decide to use an LLM 40 that has been input with the same question in order to obtain multiple answers to answer that question.

[0036] The decision unit 14 may decide to use external information as the method of answering. For example, if the decision unit 14 determines that it cannot answer using only the knowledge within the LLM 40, it may decide to use information stored in other devices (external devices) (external information) to answer the question.

[0037] The decision unit 14 may determine a method of answering using external information and generate a query to acquire external information. For example, the decision unit 14 may determine that it cannot answer using only the knowledge within the LLM 40 and decide to use information stored in another device (external device) (external information) to answer the question, and generate a query to acquire such information. The generation of the query may be based on input to the LLM of a prompt that includes an instruction to generate a query to acquire external information.

[0038] The decision unit 14 may decide to use a different LLM (Low-Level Model) than LLM 40 as the method of answering. For example, if the decision unit 14 determines that the answer cannot be answered using only the knowledge contained in LLM 40, it may decide to use a different LLM (which may be included in system 3) to answer the question. In this embodiment, using a different LLM may also be described as changing the model or switching models.

[0039] The decision unit 14 may store the decision result in the storage unit 10, output it to another functional block of the device 1, or output (transmit) it to another device via a network.

[0040] The answer unit 15 outputs an answer to the question based on the answer method. More specifically, the answer unit 15 may output an answer to the question based on the answer method determined by the decision unit 14. Alternatively, the answer unit 15 may output an answer to the question (the question entered into the LLM 40) based on the answer method for the question (the question entered into the LLM 40) determined by the decision unit 14, which is based on multiple answers obtained by the acquisition unit 12 and obtained by multiple inputs of the same question into the LLM 40. The output (of the answer) of the answer unit 15 may also be based on the input of a prompt to the LLM that includes an instruction to output an answer to the question based on the answer method.

[0041] The answer unit 15 may generate an answer to the question based on the answer method and output the generated answer. The generation of the answer by the answer unit 15 may be based on the input of a prompt to the LLM that includes an instruction to generate an answer to the question based on the answer method.

[0042] The response unit 15 may cause the storage unit 10 to store the response (output the response to the storage unit 10), may output the response to another functional block of the device 1, or may output (transmit) the response to another device (including the terminal 2) via a network.

[0043] Next, an example of the determination process executed by the device 1 will be described with reference to FIG. 5. FIG. 5 is a flowchart showing an example of the process executed by the device 1.

[0044] First, the acquisition unit 12 acquires a plurality of responses obtained by inputting the same question to the LLM 40 a plurality of times (step S1, acquisition step). Next, the determination unit 14 determines a response method for the question (input to the LLM 40 in step S1) based on the plurality of responses acquired in step S1 (step S2, determination step).

[0045] Next, examples of specific determination processing and response processing executed by the device 1 will be described with reference to FIG. 6. FIG. 6 is a flowchart showing an example of specific processing executed by the device 1.

[0046] First, the LLM 40 inputs a prompt which is a question (step S10). More specifically, the acquisition unit 12 acquires the question transmitted from the user (the terminal 2 of the user), converts the acquired question into a prompt format, transmits the converted prompt to the LLM 40, and the LLM 40 acquires (inputs) the prompt.

[0047] Next, the prompt input in step S10 is transmitted to the acquisition unit 12, and the acquisition unit 12 acquires sampling responses based on the acquired prompt (step S11). More specifically, the acquisition unit 12 inputs the prompt input in step S10 to the LLM 40 a plurality of times to acquire a plurality of responses (sampling responses). Note that when inputting the prompt to the LLM 40, diverse outputs may be acquired by increasing the Temperature of the LLM 40.

[0048] Next, the determination unit 14 determines a response method based on the sampling responses acquired in step S11 (step S12). For example, the determination unit 14 may determine, based on the sampling responses acquired in step S11, whether to generate a query (for acquiring external information) (whether it is necessary to search for external information) or whether it is necessary to change the (LLM) model. When the determination unit 14 determines that neither the necessity of query generation (the necessity of external information search) nor the necessity of model selection exists, the process proceeds to the subsequent step S13. When the determination unit 14 determines that the necessity of query generation exists, it causes the LLM 40 to generate a query, acquires external information through an external search using the generated query, and proceeds to the subsequent step S13. When the determination unit 14 determines that the necessity of model selection exists, it selects a model (selects an LLM different from the LLM 40), and proceeds to the subsequent step S13.

[0049] Next, the response unit 15 outputs a response based on the response method determined in step S12 (step S13). More specifically, the response unit 15 outputs a response to the prompt (question) input in step S10 based on the response method determined in step S12. Note that when it is determined in step S12 that neither external information search nor model selection is necessary, the response unit 15 outputs the (final) response using the prompt as it is (the prompt input in step S10). Further, when it is determined in step S12 that external information search is necessary, the response unit 15 outputs the (final) response using the external information search result and the prompt input in step S10. Further, when it is determined in step S12 that another model is to be used, the response unit 15 switches the model and outputs the (final) response based on the switched model.

[0050] Next, with reference to Figures 7 to 36, we will explain from another perspective an example of the specific decision processing and answer processing performed by Device 1. Figures 7 to 16 explain an example of specific processing without sampling responses, Figures 23 to 36 explain an example of specific processing with sampling responses, and the remaining Figures 17 to 22 explain an example of answer processing common to both processes. In this process, we assume a scenario in which Device 1 answers questions about an idol group. As a premise, we assume that LLM40 does not have information about the members who served as center in "Garururu," but does have information about Kuroishi Ai's date of birth.

[0051] Figure 7 is a sequence diagram (part 1) showing an example of a specific process performed by device 1 without a sampling response.

[0052] First, terminal 2 sends a question to device 1 (step S20). Here, the question is, "What is the birthdate of the member who served as the center in 'Garururu'?" Next, the generation unit 13 of device 1 generates a knowledge judgment prompt 1 based on the question received in step S20 (step S21).

[0053] Figure 8 shows an example of the knowledge judgment prompt 1. In the knowledge judgment prompt 1 shown in Figure 8, the "{question}" is entered with the question "What is the birthdate of the member who served as the center in 'Garururu'?" which was acquired by the device 1 (its acquisition unit 12) in step S20.

[0054] Next, the generation unit 13 sends the knowledge judgment prompt 1 generated in step S21 to the LLM 40 (step S22). Next, the LLM 40 executes the knowledge judgment prompt 1 received in step S22 by inputting it, and outputs query 1, "Who were the members who served as the center in 'Garururu'?" as an output query. Figure 9 is a diagram showing an example of output based on the knowledge judgment prompt 1. Next, the LLM 40 sends the output query 1 to the device 1 (generation unit 13) (as a response to the transmission in step S22) (step S23).

[0055] Next, the acquisition unit 12 performs an external search on Wikipedia (registered trademark) based on the query 1 acquired in step S23 (step S24), and receives query result 1, which is the search result and external information, as a response: "The member who served as the center in 'Garururu' is Ai Kuroishi." (step S25). Figure 10 shows an example of the first search result of the external information.

[0056] Figure 11 is a sequence diagram (part 2) showing an example of a specific process without sampling response performed by the device 1. The sequence diagram shown in Figure 11 is a continuation of the sequence diagram shown in Figure 7. That is, following step S25 in the sequence diagram shown in Figure 7, step S30 in the sequence diagram shown in Figure 11 is executed.

[0057] First, the generation unit 13 of the device 1 generates a knowledge judgment prompt 2 based on the question, query 1, and query result 1 acquired in steps S20, S23, and S25, respectively, in Figure 7 (step S30).

[0058] Figure 12 is a diagram (part 1) showing an example of knowledge judgment prompt 2. Figure 13 is a diagram (part 2) showing an example of knowledge judgment prompt 2. The knowledge judgment prompt 2 shown in Figures 12 and 13 is a single entity, with the first half shown in Figure 12 and the second half in Figure 13. In other words, the knowledge judgment prompt 2 shown in Figure 13 is a continuation of the knowledge judgment prompt 2 shown in Figure 12.

[0059] In the knowledge judgment prompt 2 shown in Figure 13, "2" is entered in "{loop_count}" to indicate that this is the second process following step S22 in Figure 7, the question "What is the birthdate of the member who served as the center in 'Garururu'?" which was obtained in step S20 in Figure 7, and query 1 "Who is the member who served as the center in 'Garururu'?" and query result 1 "The member who served as the center in 'Garururu' is Ai Kuroishi." which were obtained in steps S23 and S25 in Figure 7, respectively.

[0060] Next, the generation unit 13 sends the knowledge judgment prompt 2 generated in step S30 to the LLM 40 (step S31). Next, the LLM 40 executes the knowledge judgment prompt 2 received in step S31 by inputting it, and outputs query 2, "What is Ai Kuroishi's date of birth?" as an output query. Figure 14 shows an example of the first output based on the knowledge judgment prompt 2. Next, the LLM 40 sends the output query 2 to the device 1 (generation unit 13) (as a response to the transmission in step S31) (step S32).

[0061] Next, the acquisition unit 12 performs an external search on Wikipedia (registered trademark) based on the query 2 acquired in step S32 (step S33), and receives the query result 2, which is the search result and external information, "Ai Kuroishi's date of birth is October 22, 1994," as a response (step S34). Figure 15 shows an example of the second search result for external information.

[0062] In the specific processing without sampling responses, the problem lies in the processing of steps S32 to S34. Specifically, as assumed above, although LLM40 has the information of when Ai Kuroishi was born, LLM40 cannot accurately determine that it has this information, and therefore determines that the information does not exist, resulting in the unnecessary external information search in steps S33 and S34.

[0063] Next, the generation unit 13 of the device 1 generates (regenerates) a knowledge judgment prompt 2 based on the question, query 1, query result 1, query 2, and query result 2 acquired in steps S20, S23, and S25 of Figure 7, and in steps S32 and S34 of Figure 8, respectively (step S35).

[0064] In this case, the "{loop_count}" of the knowledge judgment prompt 2 shown in Figure 13 is entered with "3", indicating that this is the third process following step S22 in Figure 7 and step S31 in Figure 11. The question obtained in step S20 of Figure 7, "What is the birthdate of the member who served as the center in 'Garururu'?" is entered into "{question}". The query 1 "Who is the member who served as the center in 'Garururu'?", query result 1 "The member who served as the center in 'Garururu' is Ai Kuroishi.", query 2 "When is Ai Kuroishi's birthdate?", and query result 2 "Ai Kuroishi's birthdate is October 22, 1994." are entered into "reference_QA", which were obtained in steps S23 and S25 of Figure 7 and steps S32 and S34 of Figure 11, respectively.

[0065] Next, the generation unit 13 transmits the knowledge judgment prompt 2 generated (regenerated) in step S35 to the LLM 40 (step S36). Next, the LLM 40 executes the knowledge judgment prompt 2 received in step S36 by inputting it, and outputs "Termination" as output. Figure 16 shows an example of the second output based on the knowledge judgment prompt 2. Next, the LLM 40 transmits the output to the device 1 (generation unit 13) (as a response to the transmission in step S36) (step S37).

[0066] Furthermore, device 1 repeatedly performs the processes shown in steps S33 to S36 until it receives the message "termination".

[0067] Figure 17 is a sequence diagram (part 1) showing an example of the response processing performed by the device 1. The sequence diagram shown in Figure 17 shows the response processing that follows the processing up to that point (for example, the processing shown in Figures 7 and 11, or for example, the processing shown in Figures 23, 26, 30, and 33 described later) when no external search was performed in the preceding processing.

[0068] First, the generation unit 13 generates a final answer acquisition prompt 1 based on the question acquired in step S20 of Figure 7 (step S40).

[0069] Figure 18 shows an example of the final answer acquisition prompt 1. In the final answer acquisition prompt 1 shown in Figure 18, the question "What is the birthdate of the member who served as the center in 'Garururu'?" is entered into "{question}".

[0070] Next, the generation unit 13 sends the final answer acquisition prompt 1 generated in step S40 to the LLM 40 (step S41). Next, the LLM 40 executes by inputting the final answer acquisition prompt 1 received in step S41, and outputs the answer "October 22, 1994" as output. Figure 19 shows an example of output based on the final answer acquisition prompt 1. Next, the LLM 40 sends the outputted answer to the device 1 (generation unit 13) (as a response to the transmission in step S41) (step S42). Next, the answer unit 15 of the device 1 sends the answer acquired in step S42 to the terminal 2 (as a response to the question in step S20 in Figure 7) (step S43), and the answer is displayed to the user, for example, on the screen of the terminal 2.

[0071] Figure 20 is a sequence diagram (part 2) showing an example of the response processing performed by device 1. The sequence diagram shown in Figure 20 shows the response processing that follows the previous processing when an external search is performed in the preceding processing (for example, the processing shown in Figures 7 and 11, or for example, the processing shown in Figures 23, 26, 30 and 33 described later).

[0072] First, the generation unit 13 generates a final response acquisition prompt 2 based on the question, query 1, query result 1, query 2, and query result 2 acquired in steps S20, S23, and S25 in Figure 7, and in steps S32 and S34 in Figure 8, respectively (step S50).

[0073] Figure 21 shows an example of the final answer acquisition prompt 2. In the final answer acquisition prompt 2 shown in Figure 21, the question "What is the birthdate of the member who served as the center in 'Garururu'?" obtained in step S20 of Figure 7 is entered into "{question}", and the query result 1 "Who is the member who served as the center in 'Garururu'?", query result 1 "The member who served as the center in 'Garururu' is Ai Kuroishi.", query 2 "What is Ai Kuroishi's birthdate?", and query result 2 "Ai Kuroishi's birthdate is October 22, 1994." obtained in steps S23 and S25 of Figure 7 and steps S32 and S34 of Figure 8, respectively.

[0074] Next, the generation unit 13 sends the final answer acquisition prompt 2 generated in step S50 to the LLM 40 (step S51). Next, the LLM 40 executes by inputting the final answer acquisition prompt 2 received in step S51, and outputs the answer "October 22, 1994" as output. Figure 22 is a diagram showing an example of output based on the final answer acquisition prompt 2. Next, the LLM 40 sends the outputted answer to the device 1 (generation unit 13) (as a response to the transmission in step S51) (step S52). Next, the answer unit 15 of the device 1 sends the answer acquired in step S52 to the terminal 2 (as a response to the question in step S20 in Figure 7) (step S53), and the answer is displayed to the user, for example, on the screen of the terminal 2.

[0075] Figure 23 is a sequence diagram (part 1) showing an example of a specific process with sampling responses performed by device 1.

[0076] First, terminal 2 sends a question to device 1 (step S60). The question is, "What is the birthdate of the member who served as the center in 'Garururu'?" Next, the generation unit 13 of device 1 generates a sampling response acquisition prompt 1 based on the question received in step S60 (step S61).

[0077] Figure 24 shows an example of the sampling response acquisition prompt 1. In the sampling response acquisition prompt 1 shown in Figure 24, the question "What is the birthdate of the member who served as the center in 'Garururu'?" is entered into "{question}", which was acquired by the device 1 (its acquisition unit 12) in step S60.

[0078] Next, the generation unit 13 sends the sampling response acquisition prompt 1 generated in step S61 to the LLM 40 (step S62). Then, the LLM 40 executes (for the first time) by inputting the sampling response acquisition prompt 1 received in step S62, and outputs "Sampling response: 0" and "The member who served as the center in 'Garururu' is Ai Kuroishi of Sanno 48. Her date of birth is October 22, 1994." Next, the LLM 40 sends the outputted "Sampling response: 0" to the device 1 (generation unit 13) (as a response to the transmission in step S62) (step S63), and the acquisition unit 12 acquires it.

[0079] Next, the generation unit 13 sends the sampling response acquisition prompt 1 generated in step S61 to the LLM 40 (step S64). Next, the LLM 40 executes (for the second time) by inputting the sampling response acquisition prompt 1 received in step S64, and outputs "Sampling response: 1" as "The member who served as the center in 'Garururu' is Kana Namagome of Sanno 48. Her date of birth is March 24, 1997." Next, the LLM 40 sends the outputted "Sampling response: 1" to the device 1 (generation unit 13) (as a response to the transmission in step S64) (step S65), and the acquisition unit 12 acquires it.

[0080] The generation unit 13 may also obtain 10 sampled responses by repeating the process of inputting the sampling response acquisition prompt 1 to the LLM 40 10 times by raising the Temperature of the LLM 40 (for example, to "0.8").

[0081] Figure 25 shows an example of output based on the sampling response acquisition prompt 1. In the output shown in Figure 25, each sampling response acquired by the generation unit 13 from the LLM 40 is arranged in the order it was acquired.

[0082] Figure 26 is a sequence diagram (part 2) showing an example of a specific process with sampling responses performed by device 1. The sequence diagram shown in Figure 26 is a continuation of the sequence diagram shown in Figure 23. That is, following step S65 of the sequence diagram shown in Figure 23 (more precisely, after device 1 has acquired all the sampling responses (for example, 10 times)), step S70 of the sequence diagram shown in Figure 26 is executed.

[0083] First, the generation unit 13 of the device 1 generates a knowledge judgment prompt 3 based on the questions and sampled answers obtained in steps S60, S63, and S65, etc., in Figure 23 (step S70).

[0084] Figure 27 is a diagram (part 1) showing an example of the knowledge judgment prompt 3. Figure 28 is a diagram (part 2) showing an example of the knowledge judgment prompt 3. The knowledge judgment prompt 3 shown in Figures 27 and 28 is a single entity, with the first half shown in Figure 27 and the second half in Figure 28. In other words, the knowledge judgment prompt 3 shown in Figure 28 is a continuation of the knowledge judgment prompt 3 shown in Figure 27.

[0085] In the knowledge judgment prompt 3 shown in Figure 28, the question "What is the birthdate of the member who served as the center in 'Garururu'?" is entered into "{question}", which was obtained in step S60 of Figure 23, and the sampled answer shown in Figure 25, which was obtained in steps S63 and S65 of Figure 23, is entered into "sampling_answer".

[0086] Next, the generation unit 13 sends the knowledge judgment prompt 3 generated in step S70 to the LLM 40 (step S71). The LLM 40 then executes the knowledge judgment prompt 3 received in step S71 by inputting it, and outputs query 3, "Who were the members who served as the center in 'Garururu'?" as an output query. Figure 29 shows an example of the output based on the knowledge judgment prompt 3. The LLM 40 then sends the output query 3 to the device 1 (generation unit 13) (as a response to the transmission in step S71) (step S72). The processing in steps S70 to S72 corresponds to the process of determining how to answer the question based on multiple answers.

[0087] Next, the acquisition unit 12 performs an external search on Wikipedia (registered trademark) based on the query 3 acquired in step S72 (step S73), and receives the query result 3, which is the search result and external information, as a response: "The member who served as the center in 'Garururu' is Ai Kuroishi." (step S74). An example of query result 3 is the same as that shown in Figure 10.

[0088] Figure 30 is a sequence diagram (part 3) showing an example of a specific process with sampling response performed by device 1. The sequence diagram shown in Figure 30 is a continuation of the sequence diagram shown in Figure 26. That is, following step S74 in the sequence diagram shown in Figure 26, step S80 in the sequence diagram shown in Figure 30 is executed.

[0089] First, the generation unit 13 of the device 1 generates a sampling response acquisition prompt 2 based on the question, query 3, and query result 3 acquired in step S60 in Figure 23 and in steps S72 and S74 in Figure 26, respectively (step S80).

[0090] Figure 31 shows an example of the sampling response acquisition prompt 2. In the sampling response acquisition prompt 2 shown in Figure 31, "1" is entered in "{loop_count}" to indicate that it is the first processing, "What is the date of birth of the member who served as the center in 'Garururu'?" is entered in "{question}", which is the question acquired by the device 1 (its acquisition unit 12) in step S60 of Figure 23, and "{reference_QA}" is entered in query 3 "Who is the member who served as the center in 'Garururu'?" and query result 3 "The member who served as the center in 'Garururu' is Ai Kuroishi." which were acquired in steps S72 and S74 of Figure 26, respectively.

[0091] Next, the generation unit 13 sends the sampling response acquisition prompt 2 generated in step S80 to the LLM 40 (step S81). Next, the LLM 40 executes (the first time) by inputting the sampling response acquisition prompt 2 received in step S81, and outputs "Sampling response: 10" which means "Ai Kuroishi's date of birth is October 22, 1994." Next, the LLM 40 sends the outputted "Sampling response: 10" to the device 1 (generation unit 13) (as a response to the transmission in step S81) (step S82), and the acquisition unit 12 acquires it.

[0092] Next, the generation unit 13 sends the sampling response acquisition prompt 2 generated in step S80 to the LLM 40 (step S83). Next, the LLM 40 executes (for the second time) by inputting the sampling response acquisition prompt 2 received in step S83, and outputs "Sampling response: 11" which reads "Ai Kuroishi's date of birth is October 22, 1994." Next, the LLM 40 sends the outputted "Sampling response: 11" to the device 1 (generation unit 13) (as a response to the transmission in step S83) (step S84), and the acquisition unit 12 acquires it.

[0093] The generation unit 13 may also obtain 10 sampled responses by repeating the process of inputting the sampling response acquisition prompt 2 to the LLM 40 10 times by raising the Temperature of the LLM 40 (for example, to "0.8").

[0094] Figure 32 shows an example of output based on the sampling response acquisition prompt 2. In the output shown in Figure 32, each sampling response acquired by the generation unit 13 from the LLM 40 is arranged in the order it was acquired.

[0095] Figure 33 is a sequence diagram (part 4) showing an example of a specific process with sampling responses performed by device 1. The sequence diagram shown in Figure 33 is a continuation of the sequence diagram shown in Figure 30. That is, following step S84 of the sequence diagram shown in Figure 30 (more precisely, after device 1 has acquired all the sampling responses (for example, 10 times)), step S90 of the sequence diagram shown in Figure 33 is executed.

[0096] First, the generation unit 13 of the device 1 generates a knowledge judgment prompt 4 based on the questions, queries 3, query results 3, and sampled responses obtained in steps S60 in Figure 23, steps S72 and S74 in Figure 26, and steps S82 and S84 in Figure 30, respectively (step S90).

[0097] Figure 34 is a diagram (part 1) showing an example of the knowledge judgment prompt 4. Figure 35 is a diagram (part 2) showing an example of the knowledge judgment prompt 4. The knowledge judgment prompt 4 shown in Figures 34 and 35 is a single entity, with the first half shown in Figure 34 and the second half in Figure 35. In other words, the knowledge judgment prompt 4 shown in Figure 35 is a continuation of the knowledge judgment prompt 4 shown in Figure 34.

[0098] In the knowledge judgment prompt 4 shown in Figure 35, "2" is entered into "{loop_count}" to indicate that this is the second processing following step S71 in Figure 26, "{question}" is entered into "{question}", which is the question acquired by device 1 (its acquisition unit 12) in step S60 in Figure 23, "What is the birthdate of the member who served as the center in 'Garururu'?", "{reference_QA}" is entered into "Query 3 "Who is the member who served as the center in 'Garururu'?" and query result 3 "The member who served as the center in 'Garururu' is Ai Kuroishi." acquired in steps S72 and S74 of Figure 26 respectively, and "sampling_answer" is entered into the sampled answer shown in Figure 32, which was acquired in steps S82 and S84 of Figure 30, etc.

[0099] Next, the generation unit 13 transmits the knowledge judgment prompt 4 generated in step S90 to the LLM 40 (step S91). Next, the LLM 40 executes by inputting the knowledge judgment prompt 4 received in step S91 and outputs "Finished". Figure 36 is a diagram showing an example of output based on the knowledge judgment prompt 4. Next, the LLM 40 transmits the output to the device 1 (generation unit 13) (as a response to the transmission in step S91) (step S92). The processing in steps S90 to S92 corresponds to the process of determining how to answer a question based on multiple answers.

[0100] Furthermore, device 1 repeatedly performs the processes shown in steps S73 to S91 until it receives the message "termination".

[0101] The response to terminal 2 following step S92 is the same as that explained using Figures 17 to 22, so the explanation is omitted.

[0102] As described above, in the specific processing with sampled responses, the first sampling (steps S62-S65 in Figure 23, etc.) does not yield a match in the response (Figure 25), so query 3 is generated (step S72 in Figure 26). On the other hand, in the second sampling (steps S81-S84 in Figure 30, etc.), the response (Figure 32) matches (indicating that LLM 40 possesses the knowledge), so no query is generated (step S92 in Figure 33). In this way, the number of queries (searches for external information) can be reduced compared to the specific processing without sampled responses. In this embodiment, external information was used when the responses did not match, but it is also possible to have other high-performance (LLM) models provide the response.

[0103] In specific processes involving sampled responses, multi-stage reasoning is performed, but the answer may be provided with only one inference (external search). For example, if an external search is necessary at Knowledge Judgment Prompt 3, the external search may be performed; otherwise, the system may proceed to the final answer. However, since multi-stage reasoning is not performed, the statement "However, in this process... consider what question should be asked first." in Knowledge Judgment Prompt 3 should be deleted. Also, unlike Sampled Response Acquisition Prompt 2 and Knowledge Judgment Prompt 4, external searches are not performed repeatedly; the answer is provided with only one inference (external search).

[0104] The determination unit 14 may estimate the type of hallucination based on multiple responses and determine the response method according to the estimated type of hallucination.

[0105] Types of hallucination include, for example, factual hallucination ("Fact-Conflicting Hallucination") and contextual hallucination ("Input-Conflicting Hallucination" and "Context-Conflicting Hallucination"). Fact-conflicting hallucination is output that does not conform to factual knowledge and can lead to user misunderstanding. Input-conflicting hallucination is output that deviates from user input, i.e., output that does not conform to instructions. Context-conflicting hallucination is output that does not conform to pre-generated text and occurs in long responses or multi-turn responses.

[0106] In the case of factual hallucination, the decision unit 14 may decide to respond using external information. In the case of contextual hallucination, the decision unit 14 may decide to respond using a different LLM (a different model, a higher-performing model) than the LLM 40.

[0107] The decision unit 14 may estimate the type of hallucination (e.g., "factual hallucination," "contextual hallucination," "no hallucination") using hallucination judgment prompts as shown in Figures 37 to 39. Figure 37 is an example (1) of a hallucination judgment prompt. Figure 38 is an example (2) of a hallucination judgment prompt. Figure 39 is an example (3) of a hallucination judgment prompt. There is only one hallucination judgment prompt as shown in Figures 37 to 39, with the first third shown in Figure 37, the next third in Figure 38, and the remaining third in Figure 39. That is, the hallucination judgment prompt shown in Figure 37 is followed by the hallucination judgment prompt shown in Figure 38, and the hallucination judgment prompt shown in Figure 38 is followed by the hallucination judgment prompt shown in Figure 39. The "{question}" and "sampling_answer" shown in Figure 39 are input with the same content as the specific processing described above. Furthermore, the hallucination decision prompt is executed by LLM40, similar to the specific processing described above.

[0108] The decision unit 14 may, using the hallucination judgment prompt, aim to suppress hallucination by acquiring knowledge from an external source through an external search if it determines that factual hallucination is occurring. Alternatively, the decision unit 14 may, using the hallucination judgment prompt, aim to suppress hallucination by using a higher-performance model if it determines that contextual hallucination is occurring. Furthermore, if the decision unit 14 determines that no hallucination is occurring using the hallucination judgment prompt, it may proceed to the final answer without performing an external search or model selection. The decision unit 14 may also determine the presence or absence of hallucination using a sampled response and a model used in cases where there is no hallucination (e.g., LLM40).

[0109] The decision unit 14 may determine the response method based on a specialized model (decision model) for detecting hallucination. The specialized model may be generated by the learning unit (not shown) of the device 1 using training data as shown in Figure 40. For example, the decision unit 14 may input the question and sampled response into the specialized model and determine the response method based on the conclusion output by the specialized model (e.g., "no external search or model selection required", "external search required", "model selection required").

[0110] If the decision unit 14 determines that it is necessary to use an agent based on multiple answers obtained by inputting the same question multiple times to the LM40, it may use the agent utilization unit (not shown) of the device 1.

[0111] In the agent utilization unit, information necessary for inference may be collected using the agent, and the collected information may be added to the answer output by the answer unit 15. For example, in response to the user's question, "According to Company X's internal regulations, can I use an airplane to travel from Tokyo to Osaka?", if fluctuations occur in the answer as a result of obtaining sampled answers, the Company X internal data acquisition agent may be used to obtain information from the travel expense application document, and this information may be added to the answer in a context-learning manner.

[0112] In the agent utilization section, the final answer may be derived using an agent instead of the answer section 15. For example, in response to a user question, "Has the number of mobile phone contracts at Company X been increasing recently?", an internal data analysis agent may be used to have the agent handle everything from data acquisition to output.

[0113] Next, the operation and effects of the apparatus 1 according to this embodiment will be described.

[0114] The device 1 includes an acquisition unit 12 that acquires multiple answers obtained by inputting the same question multiple times to the LLM 40 (generating AI), and a determination unit 14 that determines a method of answering the question based on the multiple answers. With this configuration, the method of answering the question is determined based on the multiple answers obtained by inputting the same question multiple times to the LLM 40. In other words, the method of answering the question can be determined. Furthermore, by using multiple answers obtained by inputting the same question multiple times to the LLM 40, the method of answering the question can be determined more accurately, for example, based on the reliability of the knowledge within the LLM 40. Also, once the method of answering the question is determined, for example, another process based on the determined answer method can be performed (automatically).

[0115] In device 1, the multiple responses may consist of the responses output each time the same question is input to the LLM 40 multiple times. With this configuration, for example, the multiple responses consisting of the responses output each time the same question is input to the LLM 40 multiple times will be diverse responses, and therefore decisions that take diversity into account can be made.

[0116] In device 1, multiple responses may be output by an LLM40 set to have a higher randomness of output than the normal setting. With this configuration, the multiple responses will be more random (or diverse) than with the normal setting, and therefore decisions can be made that take randomness (or diversity) into account.

[0117] In the device 1, the decision of the decision unit 14 may be based on the input of a prompt to the LLM (LLM 40 or an LLM other than LLM 40) that includes an instruction to determine how to answer a question based on multiple answers. This configuration allows for more flexible decision-making, for example, by utilizing the flexibility of the LLM.

[0118] The determination unit 14 of the device 1 may determine the method of answering using the LLM 40 as the answer method. With this configuration, for example, it is possible to answer questions more easily and at a lower cost based on the determined answer method.

[0119] The determination unit 14 of the device 1 may determine a method of answering using external information as the answer method. With this configuration, for example, it is possible to provide a more accurate answer to a question using external information based on the determined answer method.

[0120] The determination unit 14 of the device 1 may determine a method of answering using external information and generate queries to acquire external information. With this configuration, for example, it is possible to provide more accurate answers to questions using external information based on the determined answer method, and to acquire external information more easily and accurately based on the generated queries.

[0121] The determination unit 14 of the device 1 may determine a method of answering using an LLM different from LLM 40 as the answer method. With this configuration, for example, it is possible to provide a more accurate answer to the question using an LLM different from LLM 40 based on the determined answer method.

[0122] The device 1 may further include an answer unit 15 that outputs an answer to a question based on an answer method. With this configuration, for example, an answer to a question is output based on a determined, more accurate answer method, thus enabling the output of a more accurate answer.

[0123] Device 1 optimizes external searches using sampling. By determining whether or not to perform external searches and model selection using sampling, Device 1 provides a method to prevent hallucination caused by errors in model selection specific to external searches. Device 1 provides a method to optimize the selection of models to be used or the use of external information by pre-determining the reliability of inference by using repeated sampling before inference in LLM40.

[0124] Two reasons can be cited for this. Firstly, while LLM (Generative AI) can store knowledge within the model through pre-training, it is conceivable that external information can be utilized using external data and retrieval when knowledge that has not been learned is required. Secondly, in recent years, models have become larger in scale, and while larger models are high-performance, the inference cost (hourly rate, etc.) also increases. Herein, the model does not accurately grasp the knowledge it holds within itself. Therefore, methods such as using the model's internal parameters (weights, biases, etc.) to judge the reliability of the knowledge within the model can be considered.

[0125] Three challenges can be identified. First, external searches carry the risk of new hallucinations, such as errors in document retrieval and information extraction. Therefore, external searches should be performed only when necessary. Second, while larger models offer higher performance, their inference costs increase. Selecting the optimal model size for the input is considered important. Third, the method of using the internal parameters of the model described above is limited to certain models and cannot be used with models that do not allow the acquisition of internal parameters, such as the widely used OpenAI™ GPT.

[0126] Device 1 uses an iterative sampling method, which is a technique for determining the reliability of a model's knowledge, to clarify whether an answer can be provided using only the knowledge within the model, and to determine the need for external information and the need to use other models. Because Device 1 can determine the reliability of the knowledge within the model more accurately than when sampling is not used, it is possible to determine the need for external information and the need to use other models more accurately than when the technique is not used. This suppresses the search for unnecessary external information and reduces the occurrence of hallucination caused by external searches. In addition, since high-performance and high-cost models are used only when necessary, it leads to a reduction in inference costs. Furthermore, because it can be used with many models compared to methods that use internal model parameters, it can be used even for models where only input and output can be checked, such as OpenAI® models.

[0127] The apparatus 1 of this disclosure may have the following configuration.

[0128] [1] An apparatus comprising: an acquisition unit that acquires multiple responses obtained by inputting the same input information multiple times to a generating AI; and a determination unit that determines a method for responding to the input information based on the multiple responses.

[0129] [2] The apparatus according to [1], wherein the plurality of responses consist of responses output each time by inputting the same input information to the generating AI multiple times.

[0130] [3] The apparatus according to [1] or [2], wherein the multiple responses are output by the generating AI, which is set to have a higher randomness of output than the normal setting.

[0131] [4] The apparatus according to any one of [1] to [3], wherein the determination of the determination unit is based on input to a prompt generation AI that includes instructions to determine the method of responding to the input information based on the plurality of responses.

[0132] [5] The apparatus according to any one of [1] to [4], wherein the determination unit determines a method of answering using the generating AI as the answer method.

[0133] [6] The apparatus according to any one of [1] to [5], wherein the determination unit determines a method of answering using external information as the answer method.

[0134] [7] The apparatus according to any one of [1] to [6], wherein the determination unit determines a method of answering using external information as the answer method and generates a query to obtain the external information.

[0135] [8] The apparatus according to any one of [1] to [7], wherein the determination unit determines a method of answering using a generation AI different from the generation AI as the answer method.

[0136] [9] The apparatus according to any one of [1] to [8], further comprising a response unit that outputs a response to the input information based on the response method.

[0137] The block diagrams used in the description of the above embodiments show functional units. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or it may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired or wireless connections). A functional block may also be realized by combining software with the one or more of the above devices.

[0138] Functions include, but are not limited to, judgment, decision, determination, calculation, calculation, processing, derivation, investigation, exploration, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, assumption, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating (mapping), and assigning. For example, a functional block (configuration part) that enables transmission is called a transmitting unit or transmitter. In all cases, as mentioned above, the method of implementation is not particularly limited.

[0139] For example, Apparatus 1 in one embodiment of the present disclosure may function as a computer that processes the determination method and response method of the present disclosure. Figure 41 is a diagram showing an example of the hardware configuration of Apparatus 1 according to one embodiment of the present disclosure. Physically, Apparatus 1 described above may be configured as a computer device including a processor 1001, memory 1002, storage 1003, communication device 1004, input device 1005, output device 1006, bus 1007, etc.

[0140] In the following explanation, the term "device" can be replaced with "circuit," "device," "unit," etc. The hardware configuration of device 1 may include one or more of the devices shown in the figure, or it may be configured to omit some of the devices.

[0141] Each function in device 1 is realized by loading predetermined software (programs) onto hardware such as the processor 1001 and memory 1002, which allows the processor 1001 to perform calculations, control communication by the communication device 1004, and control at least one of data reading and writing in the memory 1002 and storage 1003.

[0142] The processor 1001 controls the entire computer, for example, by running an operating system. The processor 1001 may be composed of a central processing unit (CPU) that includes interfaces with peripheral devices, control devices, arithmetic units, registers, etc. For example, the setting unit 11, acquisition unit 12, generation unit 13, determination unit 14, and answer unit 15 described above may be implemented by the processor 1001.

[0143] Furthermore, the processor 1001 reads programs (program code), software modules, data, etc., from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes accordingly. The program used is one that causes the computer to execute at least a part of the operations described in the above embodiment. For example, the setting unit 11, acquisition unit 12, generation unit 13, determination unit 14, and answer unit 15 may be stored in the memory 1002 and implemented by a control program that operates on the processor 1001, and other functional blocks may be implemented similarly. The above-described various processes have been explained as being executed by one processor 1001, but they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may also be transmitted from a network via a telecommunications line.

[0144] The memory 1002 is a computer-readable recording medium and may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. The memory 1002 may also be called a register, cache, main memory, etc. The memory 1002 can store executable programs (program code), software modules, etc., for carrying out a wireless communication method according to one embodiment of the present disclosure.

[0145] The storage 1003 is a computer-readable recording medium and may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., Compact Disc, Digital Multipurpose Disc, Blu-ray® Disc), a smart card, flash memory (e.g., a card, stick, key drive), a floppy® disk, a magnetic strip, etc. The storage 1003 may also be called an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, server, or other suitable medium including at least one of memory 1002 and storage 1003.

[0146] The communication device 1004 is hardware (transceiver / receiver device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as a network device, network controller, network card, communication module, etc. The communication device 1004 may be configured to include, for example, a high-frequency switch, duplexer, filter, frequency synthesizer, etc., in order to implement at least one of frequency division duplex (FDD) and time division duplex (TDD). For example, the setting unit 11, acquisition unit 12, generation unit 13, determination unit 14, and response unit 15 described above may be implemented by the communication device 1004.

[0147] The input device 1005 is an input device that accepts input from an external source (e.g., a keyboard, mouse, microphone, switch, button, sensor, etc.). The output device 1006 is an output device that outputs to an external source (e.g., a display, speaker, LED lamp, etc.). The input device 1005 and the output device 1006 may be configured as an integrated unit (e.g., a touch panel).

[0148] Furthermore, each device, such as the processor 1001 and memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or different buses may be configured for each device.

[0149] Furthermore, the device 1 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array), and some or all of each functional block may be realized by such hardware. For example, the processor 1001 may be implemented using at least one of these hardware components.

[0150] The notification of information is not limited to the manner / embodiments described herein and may be carried out by other means.

[0151] Each aspect / embodiment described in this disclosure may be applied to at least one of the following systems: LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (new Radio), W-CDMA®, GSM®, CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi®), IEEE 802.16 (WiMAX®), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth®, and other appropriate systems, as well as next-generation systems extended based thereon. Furthermore, multiple systems may be applied in combination (for example, a combination of at least one of LTE and LTE-A with 5G).

[0152] The processing procedures, sequences, flowcharts, etc., of each aspect / embodiment described in this disclosure may be reordered, provided they do not contradict each other. For example, the methods described in this disclosure present various step elements using exemplary order and are not limited to the specific order presented.

[0153] Input and output information may be stored in a specific location (e.g., memory) or managed using a management table. Input and output information may be overwritten, updated, or appended to. Output information may be deleted. Input information may be transmitted to other devices.

[0154] The determination may be made by a value represented by one bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).

[0155] Each aspect / embodiment described in this disclosure may be used individually, in combination, or switched between as needed during implementation. Furthermore, notification of specific information (e.g., notification that "X is") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0156] Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the intent and scope of the present disclosure as defined by the claims. Therefore, the descriptions in the present disclosure are illustrative and not intended to be restrictive in any way.

[0157] Software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, and so on, whether they are called software, firmware, middleware, microcode, hardware description languages, or by any other name.

[0158] Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technology (such as infrared or microwave), then at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0159] The information, signals, etc. described in this disclosure may be represented using any of the various different techniques. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0160] In addition, terms used in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meaning.

[0161] The terms “system” and “network” as used in this disclosure are interchangeable.

[0162] Furthermore, the information, parameters, etc., described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0163] The names used for the parameters described above are not restrictive in any way. Furthermore, the formulas and other expressions using these parameters may differ from those expressly disclosed in this disclosure.

[0164] The terms “determining” and “decision” as used in this disclosure may encompass a wide variety of actions. “Determining” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in tables, databases or other data structures), and ascertaining. “Determining” may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory). Furthermore, “determining” may include resolving, selecting, choosing, establishing, and comparing. In other words, "judgment" and "decision" can include considering that some action has been "judged" or "decided." Also, "judgment (decision)" can be reinterpreted as "assuming," "expecting," or "considering."

[0165] The terms “connected,” “coupled,” and any variations thereof mean any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” with each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, “connection” may be reinterpreted as “access.” As used in this disclosure, two elements may be considered to be “connected” or “coupled” with each other using at least one of one or more wires, cables, and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain, and optical (both visible and invisible) domain.

[0166] In this disclosure, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on."

[0167] Any reference to elements using the designations “first,” “second,” etc., as used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Accordingly, references to the first and second elements do not imply that only two elements may be employed, or that the first element must precede the second element in any way.

[0168] In the configuration of each of the above devices, "means" may be replaced with "part," "circuit," "device," etc.

[0169] Where the terms “include,” “including,” and variations thereof are used in this disclosure, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to mean exclusive OR.

[0170] In this disclosure, if articles are added by translation, such as a, an, and the in English, this disclosure may include the fact that the noun following these articles is plural.

[0171] In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combine" may be interpreted similarly to "different."

[0172] 1...Device, 2...Terminal, 3...System, 4...LLM Server, 10...Storage Unit, 11...Setting Unit, 12...Acquisition Unit, 13...Generation Unit, 14...Decision Unit, 15...Response Unit, 40...LLM, 1001...Processor, 1002...Memory, 1003...Storage, 1004...Communication Device, 1005...Input Device, 1006...Output Device, 1007...Bus.

Claims

1. An apparatus comprising: an acquisition unit that acquires multiple responses obtained by inputting the same input information multiple times to a generating AI; and a determination unit that determines a method for responding to the input information based on the multiple responses.

2. The apparatus according to claim 1, wherein the plurality of responses consist of responses output each time the same input information is input multiple times to the generating AI.

3. The apparatus according to claim 1, wherein the multiple responses are output by the generating AI, which is set to have a higher randomness of output than the normal setting.

4. The apparatus according to claim 1, wherein the decision of the decision unit is based on input to a prompt generation AI that includes instructions to determine the method of responding to the input information based on the plurality of responses.

5. The apparatus according to claim 1, wherein the determination unit determines a method of answering using the generating AI as the answer method.

6. The apparatus according to claim 1, wherein the determination unit determines a method of answering using external information as the answer method.

7. The apparatus according to claim 1, wherein the determination unit determines a method of answering using external information as the answer method and generates a query to obtain the external information.

8. The apparatus according to claim 1, wherein the determination unit determines a method of answering using a generation AI different from the generation AI as the answer method.

9. The apparatus according to claim 1, further comprising a response unit that outputs a response to the input information based on the response method described above.

10. A method performed by a computer, comprising: an acquisition step of obtaining multiple responses obtained by inputting the same input information multiple times to a generating AI; and a determination step of determining a method for responding to the input information based on the multiple responses.