Presentation order determination device and presentation order determination method

The presentation order determination device and method address the challenge of ordering answer bases by their reliance on the answer, allowing for a credible assessment of multiple bases through a systematic approach.

WO2026239436A1PCT designated stage Publication Date: 2026-11-19NTT DOCOMO INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2025/017176
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-05-12
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing information processing systems do not effectively order answer bases according to their degree of reliance on the answer, making it difficult to assess the credibility of multiple bases provided.

Method used

A presentation order determination device and method that acquires multiple bases for an answer and determines their presentation order based on the degree of reliance of each base on the answer, using a system that includes a basis acquisition unit and a determination unit to sort and display the bases accordingly.

Benefits of technology

Enables the review of answer bases in an order that reflects their reliance on the answer, enhancing the credibility assessment of multiple bases.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2025017176_19112026_PF_FP_ABST
    Figure JP2025017176_19112026_PF_FP_ABST
Patent Text Reader

Abstract

This presentation order determination device includes: a basis acquisition unit that acquires at least one basis for an answer generated on the basis of a question and information related to the question; and a determination unit that, when the at least one basis is a plurality of bases for the answer, determines the presentation order of the plurality of bases on the basis of the degree of reliance of the answer on each of the plurality of bases.
Need to check novelty before this filing date? Find Prior Art

Description

Presentation order determination device and presentation order determination method

[0001] The present disclosure relates to a technique for determining a presentation order.

[0002] Patent Document 1 discloses an information processing apparatus that outputs an answer to a question and the basis for the answer.

[0003] International Publication No. 2024 / 261982

[0004] When the information processing apparatus disclosed in Patent Document 1 outputs a plurality of answer bases, there is a conceivable need to check the answer bases in an order based on the degree of reliance on the answer.

[0005] An object of the present disclosure is to provide a technique capable of meeting the need to check answer bases in an order based on the degree of reliance on the answer.

[0006] A presentation order determination device according to one aspect of the present disclosure includes: a basis acquisition unit that acquires at least one basis for an answer generated based on a question and information related to the question; and when the at least one basis is a plurality of bases for the answer, a determination unit that determines a presentation order of the plurality of bases based on the degree of reliance of each of the plurality of bases on the answer.

[0007] A presentation order determination method according to another aspect of the present disclosure includes: acquiring at least one basis for an answer generated based on a question and information related to the question; and when the at least one basis is a plurality of bases for the answer, determining a presentation order of the plurality of bases based on the degree of reliance of each of the plurality of bases on the answer.

[0008] According to the present disclosure, it is possible to provide a technique capable of meeting the need to check answer bases in an order based on the degree of reliance on the answer.

[0009] This is a diagram showing an example of the response system RS. This is a diagram showing an example of terminal 10. This is a diagram showing an example of server 30. This is a diagram showing an example of system prompt SP1. This is a diagram showing an example of system prompt SP2. This is a diagram showing an example of system prompt SP3. This is a diagram showing an example of question A1. This is a diagram showing an example of expert information B1. This is a diagram showing an example of prompt P1. This is a diagram showing an example of answer C1. This is a diagram showing an example of prompt P2. This is a diagram showing an example of at least one basis D1. This is a diagram showing a decision unit 5 which is an example of decision unit 335. This is a diagram showing an example of prompt P3. This is a diagram showing an example of dependence E1. This is a flowchart for explaining an example of the operation of the system prompt setting unit 331. This is a flowchart for explaining an example of the operation of server 30. This is a diagram showing an example of system prompt SP4. This is a diagram showing an answer C2 which is an example of answer C1. This is a diagram showing an example of two or more basis D2s. This is a diagram showing an example of prompt P4. This is a diagram showing an example of sort result F2 of two or more basis D2s. This is a diagram showing another example of answer C1, which is answer C3. This is a diagram showing another example of two or more basis D2s. This is a diagram showing an example of prompt P5. This is a diagram showing an example of sort result F3 of multiple basis D31s and D32s. This figure shows a determination unit 6, which is another example of a determination unit 335.

[0010] 1. Embodiments 1-1. Response System RS Figure 1 shows an example of the response system RS. The response system RS automatically responds to question A1 from user U1. The response system RS includes a terminal 10, an LLM (large language model) system 20, and a server 30.

[0011] Terminal 10 is, for example, a smartphone. However, terminal 10 is not limited to a smartphone. For example, terminal 10 may be a tablet or a personal computer. Terminal 10 can communicate with server 30 via a communication network NW. Terminal 10 may also communicate with server 30 without using a communication network NW. Terminal 10 is used by user U1. Terminal 10 receives question A1 from user U1.

[0012] The LLM system 20 is composed of multiple computers. The LLM system 20 may be composed of a single computer. The LLM system 20 functions as LLM 21. LLM 21 is, for example, a generative AI (Artificial Intelligence). LLM 21 generates information such as text in response to prompts. LLM 21 can communicate with server 30 via a communication network NW. LLM 21 may also be able to communicate with server 30 without using a communication network NW. LLM 21 is used by server 30, for example. LLM 21 may be included in server 30. LLM 21 may also be able to communicate with terminal 10.

[0013] Server 30 is composed of multiple computers. Server 30 may be composed of a single computer. Server 30 can communicate with terminal 10 via a communication network NW. Server 30 may also communicate with terminal 10 without using a communication network NW. Server 30 can communicate with LLM 21 via a communication network NW. Server 30 may also communicate with LLM 21 without using a communication network NW.

[0014] Server 30 stores specialized information B1. Server 30 may also retrieve specialized information B1 from other devices that store specialized information B1. Specialized information B1 is information that may not have been learned by LLM 21. For example, specialized information B1 is information about the latest knowledge. Specialized information B1 may include information about the knowledge held by a company. Specialized information B1 is information related to question A1. Specialized information B1 is used, for example, in-context learning (ICL).

[0015] Server 30 receives question A1 from terminal 10. Server 30 obtains answer C1 to question A1 from LLM 21 using question A1 and specialized information B1. Server 30 displays answer C1 on terminal 10 by transmitting it to terminal 10. In other words, server 30 presents answer C1 to user U1 using terminal 10.

[0016] The server 30 obtains at least one piece of evidence D1 for answer C1 from the LLM 21 by using answer C1 and expert information B1. The at least one piece of evidence D1 is information contained in expert information B1 that serves as the basis for answer C1.

[0017] If at least one piece of evidence D1 is one of several pieces of evidence D1 relating to answer C1, the server 30 determines the order in which to present the multiple pieces of evidence D1 based on the degree of reliance E1 of each piece of evidence D1 to answer C1. For example, the server 30 sorts the multiple pieces of evidence D1 based on the degree of reliance E1. The server 30 sends the sorted pieces of evidence D1 to the terminal 10, causing the terminal 10 to display the multiple pieces of evidence D1 in the order in which they were presented based on the degree of reliance E1. In other words, the server 30 presents the multiple pieces of evidence D1 to user U1 in the order in which they were presented based on the degree of reliance E1 by using the terminal 10.

[0018] For example, the server 30 sends multiple pieces of evidence D1 to the terminal 10 in order of decreasing dependence E1, thereby displaying the multiple pieces of evidence D1 on the terminal 10 in order of decreasing dependence E1. In other words, the server 30 presents multiple pieces of evidence D1 to the user U1 in order of decreasing dependence E1 by using the terminal 10.

[0019] The following describes examples of terminal 10 and server 30 included in the response system RS.

[0020] 1-2. Terminal 10 Figure 2 shows an example of terminal 10. Terminal 10 includes an input device 11, a display device 12, a communication device 13, a storage device 14, a processing device 15, and a bus 16.

[0021] Bus 16 is a wiring configuration for communicating information. Bus 16 connects the input device 11, the display device 12, the communication device 13, the storage device 14, and the processing device 15 to each other. Bus 16 may consist of, for example, a single bus. Bus 16 may also consist of buses provided between each device.

[0022] The input device 11 is the user interface of the terminal 10. The input device 11 includes a touch panel. In addition to the touch panel, the input device 11 may include a plurality of operation keys. The input device 11 may include a plurality of operation keys without including a touch panel. The input device 11 may include a voice input device. The input device 11 receives various inputs from the user U1. For example, the input device 11 receives a question A1 from the user U1.

[0023] The display device 12 displays various information. For example, the display device 12 displays question A1, answer C1, and at least one piece of evidence D1. For example, the display device 12 displays question A1, answer C1, and at least one piece of evidence D1 together. The display device 12 may also display question A1 separately from answer C1 and at least one piece of evidence D1. In this case, the display device 12 displays answer C1 and at least one piece of evidence D1 together. The display device 12 may also display question A1, answer C1, and at least one piece of evidence D1 separately.

[0024] The communication device 13 can communicate with the server 30 via the communication network NW. The communication device 13 may also be able to communicate with the server 30 without using the communication network NW. The communication device 13 sends question A1 to the server 30. The communication device 13 receives answer C1 and at least one piece of evidence D1 from the server 30.

[0025] The storage device 14 is a recording medium that can be read by the processing device 15. The storage device 14 includes at least one memory. The storage device 14 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory) and VRAM (Video Random Access Memory). The storage device 14 stores program PG1. Program PG1 includes at least one instruction.

[0026] The processing unit 15 includes at least one CPU (Central Processing Unit). The at least one CPU is an example of at least one processor. The at least one processor is an example of at least one computer. The processing unit 15 reads the program PG1 from the storage device 14. The processing unit 15 functions as an operation control unit 151 by executing the program PG1. The operation control unit 151 may be composed of circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

[0027] The operation control unit 151 controls the operation of the terminal 10. For example, the operation control unit 151 causes the communication device 13 to send question A1 to the server 30. The operation control unit 151 causes the display device 12 to display question A1, answer C1, and at least one piece of evidence D1.

[0028] 1-3. Server 30 Figure 3 shows an example of a server 30. Server 30 is an example of a presentation order determination device. Server 30 includes a communication device 31, a storage device 32, a processing device 33, and a bus 34.

[0029] Bus 34 is a wiring system for communicating information. Bus 34 connects the communication device 31, the storage device 32, and the processing device 33 to each other. Bus 34 may consist of, for example, a single bus. Bus 34 may also consist of buses provided between each device.

[0030] The communication device 31 can communicate with the terminal 10 via the communication network NW. The communication device 31 may also communicate with the terminal 10 without using the communication network NW. The communication device 31 can communicate with the LLM 21 via the communication network NW. The communication device 31 may also communicate with the LLM 21 without using the communication network NW. The communication device 31 may also communicate with an administrator terminal (not shown) operated by the administrator of the server 30.

[0031] The storage device 32 is a recording medium readable by the processing device 33. The storage device 32 includes at least one memory. The storage device 32 includes, for example, non-volatile memory and volatile memory. The storage device 32 stores a program PG2 and specialized information B1. The storage device 32 stores, for example, specialized information B1 obtained from another device. The program PG2 includes at least one instruction.

[0032] The processing unit 33 is another example of a presentation order determination device. The processing unit 33 includes at least one CPU. The processing unit 33 reads the program PG2 from the storage device 32. The processing unit 33 functions as a plurality of elements by executing the program PG2. The plurality of elements include a system prompt setting unit 331, a question receiving unit 332, an answer acquisition unit 333, a justification acquisition unit 334, and a determination unit 335. Each of the plurality of elements may be composed of a circuit such as a DSP, ASIC, or FPGA.

[0033] Furthermore, the processing unit 33 does not necessarily have to function as any of the system prompt setting unit 331, question receiving unit 332, or answer acquisition unit 333. In this case, the system prompt setting unit 331, question receiving unit 332, and answer acquisition unit 333 may be mounted on an information processing device different from the server 30, for example.

[0034] The system prompt setting unit 331 sets the system prompts. For example, the system prompt setting unit 331 sets the system prompts in response to instructions from the administrator terminal. The system prompt setting unit 331 sets the system prompt SP1 for answer C1, the system prompt SP2 for basis D1, and the system prompt SP3 for degree of reliance E1.

[0035] Figure 4 shows an example of a system prompt SP1. The system prompt SP1 includes a request form SP11, a question field SP12, and a specialized information field SP13.

[0036] Request statement SP11 is a statement requesting that answer C1 to question A1 be generated using specialized information B1. In Figure 4, request statement SP11 is a statement that reads, "Please output the answers to the following questions, referring to the following specialized information." Request statement SP11 is not limited to the statement shown in Figure 4, as long as it is a statement requesting that answer C1 to question A1 be generated using specialized information B1.

[0037] The question field SP12 is where question A1 is entered. In the system prompt SP1, the question field SP12 is blank.

[0038] The Specialized Information field SP13 is where Specialized Information B1 is written. In the System Prompt SP1, the Specialized Information field SP13 is blank.

[0039] Figure 5 shows an example of a system prompt SP2. The system prompt SP2 includes a request form SP21, a response field SP22, and a specialized information field SP23.

[0040] Request SP21 is a sentence that requests the extraction of at least one basis D1 for answer C1 from expert information B1. In Figure 5, request SP21 is the sentence: "The answer below was generated by LLM with reference to the expert information below. Please extract the sentences used as basis when generating this answer from the expert information. It is acceptable to have multiple sentences as basis." Request SP21 is not limited to the sentence shown in Figure 5, as long as it requests the extraction of at least one basis D1 from expert information B1.

[0041] The answer field SP22 is where answer C1 is entered. In the system prompt SP2, the answer field SP22 is blank.

[0042] The Specialized Information field SP23 is where Specialized Information B1 is written. In the System Prompt SP2, the Specialized Information field SP23 is blank.

[0043] Figure 6 shows an example of a system prompt SP3. The system prompt SP3 includes a request form SP31, a response field SP32, and a reason field SP33.

[0044] The request sentence SP31 is a sentence that requests generation of the reliance degree E1 based on the answer C1 and the plurality of grounds D1. In FIG. 6, the request sentence SP31 is a sentence stating "The following answer is generated based on the following grounds. Compare the answer with each ground, and evaluate how much each ground is relied on by the answer at the time of answer generation using a 5-level score from 1 to 5. Note that a score of 1 indicates a low reliance degree, and a score of 5 indicates a high reliance degree". The request sentence SP31 is not limited to the sentence shown in FIG. 6 as long as it is a sentence that requests generation of the reliance degree E1 based on the answer C1 and the plurality of grounds D1.

[0045] The answer column SP32 is a column in which the answer C1 is described. In the system prompt SP3, the answer column SP32 is a blank column.

[0046] The ground column SP33 is a column in which the plurality of grounds D1 are described. In the system prompt SP3, the ground column SP33 is a blank column.

[0047] In FIG. 3, the question reception unit 332 receives the question A1 from the terminal 10. The question reception unit 332 receives the question A1 from the terminal 10 via the communication device 31. Hereinafter, for simplification of description, descriptions related to the communication device 31 are omitted.

[0048] The answer acquisition unit 333 acquires the answer C1 for the question A1 from the LLM 21. The answer acquisition unit 333 first generates a prompt P1 related to the answer C1. The answer acquisition unit 333 generates the prompt P1 by describing the question A1 and the specialized information B1 in the system prompt SP1.

[0049] The answer acquisition unit 333 provides the prompt P1 to the LLM 21 by transmitting the prompt P1 to the LLM 21. The LLM 21 generates the answer C1 in response to the prompt P1. The answer acquisition unit 333 acquires the answer C1 from the LLM 21.

[0050] FIG. 7 is a diagram illustrating an example of the question A1. The question A1 is a question stating "Please tell me about Service A.". The question A1 is not limited to the question shown in FIG. 7 and can be modified as appropriate.

[0051] Figure 8 shows an example of Specialized Information B1. Specialized Information B1 includes Specialized Information B11, Specialized Information B12, and Specialized Information B13. Specialized Information B11 is the information that "The fee for Service A is 1,000 yen per month." Specialized Information B12 is the information that "The fee for Service B is 1,200 yen per month." Specialized Information B13 is the information that "Members of Service C can use Service A for free." Specialized Information B1 is not limited to the specialized information shown in Figure 8 and can be changed as appropriate.

[0052] Figure 9 shows an example of prompt P1 generated by writing question A1 and expert information B1 in system prompt SP1. In prompt P1, question A1 is written in the question field SP12 of system prompt SP1. Also, in prompt P1, expert information B1 is written in the expert information field SP13 of system prompt SP1. Prompt P1 is an example of an instruction to generate answer C1 to question A1 using expert information B1. Note that the instruction to generate answer C1 to question A1 using expert information B1 is not limited to prompt P1 shown in Figure 9 and can be changed as appropriate.

[0053] Figure 10 shows an example of answer C1. Answer C1 is, "Service A is a subscription service, but members of Service C can use it free of charge." Answer C1 is not limited to the answer shown in Figure 10 and can be changed as appropriate.

[0054] In Figure 3, the evidence acquisition unit 334 acquires at least one piece of evidence D1 for the answer C1. First, the evidence acquisition unit 334 generates a prompt P2 related to the piece of evidence D1. The evidence acquisition unit 334 generates prompt P2 by describing the answer C1 and the expert information B1 in the system prompt SP2.

[0055] The basis acquisition unit 334 provides prompt P2 to the LLM 21 by transmitting prompt P2 to the LLM 21. The LLM 21 generates at least one basis D1 in response to prompt P2. The basis acquisition unit 334 acquires at least one basis D1 from the LLM 21.

[0056] Figure 11 shows an example of prompt P2. In prompt P2, answer C1 is written in the answer field SP22 of system prompt SP2. Also, in prompt P2, expert information B1 is written in the expert information field SP23 of system prompt SP2. Prompt P2 is an example of an instruction to generate the basis for answer C1 using expert information B1. The instruction to generate the basis for answer C1 using expert information B1 is not limited to prompt P2 shown in Figure 11 and can be changed as appropriate.

[0057] Figure 12 shows an example of at least one basis D1. In Figure 12, at least one basis D1 is composed of multiple basis items D11 and D13. Basis D11 is identical to expert information B11. Basis D13 is identical to expert information B13. The example shown in Figure 12 is an example in which expert information B11 and expert information B13 are each determined as the basis for answer C1.

[0058] In Figure 3, the decision unit 335 presents the answer C1 and the basis D1 if at least one basis D1 is one basis D1. For example, if at least one basis D1 is one basis D1, the decision unit 335 transmits the answer C1 and the basis D1 to the terminal 10, causing the display device 12 of the terminal 10 to display the answer C1 and the basis D1. That is, if at least one basis D1 is one basis D1, the decision unit 335 presents the answer C1 and the basis D1 to the user U1 using the display device 12.

[0059] If at least one basis D1 is one of multiple basis D1s, the decision unit 335 determines the order in which the multiple basis D1s are presented based on the degree of reliance E1 of each basis D1 to the answer C1. The decision unit 335 obtains the degree of reliance E1 from, for example, LLM21.

[0060] The decision unit 335 sorts the multiple pieces of evidence D1 based on the degree of reliance E1. The decision unit 335 transmits the multiple pieces of evidence D1 sorted based on the degree of reliance E1 to the terminal 10, causing the multiple pieces of evidence D1 to be displayed on the display device 12 in the order in which they are presented based on the degree of reliance E1. In other words, the decision unit 335 uses the display device 12 to present the multiple pieces of evidence D1 to the user U1 in the order in which they are presented based on the degree of reliance E1.

[0061] For example, the decision unit 335 transmits multiple pieces of evidence D1 to the terminal 10 in order of decreasing reliance E1, thereby causing the display device 12 to display the multiple pieces of evidence D1 in order of decreasing reliance E1. In other words, the decision unit 335 presents multiple pieces of evidence D1 to the user U1 in order of decreasing reliance E1 by using the display device 12.

[0062] Figure 13 shows a decision unit 5, which is an example of a decision unit 335. The decision unit 5 obtains the degree of dependence E1 from the LLM 21. The decision unit 5 includes an instruction generation unit 51, a provision unit 52, and a presentation order determination unit 53.

[0063] The instruction generation unit 51 generates a prompt P3 related to the degree of dependence E1. The instruction generation unit 51 generates prompt P3 by describing the answer C1 and multiple pieces of evidence D1 in the system prompt SP3.

[0064] Figure 14 shows an example of prompt P3. In prompt P3, the answer C1 is written in the answer field SP32 of system prompt SP3. Also in prompt P3, multiple pieces of evidence D11 and D13 are written in the basis field SP33 of system prompt SP3. Prompt P3 is an example of an instruction to generate a degree of dependence E1 based on the answer C1 and the multiple pieces of evidence D1. The instruction to generate a degree of dependence E1 based on the answer C1 and the multiple pieces of evidence D1 is not limited to prompt P3 shown in Figure 14 and can be changed as appropriate.

[0065] The supply unit 52 provides prompt P3 to the LLM21 by transmitting prompt P3 to the LLM21. The LLM21 generates a degree of dependence E1 in response to prompt P3.

[0066] Figure 15 shows an example of the degree of dependence E1. The degree of dependence E1 shown in Figure 15 is expressed on a scale of 1 to 5. The higher the value of the degree of dependence E1, the higher the degree of dependence E1. In Figure 15, the degree of dependence E1 to answer C1 for evidence D11 is "3". The degree of dependence E1 to answer C1 for evidence D13 is "5". Therefore, the degree of dependence E1 for answer C1 for evidence D13 is higher than the degree of dependence E1 for answer C1 for evidence D11.

[0067] The presentation order determination unit 53 determines the presentation order of multiple pieces of evidence D1 based on the degree of reliance E1. For example, the presentation order determination unit 53 first obtains the degree of reliance E1 from the LLM 21. Based on the degree of reliance E1, the presentation order determination unit 53 determines the presentation order of pieces of evidence D11 and pieces of evidence D13. For example, before displaying piece of evidence D11 on the display device 12, the presentation order determination unit 53 displays piece of evidence D13, which has a higher degree of reliance E1 than piece of evidence D11, on the display device 12. In other words, the presentation order determination unit 53 presents multiple pieces of evidence D1 to the user U1 in the presentation order based on the degree of reliance E1 by using the display device 12. Furthermore, the presentation order determination unit 53 presents multiple pieces of evidence D1 to the user U1 in descending order of their degree of reliance E1 by using the display device 12.

[0068] 1-4. Operational Explanation Figure 16 is a flowchart illustrating an example of the operation of the system prompt setting unit 331. The operation shown in Figure 16 is performed before question A1 is received.

[0069] In step S101, the system prompt setting unit 331 sets a system prompt SP1 for answer C1, a system prompt SP2 for basis D1, and a system prompt SP3 for degree of reliance E1, in response to instructions from the administrator terminal.

[0070] Figure 17 is a flowchart illustrating an example of the operation of server 30. It is assumed that server 30 includes a determination unit 5, which obtains the degree of dependence E1 from LLM 21, as a determination unit 335.

[0071] In step S201, the question receiving unit 332 receives question A1 from terminal 10.

[0072] Next, in step S202, the answer acquisition unit 333 generates prompt P1 by writing question A1 and expert information B1 to system prompt SP1.

[0073] Next, in step S203, the response acquisition unit 333 provides prompt P1 to the LLM21 by transmitting prompt P1 to the LLM21. The LLM21 generates response C1 in response to prompt P1.

[0074] Next, in step S204, the response acquisition unit 333 acquires the response C1 from the LLM 21. The response acquisition unit 333 may further transmit the response C1 to the terminal 10.

[0075] Next, in step S205, the evidence acquisition unit 334 generates prompt P2 by writing the answer C1 and the specialized information B1 to the system prompt SP2.

[0076] Next, in step S206, the basis acquisition unit 334 provides prompt P2 to the LLM21 by transmitting prompt P2 to the LLM21. The LLM21 generates at least one basis D1 in response to prompt P2.

[0077] Next, in step S207, the evidence acquisition unit 334 acquires at least one evidence D1 from the LLM 21. For the sake of simplicity, in the following explanation, we will assume that the at least one evidence D1 is actually multiple evidence D1s.

[0078] Next, in step S208, the instruction generation unit 51 generates prompt P3 by writing the answer C1 and a plurality of grounds D1 to the system prompt SP3.

[0079] Next, in step S209, the supply unit 52 provides prompt P3 to the LLM21 by transmitting prompt P3 to the LLM21. The LLM21 generates a degree of dependence E1 in response to prompt P3.

[0080] Next, in step S210, the presentation order determination unit 53 obtains the degree of dependence E1 from the LLM21.

[0081] Next, in step S211, the presentation order determination unit 53 determines the presentation order of the multiple pieces of evidence D1 based on the degree of reliance E1. For example, the presentation order determination unit 53 sorts the multiple pieces of evidence D1 based on the degree of reliance E1. The presentation order determination unit 53 transmits the multiple pieces of evidence D1 sorted based on the degree of reliance E1 to the terminal 10, causing the multiple pieces of evidence D1 to be displayed on the display device 12 of the terminal 10 in the presentation order based on the degree of reliance E1. In other words, the presentation order determination unit 53 presents the multiple pieces of evidence D1 to the user U1 in the presentation order based on the degree of reliance E1 by using the display device 12.

[0082] For example, the presentation order determination unit 53 transmits multiple pieces of evidence D1 to the terminal 10 in order of decreasing dependence E1, thereby causing the multiple pieces of evidence D1 to be displayed on the display device 12 in order of decreasing dependence E1. In other words, the presentation order determination unit 53 uses the display device 12 to present multiple pieces of evidence D1 to the user U1 in order of decreasing dependence E1.

[0083] 1-5. Summary of Embodiments The server 30 includes a basis acquisition unit 334 and a decision unit 335. The basis acquisition unit 334 acquires at least one basis D1 for the answer C1 generated based on question A1 and expert information B1 related to question A1. If the at least one basis D1 is one of several basis D1s for the answer C1, the decision unit 335 determines the order in which the multiple basis D1s are presented based on the degree of reliance E1 of each basis D1 on the answer C1. This meets the need to review the basis for the answer in an order based on the degree of reliance on the answer.

[0084] The decision unit 335 presents multiple pieces of evidence D1 in order of decreasing reliance on them E1. This meets the need to review the evidence for the answer in order of decreasing reliance on it.

[0085] A decision unit 5, which is an example of a decision unit 335, includes a provision unit 52 and a presentation order determination unit 53. The provision unit 52 provides a prompt P3 to the LLM 21 if at least one basis D1 is one of several basis D1s. The prompt P3 is an example of an instruction to generate a degree of dependence E1 based on the answer C1 and the multiple basis D1s. The presentation order determination unit 53 determines the presentation order of the multiple basis D1s based on the degree of dependence E1 generated by the LLM 21 in response to the prompt P3. This allows the LLM 21 to generate the degree of dependence E1. Therefore, the decision unit 5 does not need to generate the degree of dependence E1. Consequently, the processing load performed by the decision unit 5 can be reduced.

[0086] The decision unit 5 further includes an instruction generation unit 51. The instruction generation unit 51 generates a prompt P3 based on the answer C1 and a plurality of grounds D1. The providing unit 52 provides the prompt P3 generated by the instruction generation unit 51 to the LLM 21. Therefore, a prompt P3 can be generated based on the answer C1 and a plurality of grounds D1.

[0087] Furthermore, if multiple evidence D1 includes two or more evidence D2 having the same degree of reliance E1, the decision unit 5 may present the two or more evidence D2 in an order based on a degree of reliance E2 that is more detailed than the degree of reliance E1. For example, if the degree of reliance E1 is expressed in five stages from 1 to 5, the degree of reliance E2 is expressed in 20 stages from 1 to 20. Therefore, even if two or more evidence D2 have the same degree of reliance E1, there is a high probability that they will have different degrees of reliance E2. Furthermore, the degree of reliance E2 does not have to be expressed in 20 stages from 1 to 20 if it is more detailed than the degree of reliance E1.

[0088] The decision unit 5 may use the LLM 21 to determine the presentation order based on the degree of reliance E2 of two or more pieces of evidence D2.

[0089] Figure 18 shows an example of a system prompt SP4 regarding the order in which two or more pieces of evidence D2 are presented. The system prompt SP4 includes a request form SP41, a response field SP42, and a evidence field SP43.

[0090] Request statement SP41 is a sentence that requests sorting of two or more pieces of evidence D2 based on the degree of reliance E2 of each piece of evidence D2 to the answer C1. In Figure 18, request statement SP41 is the sentence: "The answers below were generated based on the following evidence. Compare each piece of evidence with the answer and sort them in descending order of the degree of reliance at the time of answer creation." Request statement SP41 is not limited to the sentence shown in Figure 18, as long as it is a sentence that requests sorting of two or more pieces of evidence D2 based on the degree of reliance E2 of each piece of evidence D2 to the answer C1.

[0091] The answer field SP42 is where answer C1 is entered. In the system prompt SP4, the answer field SP42 is blank.

[0092] The "Reason" field SP43 is where two or more "Reason" D2 statements are entered. In the system prompt SP4, the "Reason" field SP43 is blank.

[0093] Figure 19 shows an example of answer C2, which is answer C1. Answer C2 is, "Members of Service A can use Service B at a 50% discount."

[0094] Figure 20 shows an example of two or more pieces of evidence D2. In Figure 20, two or more pieces of evidence D2 are composed of multiple pieces of evidence D21 and D22. Each of the multiple pieces of evidence D21 and D22 is the basis for answer C2. Evidence D21 is specialized information stating, "Members of Service A can use Service B at half price." Evidence D22 is specialized information stating, "Members of Service A can use Service B at a 50% discount."

[0095] The instruction generation unit 51 generates prompt P4 by writing the answer C2 and multiple justifications D21 and D22 to the system prompt SP4.

[0096] Figure 21 shows an example of prompt P4. In prompt P4, the answer C2 is written in the answer column SP42 of system prompt SP4. Also in prompt P4, multiple justifications D21 and D22 are written in the justification column SP43 of system prompt SP4. Prompt P4 is an example of an instruction to sort two or more justifications D2 based on the degree of reliance E2 of each of the two or more justifications D2 to the answer C1. The instruction to sort two or more justifications D2 based on the degree of reliance E2 is not limited to prompt P4 shown in Figure 21 and can be changed as appropriate.

[0097] The providing unit 52 provides prompt P4 to the LLM21 by sending prompt P4 to the LLM21. In response to prompt P4, the LLM21 sorts two or more evidence D2 based on the degree of dependence E2.

[0098] Figure 22 shows an example of the sorting result F2 for two or more pieces of evidence D2. In Figure 22, piece of evidence D22 is located before piece of evidence D21. That is, because the degree of reliance E2 of piece of evidence D22 on answer C2 is higher than the degree of reliance E2 of piece of evidence D21 on answer C2, piece of evidence D22 is located before piece of evidence D21.

[0099] The presentation order determination unit 53 determines the presentation order of two or more pieces of evidence D2 based on the sort result F2 of the two or more pieces of evidence D2. For example, the presentation order determination unit 53 first obtains the sort result F2 from the LLM 21. The presentation order determination unit 53 determines the order of the two or more pieces of evidence D2 shown in the sort result F2 as the presentation order of the two or more pieces of evidence D2. For example, the presentation order determination unit 53 displays piece of evidence D22 on the display device 12 before displaying piece of evidence D21 on the display device 12. That is, the presentation order determination unit 53 presents two or more pieces of evidence D2 to the user U1 in the presentation order based on the sort result F2 by using the display device 12.

[0100] Figure 23 shows another example of answer C1, which is answer C3. Answer C3 is, "Customers who wish to cancel service A should do so by the end of the fiscal year."

[0101] Figure 24 shows another example of two or more grounds D2. In Figure 24, two or more grounds D2 are multiple grounds D31 and D32. Each of the multiple grounds D31 and D32 is the basis for answer C3. Ground D31 is specialized information stating, "The cancellation deadline for Service A is the end of March." Ground D32 is specialized information stating, "The cancellation deadline for our services is uniformly the end of the fiscal year."

[0102] The instruction generation unit 51 generates prompt P5 by writing the answer C3 and multiple justifications D31 and D32 to the system prompt SP4.

[0103] Figure 25 shows an example of prompt P5. In prompt P5, the answer C3 is written in the answer column SP42 of system prompt SP4. Also in prompt P5, multiple justifications D31 and D32 are written in the justification column SP43 of system prompt SP4. Prompt P5 is another example of an instruction to sort two or more justifications D2 based on the degree of reliance E2 of each justification D2 to the answer C1.

[0104] The providing unit 52 provides prompt P5 to the LLM21 by sending prompt P5 to the LLM21. In response to prompt P5, the LLM21 sorts two or more evidence D2 based on the degree of dependence E2.

[0105] Figure 26 shows an example of the sorting result F3 of multiple pieces of evidence D31 and D32. In Figure 26, piece of evidence D31 is positioned before piece of evidence D32. That is, piece of evidence D31 is positioned before piece of evidence D32 because its degree of reliance E2 on answer C3 is higher than that of piece of evidence D32.

[0106] The presentation order determination unit 53 determines the presentation order of the multiple pieces of evidence D31 and D32 based on the sort result F3 of the multiple pieces of evidence D31 and D32. For example, the presentation order determination unit 53 first obtains the sort result F3 from the LLM 21. The presentation order determination unit 53 determines the order of the multiple pieces of evidence D31 and D32 shown in the sort result F3 as the presentation order of the multiple pieces of evidence D31 and D32. For example, the presentation order determination unit 53 displays piece of evidence D31 on the display device 12 before displaying piece of evidence D32 on the display device 12. That is, the presentation order determination unit 53 presents the multiple pieces of evidence D31 and D32 to the user U1 in the presentation order based on the sort result F3 by using the display device 12.

[0107] 2. Modifications The following are examples of modifications to the above-described embodiment. Two or more modifications can be arbitrarily selected from the following examples and combined as appropriate, within the bounds of mutual non-contradictory relationships.

[0108] 2-1. First Modification In the above embodiment, a determination unit 5 that obtains the degree of dependence E1 from the LLM21 was shown as an example of the determination unit 335. However, the determination unit 335 may calculate the degree of dependence E1 without using the LLM21.

[0109] Figure 27 shows a decision unit 6, which is another example of the decision unit 335. The decision unit 6 calculates the degree of reliance E1. The decision unit 6 includes a calculation unit 61 and a presentation order determination unit 62.

[0110] The calculation unit 61 calculates the degree of reliance E1 based on the answer C1 and the multiple pieces of evidence D1. For example, the calculation unit 61 calculates the degree of reliance E1 as the vector similarity between the answer C1 and each of the multiple pieces of evidence D1.

[0111] The presentation order determination unit 62 determines the presentation order of the multiple pieces of evidence D1 based on the degree of reliance E1 calculated by the calculation unit 61. For example, the presentation order determination unit 62 sorts the multiple pieces of evidence D1 based on the degree of reliance E1 calculated by the calculation unit 61. The presentation order determination unit 62 transmits the multiple pieces of evidence D1 sorted based on the degree of reliance E1 calculated by the calculation unit 61 to the terminal 10, thereby causing the multiple pieces of evidence D1 to be displayed on the display device 12 of the terminal 10 in the presentation order based on the degree of reliance E1 calculated by the calculation unit 61. In other words, the presentation order determination unit 62 presents the multiple pieces of evidence D1 to the user U1 in the presentation order based on the degree of reliance E1 calculated by the calculation unit 61 by using the display device 12.

[0112] For example, the presentation order determination unit 62 transmits a plurality of evidence D1 to the terminal 10 in order of the degree of dependence E1 calculated by the calculation unit 61, thereby causing the plurality of evidence D1 to be displayed on the display device 12 in order of the degree of dependence E1 calculated by the calculation unit 61. In other words, the presentation order determination unit 62 uses the display device 12 to present a plurality of evidence D1 to the user U1 in order of the degree of dependence E1 calculated by the calculation unit 61.

[0113] According to the first modified example, the presentation order determination unit 62 determines the presentation order of the multiple pieces of evidence D1 based on the degree of reliance E1 calculated by the calculation unit 61. Therefore, the degree of reliance E1 can be calculated without using the LLM 21.

[0114] 2-2. Second Modification The degree of dependence E1 does not have to be expressed on a 5-point scale from 1 to 5. For example, the degree of dependence E1 may be expressed by a numerical value within the range of 0 to 1. "0" represents a low degree of dependence. "1" represents a high degree of dependence. In this case, one example of a degree of dependence E1 is "0.4". Another example of a degree of dependence E1 is "0.8".

[0115] 2-3. Each of the third modification determination unit 335, determination unit 5, and determination unit 6 may transmit a plurality of evidence D1 in order of increasing reliance E1 to the terminal 10, thereby displaying the plurality of evidence D1 in order of increasing reliance E1 on the display device 12. That is, each of the determination unit 335, determination unit 5, and determination unit 6 presents a plurality of evidence D1 to the user U1 in order of increasing reliance E1 using the display device 12.

[0116] Evidence D1 with a low degree of reliance E1 may contain ambiguous specialized information. Staff responsible for specialized information B1 may want to revise specialized information containing ambiguity. Therefore, when user U1 is staff responsible for specialized information B1, and multiple pieces of evidence D1 are presented to user U1 in order of increasing degree of reliance E1, the staff will be more likely to identify specialized information containing ambiguity. As a result, the staff will be able to revise specialized information containing ambiguity.

[0117] 2-4. Fourth Modified Example At least one basis D1 obtained by each of the decision units 335, 5, and 6 does not have to be generated by the LLM21. For example, each of the decision units 335, 5, and 6 may receive at least one basis D1 from the terminal 10. Also, the answer C1 obtained by each of the decision units 335, 5, and 6 does not have to be generated by the LLM21. For example, each of the decision units 335, 5, and 6 may receive the answer C1 from the terminal 10. In the fourth modified example as well, the need to check the basis for the answer in an order based on the degree of reliance on the answer can be met.

[0118] 3. Other (1) Each function illustrated in Figures 2, 3, 13, or 27 can be implemented by any combination of hardware and software. The method of implementing each function is not particularly limited. Each function may be implemented using one physically or logically coupled device, or it may be implemented using a device configured by directly or indirectly connecting two or more physically or logically separated devices (for example, using wired or wireless connections). Each function may be implemented by combining the above one device or the above multiple devices with software.

[0119] (2) In this disclosure, the term “apparatus” may be replaced with other terms such as circuit, device or unit.

[0120] (3) In each of the embodiments and the first to fourth modifications, each of the storage devices 14 and 32 may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., compact disc, digital multipurpose disc, Blu-ray® disc), a smart card, flash memory (e.g., card, stick, key drive), a floppy® disk, a magnetic strip, etc. The program may also be transmitted from a network via a telecommunications line.

[0121] (4) Each of the embodiments and the first to fourth modified examples is LTE (Long Term Evolution), LTE-A (LTA-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (where x is, for example, an integer or decimal), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20 may apply to at least one system utilizing UWB (Ultra-WideBand), Bluetooth®, or other appropriate systems, and to next-generation systems extended, modified, created, or defined based thereon. Alternatively, multiple systems may be applied in combination (e.g., a combination of at least one of LTE and LTE-A with 5G).

[0122] (5) The processing procedures, sequences, or flowcharts illustrated in each of the embodiments and the first to fourth modifications may be in any order, as long as they do not contradict each other. For example, the methods described in this disclosure present various step elements in an illustrative order and are not limited to any particular order presented.

[0123] (6) In each of the embodiments and the first to fourth modifications, the input and output information may be stored in a specific location (e.g., memory) or managed using a management table. The input and output information may be overwritten, updated, or appended to. The output information may be deleted. The input information may be transmitted to other devices.

[0124] (7) In each of the embodiments and the first to fourth modifications, the determination may be based on a value represented by one bit (0 or 1), on a Boolean value (true or false), or on a numerical comparison (for example, a comparison with a predetermined value).

[0125] (8) The programs illustrated in each of the embodiments and the first to fourth modifications should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, or functions, whether they are called software, firmware, middleware, microcode, or hardware description languages ​​or by other names. Furthermore, software, or instructions, etc., may be transmitted or received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, fiber optic cable, twisted pair, and digital subscriber line (DSL)) and wireless technology (such as infrared, microwave, etc.), at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0126] (9) The information described in each of the embodiments and the first to fourth modifications may be represented using any of the following different technologies. For example, the data and information that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. Notwithstanding the terms described herein and the terms necessary for understanding this disclosure, terms may be replaced with terms having the same or similar meanings.

[0127] (10) In each of the embodiments and the first to fourth modifications, the terms “system” and “network” are used interchangeably.

[0128] (11) In each of the embodiments and the first to fourth modifications, the server 30 may be, for example, a mobile station. A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or several other appropriate terms.

[0129] (12) A mobile station may also be called a transmitting device, receiving device, or communication device. A mobile station may also be a device mounted on a mobile body, or the mobile body itself. A mobile body means a movable object. The speed of movement of a mobile body is arbitrary. A mobile body can be stopped. A mobile body includes, but is not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcarts, rickshaws, ships and other watercraft, airplanes, rockets, satellites, drones (registered trademark), multicopters, quadcopters, balloons, and things mounted on them. A mobile body may be a mobile body that moves autonomously based on operational commands. A mobile body may be a vehicle (e.g., a car, an airplane, etc.), an unmanned mobile body (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). A mobile station also includes devices that do not necessarily move during communication operations. For example, the mobile station could be an IoT (Internet of Things) device such as a sensor.

[0130] (13) In each of the embodiments and the first to fourth modifications, the term “decision” may encompass a wide variety of actions. “Decision” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiry (e.g., searching in a table, database or another data structure), ascertaining, etc. “Decision” may also include, for example, receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), etc. “Decision” may also include, for example, resolving, selecting, choosing, establishing, comparing, etc. In other words, "decision" can include considering that some action has been "decided." Furthermore, "decision" can be reinterpreted as "assuming," "expecting," or "considering."

[0131] (14) In each of the embodiments and the first to fourth modifications, the term “connected,” or any variation thereof, means any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, “connection” may be read as “access.” As used in the present disclosure, two elements may be considered to be “connected” or “coupled” to each other using at least one of one or more wires, cables, and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain, and optical (both visible and invisible) domain.

[0132] (15) In each of the first embodiment and the first to fourth modifications, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on".

[0133] (16) Any reference to elements using the designations “first” and “second” as used in this disclosure does not limit the quantity or order of those elements in general. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Accordingly, references to the first and second elements do not imply that only two elements may be adopted or that the first element must precede the second element in any way.

[0134] (17) Where “include,” “including,” and variations thereof are used in each of the embodiments and the first to fourth modifications in this disclosure or in the claims, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, where the term “or” is used in this disclosure or in the claims, it is intended not to be an exclusive OR.

[0135] (18) Where articles are added in the translation, for example, a, an, and the in English, the disclosure may include the fact that the noun following these articles is plural.

[0136] (19) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a given value, or other corresponding information.

[0137] (20) In this disclosure, the term “A and B are different” may mean “A and B are different from each other.” The term may also mean “A and B are each different from C.” Terms such as “separate” and “combined” may be interpreted in the same way as “different.”

[0138] (21) Each aspect / embodiment described herein may be used individually, in combination, or switched between as needed during implementation. Furthermore, notification of certain information (e.g., notification that "it is X") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0139] (22) It will be obvious to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the invention as defined by the claims. Accordingly, the descriptions herein are for illustrative purposes only and are not intended to be restrictive in any way to the present invention. Furthermore, multiple embodiments selected from those illustrated herein may be combined.

[0140] 4. Aspects that can be understood from the above-described forms or modifications The following aspects can be understood from at least one of the above-described forms or modifications.

[0141] 4-1. First Embodiment The presentation order determination device according to the first embodiment includes a basis acquisition unit and a determination unit. The basis acquisition unit acquires at least one basis for an answer generated based on a question and information related to the question. If the at least one basis is a plurality of basis for the answer, the determination unit determines the presentation order of the plurality of basis based on the degree to which each of the plurality of basis relies on the answer.

[0142] According to this embodiment, if the at least one basis is one of several reasons for the answer, the decision unit determines the order in which the multiple reasons are presented based on the degree to which each of the multiple reasons is relied upon for the answer. This meets the need to review the reasons for the answer in an order based on the degree of reliance on the answer.

[0143] 4-2. Second Embodiment In the example of the first embodiment (second embodiment), the decision unit presents the multiple pieces of evidence in order of increasing reliance on them. This embodiment can meet the need to review the evidence for the answer in order of increasing reliance on it.

[0144] 4-3. Third Embodiment In the example of the first embodiment (third embodiment), the determination unit includes a provision unit and a presentation order determination unit. The provision unit provides the large-scale language model with instructions to generate the degree of dependence based on the answer and the multiple grounds, if the at least one ground is the multiple grounds. The presentation order determination unit determines the presentation order of the multiple grounds based on the degree of dependence generated by the large-scale language model in response to the instructions. According to this embodiment, the large-scale language model can be made to generate the degree of dependence. Therefore, the determination unit does not need to generate the degree of dependence. Consequently, the processing load performed by the determination unit can be reduced.

[0145] 4-4. Fourth Embodiment In the example of the third embodiment (fourth embodiment), the decision unit further includes an instruction generation unit that generates the instruction based on the answer and the plurality of grounds. The providing unit provides the instruction generated by the instruction generation unit to the large-scale language model. According to this embodiment, the above-mentioned instruction can be generated based on the answer and the plurality of grounds.

[0146] 4-5. Fifth Embodiment In the example of the first embodiment (fifth embodiment), the determination unit includes a calculation unit that calculates the degree of reliance based on the answer and the plurality of grounds, and a presentation order determination unit that determines the presentation order of the plurality of grounds based on the degree of reliance calculated by the calculation unit. According to this embodiment, the presentation order determination unit determines the presentation order of the plurality of grounds based on the degree of reliance calculated by the calculation unit. For this reason, the degree of reliance can be calculated without using a large-scale language model.

[0147] 4-6. Sixth Embodiment The method for determining the order of presentation according to the sixth embodiment includes obtaining at least one basis for an answer generated based on a question and information related to the question, and, if the at least one basis is a plurality of basis for the answer, determining the order of presentation of the plurality of basis based on the degree to which each of the plurality of basis relies on the answer. According to this embodiment, the same effects as the first embodiment can be achieved.

[0148] 10...Terminal, 11...Input device, 12...Display device, 13...Communication device, 14...Storage device, 15...Processing device, 151...Operation control unit, 16...Bus, 20...LLM system, 30...Server, 31...Communication device, 32...Storage device, 33...Processing device, 331...System prompt setting unit, 332...Question receiving unit, 333...Answer acquisition unit, 334...Evidence acquisition unit, 335...Decision unit, 344...Evidence acquisition unit, 34...Bus, 5...Decision unit, 51...Instruction generation unit, 52...Provision unit, 53...Presentation order determination unit, 6...Decision unit, 61...Calculation unit, 62...Presentation order determination unit.

Claims

1. A presentation order determination device comprising: a basis acquisition unit that acquires at least one basis for an answer generated based on a question and information related to the question; and, if the at least one basis is a plurality of basis for the answer, a determination unit that determines the order in which the plurality of basis are presented based on the degree of reliance of each of the plurality of basis for the answer.

2. The presentation order determination device according to claim 1, wherein the determination unit presents the plurality of grounds in order of decreasing reliance.

3. The presentation order determination device according to claim 1, comprising: a provisioning unit that provides a large-scale language model with instructions to generate the degree of dependence based on the answer and the plurality of premises when the at least one basis is the plurality of premises; and a presentation order determination unit that determines the presentation order of the plurality of premises based on the degree of dependence generated by the large-scale language model in response to the instructions.

4. The presentation order determination device according to claim 3, wherein the determination unit further includes an instruction generation unit that generates the instruction based on the answer and the plurality of grounds, and the providing unit provides the instruction generated by the instruction generation unit to the large-scale language model.

5. The presentation order determination device according to claim 1, wherein the determination unit includes a calculation unit that calculates the degree of reliance based on the answer and the plurality of grounds, and a presentation order determination unit that determines the presentation order of the plurality of grounds based on the degree of reliance calculated by the calculation unit.

6. A method for determining the order of presentation, comprising: obtaining at least one piece of evidence for an answer generated based on a question and information related to the question; and, if the at least one piece of evidence is multiple pieces of evidence for the answer, determining the order in which the multiple pieces of evidence are presented based on the degree to which each of the multiple pieces of evidence relies on the answer.