Processing device, processing method, and processing program

Abstract

A processing device (10) comprises: a question extraction unit (11) that, using an LLM (20), extracts, from a plurality of dialogues between an expert and a user, questions asked by the expert in order to elicit information from the user; a proposal extraction unit (12) that, using the LLM (20), extracts, from the plurality of dialogues, proposals made by the expert in order to solve the problem of the user; a structuring unit (13) that, using the LLM (20), converts a dialogue between the expert and the user into a flow having information indicating a question, the user's answer to the question, and the proposal accepted by the user; and a chart construction unit (14) that statistically constructs, on the basis of a plurality of the flows, one flowchart indicating the tacit knowledge and judgment of the expert.

Description

Processing Device, Processing Method, and Processing Program

[0001] The present invention relates to a processing device, a processing method, and a processing program.

[0002] In interactions between experts and general users, such as security incident handling and customer support, experts ask appropriate questions to general users based on their knowledge. Then, experts extract and organize information from the answers given by users and guide them towards problem-solving.

[0003] Such expert judgment is usually complex and confined within the expert's mind, becoming tacit knowledge, and there is a problem that it is not easy to share the expert's findings.

[0004] If such tacit knowledge of experts can be visualized, a wide range of utilization methods can be considered, such as new employee training and the development of automatic response bots.

[0005] Conventionally, a method has been proposed that uses a finite state automaton to structure and visualize the states of a large number of past conversations and utilize them for automatic response by a large language model (LLM: Large Language Models) (Non-Patent Document 1). In the method described in Non-Patent Document 1, a finite state automaton learned from training conversations is incorporated into the LLM to generate regular and compliant responses.

[0006] Yiyou Sun, Junjie Hu, Wei Cheng, Haifeng Chen, “DFA-RAG: Conversational Semantic Router for Large Language Model with Definite Finite Automaton”, 2024, [Retrieved on September 17, 2024], Internet <URL: https: / / arxiv.org / abs / 2402.04411>

[0007] However, visualizing the tacit knowledge of experts using a finite state automaton has the problem that there are extremely many branches and the interpretability is low.

[0008] The present invention has been made in view of the above, and aims to provide a processing device, a processing method, and a processing program that visualize the tacit knowledge and judgment of experts as a flowchart.

[0009] To solve the above-mentioned problems and achieve the objective, the processing apparatus according to the present invention is characterized by comprising: a first extraction unit that uses a large-scale language model to extract questions asked by an expert to elicit information from a user from a plurality of dialogues between the expert and the user; a second extraction unit that uses a large-scale language model to extract proposals made by the expert to solve the user's problem from the plurality of dialogues; a structuring unit that uses a large-scale language model to convert the dialogue between the expert and the user into a structure having information indicating the questions, the user's answers to the questions, and the proposals accepted by the user; and a construction unit that statistically constructs a flowchart showing the expert's tacit knowledge and judgment based on a plurality of the structures.

[0010] Furthermore, the processing method according to the present invention is a processing method executed by a processing device, and is characterized by including the steps of: using a large-scale language model to extract questions asked by an expert to elicit information from a user from a plurality of conversations between the expert and the user; using a large-scale language model to extract suggestions made by the expert to solve the user's problem from the plurality of conversations; using a large-scale language model to convert the conversations between the expert and the user into a structure having information indicating the questions, the user's answers to the questions, and the suggestions accepted by the user; and statistically constructing a flowchart showing the expert's tacit knowledge and judgment based on a plurality of the structures.

[0011] Furthermore, the processing program according to the present invention causes a computer to perform the following steps: using a large-scale language model to extract questions asked by an expert to elicit information from a user from multiple conversations between the expert and the user; using a large-scale language model to extract suggestions made by the expert to solve the user's problem from the multiple conversations; using a large-scale language model to convert the conversations between the expert and the user into a structure having information indicating the questions, the user's answers to the questions, and the suggestions accepted by the user; and statistically constructing a flowchart showing the expert's tacit knowledge and judgment based on multiple structures.

[0012] According to the present invention, it is possible to visualize the tacit knowledge and judgment of experts as a flowchart.

[0013] Figure 1 is a diagram showing an example of the configuration of a processing device according to an embodiment. Figure 2 is a diagram illustrating the processing of the question extraction unit shown in Figure 1. Figure 3 is a diagram illustrating the processing of the suggestion extraction unit shown in Figure 1. Figure 4 is a diagram illustrating the processing of the structuring unit shown in Figure 1. Figure 5 is a diagram illustrating the processing of the chart construction unit shown in Figure 1. Figure 6 is a flowchart showing the processing procedure of the processing method according to an embodiment. Figure 7 is a diagram showing an example of a flowchart included in the Flo-dial dataset. Figure 8 is a diagram showing an example of a dialogue linked to a flowchart included in the Flo-dial dataset. Figure 9 is a diagram showing an example of a question list extracted from a dialogue example linked to a flowchart included in the Flo-dial dataset. Figure 10 is a diagram showing an example of a suggestion list extracted from a dialogue example linked to a flowchart included in the Flo-dial dataset. Figure 11 is a diagram showing an example of a flow structured by the processing device. Figure 12 is a diagram showing an example of a flowchart generated by the processing device. Figure 13 is a diagram illustrating the processing content of the embodiment. Figure 14 is a diagram showing an example of a computer in which the processing device is realized when a program is executed.

[0014] Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to this embodiment. Furthermore, in the drawings, the same parts are denoted by the same reference numerals.

[0015] [Embodiment] In this embodiment, we propose a method for analyzing past conversations using LLM and visualizing the tacit knowledge and judgments of experts as a flowchart.

[0016] [Processing apparatus] The processing apparatus according to the embodiment will now be described. Figure 1 is a diagram showing an example of the configuration of the processing apparatus according to the embodiment.

[0017] The processing unit 10 according to this embodiment is a generation device that is realized by loading a predetermined program into a computer or the like, which includes ROM (Read Only Memory), RAM (Random Access Memory), CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The processing unit 10 also has a communication interface for sending and receiving various information with other devices connected via a network or the like.

[0018] The processing unit 10 includes a question extraction unit 11 (first extraction unit), a suggestion extraction unit 12 (second extraction unit), a structuring unit 13, a chart construction unit 14 (construction unit), and an LLM 20. The processing unit 10 takes multiple conversations between an expert and a user as input and outputs a flowchart that shows the expert's tacit knowledge and judgment.

[0019] The LLM20 processes the input text data according to the set prompts, generates text data, and outputs it. The input to the LLM20 may also be audio data.

[0020] The question extraction unit 11 uses the LLM 20 to extract questions asked by experts to elicit information from users from multiple conversations between experts and users. The question extraction unit 11 uses the LLM 20 to extract questions from multiple conversations between experts and users from the same perspective, absorbing paraphrasing and variations in notation of questions with the same meaning. The question extraction unit 11 uses either the first or second method to extract questions from all past conversations to be learned from a unified perspective, absorbing paraphrasing and variations in notation of questions with the same meaning.

[0021] The first method will be described. The question extraction unit 11 includes all of the dialogue in a prompt that instructs the LLM 20 to extract questions and list them.

[0022] A second method will now be described. Figure 2 is a diagram illustrating the processing of the question extraction unit 11 shown in Figure 1. In the second method, the question extraction unit 11 performs a first process in which it instructs the LLM 20 to extract a question list from each dialogue, using prompts that instruct it to extract questions from the dialogue and list the extracted questions. For example, the question extraction unit 11 instructs the LLM 20 to extract questions from dialogue D1 and generate a question list L11 of the extracted questions. The question extraction unit 11 performs this first process for the number of dialogues. Subsequently, the question extraction unit 11 inputs all the question lists into the LLM 20 and generates an integrated question list L10 by integrating questions of the same perspective for each question list. The question extraction unit 11 outputs the generated question list to the structuring unit 13.

[0023] The proposal extraction unit 12 uses the LLM 20 to extract proposals made by experts to solve users' problems from multiple conversations between experts and users. The proposal extraction unit 12 uses the LLM 20 to extract proposals from multiple conversations between experts and users from the same perspective, absorbing paraphrasing and variations in notation of questions with the same meaning. Similar to the question extraction unit 11, the proposal extraction unit 12 uses either the first or second method to extract proposals from all past conversations to be learned from a unified perspective, absorbing paraphrasing and variations in notation of questions with the same meaning.

[0024] The first method will be described. The suggestion extraction unit 12 includes all dialogue in a prompt that instructs the LLM 20 to extract suggestions from the dialogue and lists them.

[0025] A second method will now be described. Figure 3 is a diagram illustrating the processing of the proposal extraction unit 12 shown in Figure 1. In the second method, the proposal extraction unit 12 performs a first process instructing the LLM 20 to extract a proposal list from each dialogue, using prompts that instruct the LLM 20 to extract proposals from the dialogue and list the extracted proposals. For example, the proposal extraction unit 12 instructs the LLM 20 to extract proposals from dialogue D1 and generate a proposal list L21 of the extracted proposals. The proposal extraction unit 12 performs this first process for each dialogue. Subsequently, the proposal extraction unit 12 inputs all proposal lists into the LLM 20 and generates an integrated proposal list L20 by integrating proposals of the same perspective for each proposal list. The proposal extraction unit 12 outputs the generated proposal list to the structuring unit 13.

[0026] The structuring unit 13 uses the LLM 20 to convert the dialogue between the expert and the user into a structure containing information such as a question, the user's answer to the question, and the expert's proposal accepted by the user. Hereafter, the structure will be referred to as a flow. The structuring unit 13 outputs each flow to the chart construction unit 14.

[0027] The structuring unit 13 uses the LLM 20 to structure the dialogue into a flow of questions, answers, and suggestions. For each dialogue, the structuring unit 13 refers to the question list and the suggestion list to structure the flow (i.e., the sequence of questions and answers) that was followed. For each dialogue, the structuring unit 13 refers to the question list and the suggestion list to generate a flow that has a question, the user's answer to that question, and a suggestion for the answer.

[0028] Figure 4 is a diagram illustrating the processing of the structuring unit 13 shown in Figure 1. The structuring unit 13 uses the question list and suggestion list extracted by the question extraction unit 11 and suggestion extraction unit 12 to structure how questions were asked, what answers were obtained from the user, and what suggestions were made in an actual dialogue. The objects structured in this way correspond to the flow. The flow contains information about the questions asked, the answers to those questions, and the suggestions made, as shown in Figure 4, "Question 1, Yes, Question 3, No, Suggestion 4".

[0029] To perform this structuring, the structuring unit 13 provides the dialogue and question list to the LLM 20, tags the questions, and instructs the LLM 20 to perform the process of associating each question in the question list with the actual utterances in the dialogue. Tags may include, for example, an identification number or the word that appears most frequently in the questions. The structuring unit 13 also provides the dialogue and suggestion list to the LLM 20, tags the suggestions, and instructs the LLM 20 to perform the process of associating each suggestion in the suggestion list with the actual utterances in the dialogue.

[0030] The structuring unit 13 then extracts the user's answers to each question from the dialogue associated with each question using tags. At this time, the structuring unit 13 extracts the user's answers in a concise form such as "yes," "no," or "I don't know." Based on the questions asked by the expert to elicit information from the user, the user's answers to these questions, and the expert's suggestions accepted by the user, the structuring unit 13 generates a flow that shows the flow of questions asked in the actual dialogue, the answers obtained from the user, and the suggestions made by the expert.

[0031] The chart construction unit 14 statistically constructs a single flowchart that represents the tacit knowledge and judgment of an expert, based on multiple structures (flows) converted by the structuring unit. The chart construction unit 14 statistically analyzes the multiple flows structured by the structuring unit 13 and constructs a flowchart that represents the tacit knowledge and judgment of an expert based on the analysis results.

[0032] The chart construction unit 14 counts the number of occurrences of a 1-step transition from each question and / or suggestion to the next question and / or suggestion, processes the count statistically, and converts it into a tree structure (flowchart).

[0033] Figure 5 is a diagram illustrating the processing of the chart construction unit 14 shown in Figure 1. The chart construction unit 14 decomposes a flow (for example, flow F1) into a triplet structure consisting of a first question, an answer to the first question, and the next second question or suggestion. For example, for flow F1 which is "Question 1, Yes, Question 3, No, Suggestion 4", it is decomposed into "(Question 1, Yes, Question 3)" and "(Question 3, No, Suggestion 4)" as shown in structure T1.

[0034] The chart construction unit 14 counts the number of occurrences of each triplet structure for all flows (i.e., all interactions) (Figure 5(1)). Next, the chart construction unit 14 generates integrated data M1 by integrating the counted occurrences of each triplet structure (Figure 5(2)). The chart construction unit 14 constructs a tree structure sequentially from the triplet structures with the highest counts in the integrated data M1 to construct a flowchart Ct. The chart construction unit 14 recursively processes the construction of the tree structure based on the triplet structures and converts multiple flows into flowcharts (Figure 5(3)).

[0035] [Processing Method] Next, the processing method according to the embodiment will be described. Figure 6 is a flowchart showing the processing procedure of the processing method according to the embodiment.

[0036] First, when the processing unit 10 receives input of multiple dialogues between an expert and a user (step S11), it uses the LLM 20 to perform a question extraction process to extract questions asked by the expert to elicit information from the user from the input dialogues (step S12). The processing unit 10 then uses the LLM 20 to perform a proposal extraction process to extract suggestions made by the expert to solve the user's problem from the input dialogues (step S13). Steps S12 and S13 may be performed in series. In this case, the order in which steps S12 and S13 are performed does not matter.

[0037] The processing unit 10 uses the LLM 20 to perform structured processing, which converts the dialogue between the expert and the user into a flow containing information indicating questions, the user's answers to the questions, and the suggestions accepted by the user (step S14).

[0038] The processing unit 10 performs a chart construction process that statistically constructs a single flowchart representing the tacit knowledge and judgment of an expert, based on the multiple flows converted in the structured processing (step S15). The processing unit 10 outputs the constructed flowchart (step S16).

[0039] [Experiment] A simple experiment was conducted to evaluate the processing method according to the embodiment. This experiment utilized the Flo-dial dataset (Reference 1). Reference 1: IBM, “Flo-dial”, [Retrieved September 5, 2024], Internet <URL: https: / / dair-iitd.github.io / FloDial / >

[0040] The Flo-dial dataset is a dataset that links flowcharts with operator dialogues that respond based on those flowcharts. The Flo-dial dataset was originally created to evaluate the performance of language models that respond based on flowcharts. The Flo-dial dataset has 12 types of flowcharts and approximately 200 dialogues associated with each flowchart.

[0041] In this experiment, the Flo-dial dataset is used for the task of inferring a flowchart from a dialogue, which is the reverse of the original task. FIG. 7 is a diagram showing an example of a flowchart included in the Flo-dial dataset. In FIG. 7, the specific content of the questions and proposals shown in the flowchart is omitted, and only the titles representing the questions and proposals are displayed.

[0042] FIG. 8 is a diagram showing an example of a dialogue example associated with the flowchart included in the Flo-dial dataset. For example, the dialogue shown in FIG. 8 serves as the input to the processing device 10. In this experiment, the processing device 10 performs the aforementioned question extraction process (step S12), proposal extraction process (step S13), structuring process (step S14), and chart construction process (step S15) on the dialogue to generate a flowchart.

[0043] FIG. 9 is a diagram showing an example of a question list extracted from a dialogue example associated with the flowchart included in the Flo-dial dataset. As illustrated in FIG. 9, in the question extraction process, 33 questions from the same perspective were extracted from the dialogue example. FIG. 10 is a diagram showing an example of a proposal list extracted from a dialogue example associated with the flowchart included in the Flo-dial dataset. As illustrated in FIG. 10, in the proposal extraction process, 37 proposals from the same perspective were extracted from the dialogue example.

[0044] FIG. 11 is a diagram showing an example of a flow structured by the processing device 10. In the structuring process, the processing device 10 extracted a flow (the lower diagram in FIG. 11) from an arbitrary dialogue in the Flo-dial dataset (the upper diagram in FIG. 11). The processing device 10 used the question list (for example, see FIG. 9) and proposal list (for example, see FIG. 10) extracted from the same perspective from the dialogue examples associated with the Flo-dial dataset. As illustrated in FIG. 11, in the experiment, it can be seen that the flow of the questions asked, the answers obtained, and the proposals made, which reflect the content of the dialogue, is correctly reflected in the flow. The processing device 10 generates a plurality of flows through the structuring process.

[0045] FIG. 12 is an example of a flowchart generated by the processing device 10. In FIG. 12, for the sake of simplicity of the drawing, only the titles of questions and proposals are displayed in each step.

[0046] When compared with the flowchart shown in FIG. 7, although there are only some branches missing in the flowchart shown in FIG. 12, there are almost no meaningful differences, and it can be seen that a flowchart showing the tacit knowledge and judgment of experts has been inferred almost accurately. Also, the interpretability of the flowchart is hardly impaired, and the number of branches is such that a person can naturally interpret how to respond to the user in what kind of flow.

[0047] [Effect of the embodiment] As described above, as shown by experiments, the processing device 10 according to the embodiment can generate a flowchart showing the tacit knowledge and judgment of experts from a plurality of dialogues with high interpretability. Therefore, the processing device 10 can analyze the dialogue between the expert and the user using the LLM 20 and visualize the judgment of the expert as a flowchart.

[0048] According to the embodiment, as illustrated in FIG. 13, from the dialogue between an operator (expert) of a travel agency and a user who wants to make a travel reservation, the tacit judgment of the operator (for example, a flowchart Ct1 in the operator's mind of asking whether there is a child's bed if there is a child) can be visualized ((1) in FIG. 13).

[0049] [Regarding the system configuration of the embodiment] Each component of the processing device 10 is functionally conceptual and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of the distribution and integration of the functions of the processing device 10 is not limited to that shown in the figure, and all or part of it can be configured by functionally or physically distributing or integrating it in any unit according to various loads and usage situations.

[0050] Furthermore, each process performed in the processing unit 10 may be implemented in whole or in part by a CPU, a GPU (Graphics Processing Unit), and a program that is analyzed and executed by the CPU and GPU. Alternatively, each process performed in the processing unit 10 may be implemented as hardware using wired logic.

[0051] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated may be changed as appropriate unless otherwise specified.

[0052] [Program] Figure 14 shows an example of a computer in which the processing unit 10 is realized when a program is executed. The computer 1000 has, for example, memory 1010 and CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These parts are connected by a bus 1080.

[0053] Memory 1010 includes ROM 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0054] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define each process of the processing unit 10 are implemented as program modules 1093 in which code executable by the computer 1000 is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, a program module 1093 for performing processes similar to the functional configuration of the processing unit 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0055] Furthermore, the configuration data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.

[0056] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.

[0057] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in these embodiments. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on these embodiments are included in the scope of the present invention.

[0058] 10 Processing unit 11 Question extraction unit 12 Proposal extraction unit 13 Structuring unit 14 Chart construction unit 20 LLM

Claims

1. A processing device comprising: a first extraction unit that uses a large-scale language model to extract questions asked by an expert to elicit information from a user from multiple dialogues between the expert and the user; a second extraction unit that uses a large-scale language model to extract suggestions made by the expert to solve the user's problem from the multiple dialogues; a structuring unit that uses a large-scale language model to convert the dialogues between the expert and the user into a structure having information indicating the questions, the user's answers to the questions, and the suggestions accepted by the user; and a construction unit that statistically constructs a flowchart showing the expert's tacit knowledge and judgment based on a plurality of the structures.

2. The processing apparatus according to claim 1, wherein the first extraction unit includes all the dialogues in a prompt instructing the large-scale language model to extract the questions, or uses a prompt instructing the large-scale language model to extract a list of questions from each dialogue and list the extracted questions, and causes the large-scale language model to integrate each question list for questions of the same perspective.

3. The processing apparatus according to claim 2, wherein the second extraction unit includes all of the dialogues in a prompt instructing the large-scale language model to extract the proposals, or uses a prompt instructing the large-scale language model to extract a list of proposals from each dialogue, and causes the large-scale language model to integrate each list of proposals for each question of the same perspective.

4. The processing apparatus according to claim 3, characterized in that the structuring unit generates a structure having a flow of the question, the answer to the question, and the suggestion to the answer, for a single dialogue, by referring to the question list and the suggestion list.

5. The processing apparatus according to claim 1, wherein the construction unit decomposes the structure into a triplet structure of a first question, the answer to the first question, and the next second question or proposal, counts the number of occurrences of the triplet structure for all the structures, and constructs the flowchart by sequentially constructing a tree structure starting from the triplet structure with the highest count.

6. A processing method to be executed by a processing device, comprising: a step of using a large-scale language model to extract questions asked by an expert to elicit information from a user from a plurality of conversations between the expert and the user; a step of using a large-scale language model to extract suggestions made by the expert to solve the user's problem from the plurality of conversations; a step of using a large-scale language model to convert the conversations between the expert and the user into a structure having information indicating the questions, the user's answers to the questions, and the suggestions accepted by the user; and a step of statistically constructing a flowchart showing the expert's tacit knowledge and judgment based on a plurality of the structures.

7. A processing program for causing a computer to perform the following steps:

7. Using a large-scale language model, extract questions asked by an expert to elicit information from a user from multiple conversations between the expert and the user; using a large-scale language model, extract suggestions made by the expert to solve the user's problem from the multiple conversations; using a large-scale language model, convert the conversations between the expert and the user into a structure having information indicating the questions, the user's answers to the questions, and the suggestions accepted by the user; and statistically constructing a flowchart representing the expert's tacit knowledge and judgment based on multiple structures.

Images