Procedure manual generation device and procedure manual generation method

Abstract

This procedure manual generation device includes: a correction information generation unit configured to generate, with respect to a new procedure manual generated by a trained language model, correction information including correction portions and correction proposals on the basis of comparison with an existing procedure manual; and a correction unit configured to cause the language model to perform correction of the new procedure manual on the basis of the correction information, thereby reducing a workload on a user for obtaining a desired procedure manual.

Description

Procedure Manual Generation Apparatus and Procedure Manual Generation Method

[0001] The present invention relates to a procedure manual generation apparatus and a procedure manual generation method.

[0002] In recent years, Large Language Models (hereinafter referred to as "LLMs") have achieved dramatic progress in the field of natural language processing, and have come to be widely used for answer generation for specific tasks and decision-making support. When an LLM is intended to be used in a specific specialized field, it has not originally learned such knowledge. However, when a user provides specialized information as context in a prompt, which is a question input to the LLM, or when a mechanism such as RAG (Retrieval Augmented Generation) that extracts necessary specialized knowledge from a database according to the content of the prompt and additionally provides the specialized knowledge to the LLM is used, the LLM can take into account both general knowledge and information in a specific field, and answer as if it has specialized knowledge (Non-Patent Document 1).

[0003] However, answers generated by LLMs are not always accurate or appropriate. In particular, in the task of generating a procedure manual based on advanced specialized knowledge, there is a high possibility that the information generated by an LLM contains errors. LLMs are trained based on a general knowledge base, and even if the above-mentioned specialized knowledge is provided as context information, there is a limit to their ability to effectively utilize such information to create a procedure manual that adapts to the user's request, and they cannot provide an accurate answer in a single attempt.

[0004] Furthermore, as a study example of generating procedure manuals using LLMs, there are studies on code creation for the purpose of program code generation, while reflecting corrections to LLM-generated answers (Non-Patent Document 2). However, in this technology, corrections are made while obtaining success or failure feedback on whether the generated program can actually be executed. At present, a mechanism for creating a procedure manual for procedures whose functionality cannot be immediately executed and confirmed after generation has not been sufficiently established.

[0005] Therefore, the generated content needs to be manually reviewed and corrected by humans, which is one of the factors that hinders work efficiency.

[0006] L. Patrick, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuettler et al, "Retrieval-augmented generation for knowledge-intensive NLP tasks", Advances in Neural Information Processing Systems 33, pp.9459-9474, 2020M. Schaefer, S. Nadi, A. Eghbali and F. Tip, "An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation", in IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85-105, Jan. 2024

[0007] Generally, when attempting to create a procedure manual using LLM (Language-Led Mechanism), the desired procedure manual is not always obtained in the initial response. One way to improve the procedure manual provided in the initial response is for the questioner to request revisions sequentially to improve the response. However, manually requesting revisions requires the questioner to possess a certain level of expertise in that field.

[0008] The present invention has been made in view of the above points, and aims to reduce the workload on users in obtaining a desired procedure manual.

[0009] To address the above issues, the procedure manual generation device includes a correction information generation unit configured to generate correction information, including correction locations and proposed corrections, based on a comparison of a new procedure manual generated by a trained language model with an existing procedure manual, and a correction unit configured to cause the language model to perform corrections to the new procedure manual based on the correction information.

[0010] This can reduce the workload on users in obtaining the desired procedure manual.

[0011] This figure shows an example of the hardware configuration of the procedure manual generation device 10 in the first embodiment. This figure shows an example of the functional configuration of the procedure manual generation device 10 in the first embodiment. This is a flowchart illustrating an example of a processing procedure performed by the procedure manual generation device 10 in the first embodiment. This figure shows an example of the functional configuration of the procedure manual generation device 10 in the second embodiment. This is a flowchart illustrating an example of a processing procedure performed by the procedure manual generation device 10 in the second embodiment.

[0012] This embodiment discloses an apparatus, method, and program for automatically generating procedure manuals by automatically and iteratively modifying the responses of a Large Language Model (LLM).

[0013] A procedure manual is defined as "a document that lists, in chronological order, the actions (tasks, etc.) that a person should take to achieve a certain objective." The following is an example of the contents of a procedure manual.

[0014] "Procedure Manual for Achieving XX 1. ... 2. ... 3. ..." For procedures in the network field, an example of such a document might be: "OS Update Procedure Manual 1. Check the status of the device 2. Download the OS file 3. Transfer the downloaded file to the device to be updated..."

[0015] Embodiments of the present invention will be described below with reference to the drawings.

[0016] Figure 1 shows an example of the hardware configuration of the procedure manual generation device 10 in the first embodiment. The procedure manual generation device 10 in Figure 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, etc., which are all interconnected by bus B.

[0017] The program that implements the processing in the procedure manual generation device 10 is provided on a recording medium 101 such as a CD-ROM. When the recording medium 101 containing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101; it may also be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files and data.

[0018] The memory device 103 reads and stores a program from the auxiliary storage device 102 when a program startup command is received. The processor 104 is either a CPU or a GPU (Graphics Processing Unit), or both a CPU and a GPU, and executes the functions related to the procedure manual generation device 10 according to the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0019] Figure 2 shows an example of the functional configuration of the procedure manual generation device 10 in the first embodiment. In Figure 2, the procedure manual generation device 10 has a language model 11, a correction information generation unit 12, and a correction unit 13 in order to generate procedure manuals that need to be created based on specialized knowledge. Each of these units is realized by processing that one or more programs installed in the procedure manual generation device 10 cause the processor 104 to execute.

[0020] In the first embodiment, the procedure manual generation device 10, in the process of generating the procedure manual, repeatedly performs QA (input / output) with correction requests added, rather than just a single QA, in order to make the language model 11 aware of the points to note in the input information ("what" and "where"). Specifically, by providing a function to determine where there are deficiencies in the procedure manual answered (generated) by the language model 11 and request corrections, the output result and correction request are added to the auxiliary information, and the QA of the language model 11 is executed again. The above operations are continued until there are no deficiencies.

[0021] Language model 11 is an example of a trained language model, such as LLM. A request to generate a procedure manual that includes the objective to be achieved by the procedure manual to be generated (what objective will be achieved by the procedure manual described in the manual) and a prompt that includes similar procedure manuals (hereinafter referred to as "generation prompt") are input, and the procedure manual is generated as a response to the input generation prompt, etc. Similar procedure manuals are existing procedure manuals that have a similar objective to the procedure manual to be generated. For example, if you want to generate a procedure manual to update device A from version 3.3 to 3.4, a procedure manual for updating device A from version 3.2 to 3.3 would be a similar procedure manual. If there is other information useful in generating the procedure manual, such as a format explanation manual, unique knowledge of the target field, or past procedure manual creation examples (hereinafter referred to as "auxiliary information"), such auxiliary information may be included in the generation prompt. A format explanation manual is a document that explains what format the procedure manual is. Unique knowledge in the target field refers to information necessary for creating procedure manuals (part of specialized books and materials), and is information other than past procedure manuals. Similar procedure manuals and auxiliary information are stored, for example, in the auxiliary storage device 102.

[0022] The correction information generation unit 12 receives the new procedure manual generated by the language model 11 and one or more similar procedure manuals as input, compares the response with the similar procedure manuals, and outputs the parts that need to be corrected (parts to be corrected) and the proposed corrections in natural language.

[0023] The modification unit 13 generates a prompt (hereinafter referred to as the "modification prompt") to request the language model 11 to modify the procedure manual based on the modification information output from the modification information generation unit 12, and inputs the modification prompt to the language model 11, causing the language model 11 to perform the modification of the procedure manual.

[0024] The following describes the processing steps performed by the procedure manual generation device 10. Figure 3 is a flowchart illustrating an example of the processing steps performed by the procedure manual generation device 10 in the first embodiment.

[0025] In step S101, the language model 11 generates a new procedure document based on the input generation prompt and learned parameters. The generation prompt is text that includes a request to generate a procedure document (a sentence containing the objective to be achieved and instructions for generating a procedure document to be implemented in order to achieve that objective) and one or more similar procedure documents. For example, one example of a generation prompt is text such as, "Please generate a procedure document to achieve {objective}, referring to the following similar procedure documents. {Similar procedure documents}". Here, {objective} is replaced with a specific objective. Here, {similar procedure documents} is replaced with the contents of one or more specific similar procedure documents.

[0026] If supplementary information is available, the generation prompt may include it. In this case, the generation prompt may further include instructions to generate the procedure manual using the supplementary information as a reference. For example, one example of a generation prompt in this case would be text such as, "Generate a procedure manual to accomplish {objective}, using the following similar procedure manual and supplementary information as a reference. {Similar procedure manual} {Supplementary information}" where {Supplementary information} is replaced with specific supplementary information.

[0027] Next, the correction information generation unit 12 generates correction information for the content of the newly generated procedure manual (hereinafter referred to as the "target procedure manual") (S102). Specifically, the correction information generation unit 12 identifies correction points (procedural errors, inconsistencies, missing procedures, or areas that can be improved) in the target procedure manual based on a comparison with similar procedure manuals, and generates correction information that includes the correction points and proposed corrections. Correction information is generated for each correction point. For example, if "5. Confirm YYY." which is in a similar procedure manual, is not in the target procedure manual, correction information is generated that designates "5. Confirm YYY." as the correction point and "Is it necessary to confirm YYY?" as the proposed correction. In addition, the target procedure manual may be absolutely evaluated, and correction information may be generated that includes procedures that contradict the purpose of the target procedure manual and proposed corrections for those procedures.

[0028] The correction information may be generated by the language model 11. In this case, the correction information generation unit 12 may input a prompt to the language model 11 instructing it to include the target procedure document and similar procedure documents, compare the target procedure document and similar procedure documents, and generate correction information for the target procedure document (corrected parts and proposed corrections). Alternatively, the correction information generation unit 12 may identify procedures included in similar procedure documents but not in the target procedure document as corrected parts, and generate information including those corrected parts and proposed corrections for those parts as correction information.

[0029] If the correction information generation unit 12 generates one or more correction pieces (Yes in S103), the correction unit 13 generates a correction prompt that instructs the language model 11 to correct the target procedure document based on the correction information (S104). For example, the correction prompt includes the target procedure document, followed by a correction instruction based on the correction information. Furthermore, if step S104 is executed for the target procedure document for the N (N≧2)th time, the correction prompt also includes the correction history up to N-1 times (the correction instruction included in the correction prompt generated in step S104 and the procedure document corrected as a result). By including such a correction history in the correction prompt, information such as what kind of corrections have been made and from what perspective can be provided to the language model 11.

[0030] In this case, the correction information generated by the correction information generation unit 12 may be a specific procedure that can only be applied to the procedure manual of a similar procedure manual (a procedure specific to the similar procedure manual). Therefore, it is effective for the correction unit 13 to take measures to prevent the correction instructions in the correction prompt from becoming too specific. For example, the correction unit 13 may ask the language model 11 a question such as, "From what perspective is this procedure necessary?" regarding the correction information generated by the correction information generation unit 12, and specify the answered perspective as a proposed correction to the correction information.

[0031] Next, when the correction prompt generated by the correction unit 13 is input to the language model 11, the language model 11 generates a corrected procedure document based on the correction prompt and outputs a response that includes the corrected procedure document (S105).

[0032] Next, the outputted procedure document is used as the target procedure document, and steps S102 and beyond are repeated.

[0033] When the correction information generation unit 12 stops generating correction information (when the correction location can no longer be identified) (No in S102), the processing procedure in Figure 3 ends.

[0034] The correction information generation unit 12 may also determine whether or not there are corrections based on whether the number of steps included in the target procedure is approximately the same as the number of steps in a similar procedure manual (for example, whether the difference in the number of steps between the two is less than a threshold).

[0035] As described above, according to the first embodiment, the language model 11 automatically generates corrections and suggested corrections for the procedure document it generates, and the procedure document is automatically corrected based on these corrections and suggested corrections. As a result, the amount of correction required for the final output procedure document can be reduced. Therefore, the user's workload to obtain the desired procedure document can be reduced.

[0036] Furthermore, by adding supplementary information, it becomes possible to generate and revise procedure manuals based on internal knowledge and expertise.

[0037] Next, a second embodiment will be described. The differences between the second embodiment and the first embodiment will be described. Points not specifically mentioned in the second embodiment may be the same as in the first embodiment.

[0038] In the second embodiment, the revision record for each existing procedure manual, which is the history of procedure manual revision requests input to the language model 11 up to the time the existing procedure manual was obtained, is used to revise the new procedure manual.

[0039] In the second embodiment, the prerequisite is to store the correction record, which is the know-how for making corrections, in the DB. The correction record is a column of output from language model 11 -> correction request -> output from language model 11 -> correction request -> ...

[0040] When given a problem requiring the creation of a procedure manual, RAG retrieves relevant revision records from the DB. Subsequently, the language model 11 generates a new procedure manual based on the revision records. The procedure is modified by sequentially applying and executing the revision request column (a chronological column of revision requests for input to the language model 11) recorded in the revision record. The DB may also store the revision records output by the first embodiment (first output from the language model 11 → prompt generated by the modification unit 13 → output from the language model 11...).

[0041] Figure 4 shows an example of the functional configuration of the procedure manual generation device 10 in the second embodiment. In Figure 4, the procedure manual generation device 10 has a language model 11, an extraction unit 14, and a modification unit 15 in order to generate procedure manuals that need to be created based on specialized knowledge. Each of these units is realized by processing that one or more programs installed in the procedure manual generation device 10 cause the processor 104 to execute. The procedure manual generation device 10 also utilizes a modification record DB 16. The modification record DB 16 can be realized using, for example, an auxiliary storage device 102, or a storage device that can be connected to the procedure manual generation device 10 via a network. In Figure 4, the same reference numerals are used for parts that are the same as in Figure 2, and their descriptions are omitted.

[0042] The correction record DB 16 stores correction records for each procedure manual generated in the past.

[0043] The extraction unit 14 receives as input a generation prompt for instructing the language model 11 to generate a procedure manual, and extracts (acquires), from among the correction records stored in the correction record DB 16, some correction records having relatively high relevance to (the purpose mentioned in) the generation prompt (hereinafter referred to as "relevant correction records"). Note that the configuration of the generation prompt may be the same as that in the first embodiment.

[0044] The correction unit 15 generates a prompt for causing the language model 11 to correct the newly generated procedure manual based on the relevant correction records extracted by the extraction unit 14, and inputs the generated prompt to the language model 11, thereby causing the language model 11 to execute correction of the procedure manual.

[0045] FIG. 5 is a flowchart for explaining an example of a processing procedure executed by the procedure manual generating apparatus 10 according to the second embodiment.

[0046] In step S201, the language model 11 generates a new procedure manual (hereinafter referred to as "target procedure manual") based on the input generation prompt. Step S201 is the same process as step S101 in FIG. 3; however, the processing after step S202 may be performed by using, as an input, the procedure manual finally obtained by implementing the first embodiment without executing step S201.

[0047] Subsequently, the extraction unit 14 extracts, from the set of correction records recorded in the correction record DB 16, correction records having relatively high relevance to the purpose of procedure manual generation described in the generation prompt as "relevant correction records" (S202). A correction record is a sequence of pairs {procedure manual, correction request} each consisting of a procedure manual generated by the language model 11 and a correction request for the procedure manual. When correction is performed a plurality of times, the correction record includes a plurality of lines where one pair corresponds to one line. Hereinafter, this number of lines is denoted by N, and it is assumed that N lines of pairs form one correction record for procedure manual creation. This does not mean that the number of lines of each correction record is the same, but rather means that the number of lines of each correction record is denoted by N for convenience. Note that the N-th line of a correction record is {procedure manual, no correction required}. That is, the N-th line of a correction record includes a procedure manual that requires no correction. Therefore, one correction record includes N-1 correction requests.

[0048] In the correction record DB 16, each correction record is stored in text form. If one correction record is denoted as r_i, the correction record DB 16 including n correction records is {r_1, r_2, ..., r_n}.

[0049] The extraction unit 14 vectorizes each correction record using a text distributed representation tool (such as llamax index). Here, the result of vectorizing {r_1, r_2, ... r_n} is expressed as {vr_1, vr_2, ... vr_n}.

[0050] The extraction unit 14 similarly generates a distributed representation and vectorizes the generation prompt as well. Here, the distributed representation of the generation prompt is denoted as v. It is assumed that all distributed representations have the same vector length. The extraction unit 14 calculates the cosine similarity α between v and vr_i as follows.

[0051] α = vr_i･v / |vr_i||v| The extraction unit 14 extracts correction records r_i corresponding to k vr_i in descending order of α (order of similarity to v) as relevant correction records.

[0052] Next, the modification unit 15 assigns 1 to the variable j (S203). The variable j is used to store the order of the related modification record being processed among the k related modification records. Hereafter, the j-th related modification record will be referred to as "related modification record (j)".

[0053] Next, the modification unit 15 assigns 1 to the variable i (S204). Variable i is a variable used to store the row number of the related modification record (j) that is being processed.

[0054] Next, the modification unit 15 generates a prompt instructing that the modification indicated by the modification request on line i of the related modification record (j) be applied to the target procedure manual, and inputs this prompt (hereinafter referred to as the "modification prompt") to the language model 11 (S205).

[0055] Next, the language model 11 performs modifications to the target procedure document based on the input modification prompt and the learned parameters (S206). Specifically, the language model 11 applies the modifications based on the modification request included in the modification prompt to the target procedure document included in the modification prompt, thereby generating a procedure document that reflects the modification request. The generated procedure document becomes the new target procedure document.

[0056] Next, the modification unit 15 adds 1 to the variable i (S207) and determines whether the value of variable i has reached N (the line number of the last line of the related modification record (j)) (S208). If the value of variable i is less than N (No in S208), steps S205 onward are repeated, and the modification of the target procedure manual is performed based on the modification request contained in the next line of the related modification record (j).

[0057] If the value of variable i is N (Yes in S208), the modification unit 15 adds 1 to variable j (S209) and determines whether the value of variable j exceeds k (total number of related modification records) (S210). If the value of variable j is less than or equal to k (No in S210), steps S204 onward are repeated, and the modification of the target procedure manual is performed based on the next related modification record.

[0058] When the value of variable j exceeds k (Yes in S210), the processing procedure in Figure 5 is terminated, and the revised target procedure document is output.

[0059] As described above, the second embodiment provides the same effects as the first embodiment.

[0060] As described above, each of the embodiments provides a mechanism that automatically verifies the accuracy of responses generated by a large-scale language model (LLM) using specialized knowledge and corrects any inaccuracies. This makes it possible to streamline the human verification work that was previously required in the process of generating specialized procedures and solutions, while significantly improving reliability.

[0061] Although embodiments of the present invention have been described in detail above, the present invention is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims.

[0062] 10 Procedure manual generation device 11 Language model 12 Correction information generation unit 13 Correction unit 14 Extraction unit 15 Correction unit 16 Correction record DB 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device B Bus

Claims

1. A procedure manual generation device comprising: a correction information generation unit configured to generate correction information, including correction locations and proposed corrections, based on a comparison of a new procedure manual generated by a trained language model with an existing procedure manual; and a correction unit configured to cause the language model to perform corrections to the new procedure manual based on the correction information.

2. A procedure manual generation device comprising: an extraction unit configured to extract from a set of revision records, which include a history of revision requests for procedure manuals input to the language model up to the time the existing procedure manual was obtained, revision records that have a relatively high correlation with prompts for causing the language model to generate a new procedure manual; and a modification unit configured to cause the language model to perform revisions based on the revision requests contained in the revision records extracted by the extraction unit, for the procedure manual generated by the language model based on the prompt.

3. A method for generating a procedure manual, characterized in that a computer performs the following steps: a correction information generation step for generating correction information including corrections and proposed corrections based on a comparison of a new procedure manual generated by a trained language model with an existing procedure manual; and a correction step for causing the language model to perform corrections to the new procedure manual based on the correction information.

4. A method for generating a procedure manual, characterized in that a computer executes the following steps: an extraction step, from a set of revision records for each existing procedure manual generated by a trained language model, which include a history of revision requests for procedure manuals input to the language model up to the time the existing procedure manual was obtained, the extraction step, which extracts revision records that have a relatively high correlation with prompts for causing the language model to generate a new procedure manual; and a revision step, which causes the language model to perform revisions based on the revision requests contained in the revision records extracted by the extraction step, for the procedure manual generated by the language model based on the prompt.

Images