Information processing method, information processing system, and information processing program

Abstract

An information processing method according to the present embodiment comprises generating, on the basis of a checkpoint related to review of a document and a prompt indicating evaluation of the checkpoint, evaluation information including an evaluation result for the checkpoint and at least one improvement plan for a text indicating the checkpoint, by using a trained model.

Description

Information Processing Method, Information Processing System, and Information Processing Program 【0001】 Embodiments of the present invention relate to an information processing method, an information processing system, and an information processing program. 【0002】 Conventionally, systems for assisting in document review have been proposed. For example, a legal document evaluation method for evaluating legal documents based on evaluation rules has been proposed. 【0003】 Japanese Patent Application Laid-Open No. 2019-212315 【0004】 The problem to be solved by the present disclosure is to provide a technique for assisting in the review of a document based on checkpoints. 【0005】 The information processing method according to the embodiment generates evaluation information including an evaluation result for the checkpoint and at least one improvement plan for the checkpoint based on a checkpoint related to document review and a prompt related to evaluation of the checkpoint, using a learned model. 【0006】 FIG. 1 is a schematic diagram for explaining the outline of an information processing system according to the embodiment. FIG. 2 is a diagram showing an example of a functional block of an information processing apparatus according to the embodiment. FIG. 3 is a diagram showing an example of a functional block of a user terminal according to the present embodiment. FIG. 4 is a diagram showing an example of a hardware configuration of a computer according to the embodiment. FIG. 5 is a flowchart showing the flow of checkpoint evaluation processing according to the embodiment. FIG. 6 is a diagram showing an example of an input screen for checkpoints and check additional information according to the embodiment. FIG. 7 is a diagram showing an example of data to be evaluated regarding the evaluation of a checkpoint according to the embodiment. FIG. 8 is a diagram showing an example of the content of evaluation information corresponding to the data after evaluation by the LLM according to the embodiment. FIG. 9 is a diagram showing an example of evaluation information when the checkpoint is highly evaluated according to the embodiment. FIG. 10 is a diagram showing an example of evaluation information when the checkpoint is poorly evaluated according to the embodiment. 【0007】The embodiments will be described in detail below with reference to the drawings. In the following description, components having substantially the same function and configuration will be denoted by the same reference numerals, and will be described only when necessary. Furthermore, the embodiments can be combined with other embodiments, modifications, and prior art. 【0008】 Figure 1 is a schematic diagram showing an example of an information processing system 100 according to this embodiment. The information processing system 100 shown in Figure 1 functions as a support system for users when reviewing documents. For example, the information processing system 100 according to this embodiment includes at least an information processing device 1, at least one user terminal (terminal device) 3 that can communicate with the information processing device 1 via a network, and an external information processing device 5 that can communicate with the information processing device 1 via a network. The external information processing device 5 may be provided by a vendor other than the vendor that provides the information processing device 1 or the information processing system 100 to the user. 【0009】 In this embodiment, the user terminal 3 may be one or more. Also, the user terminal 3 may be one or more users. For example, the vendor providing the information processing system 100 in this embodiment issues tenants to multiple organizations such as companies. One or more users authorized by such organizations may use the information processing system 100 to process documents stored in the tenant issued to the organization, to use a playbook function to review such documents, and to evaluate checkpoints in such reviews. 【0010】 Here, the information processing system 100 according to this embodiment is a system that assists in viewing (displaying) documents on, for example, a browser or a document editing application. In this specification, "document" means any document in any language that is created in accordance with certain rules or standards, such as a rule, law, custom, etc. 【0011】Document data is data related to a document, regardless of its format. While typically text data, it can also include image data, tabular data, and other types of data. Furthermore, a document or document data may include metadata created by word processing software (version number, creation date, update date, author, updater, etc.), formatting information (decorations such as underlining and markers, fonts, indentation, etc.), change history information, comment information, and more. 【0012】 In the following explanation, to make the details clearer, the documents handled by the information processing system 100 will be assumed to be legal documents in Japanese, particularly contracts. However, the legal documents handled by the information processing system 100 are not limited to contracts; they may also be, for example, internal organizational regulations. For instance, the legal documents handled by the information processing system 100 may be internal regulations in English. Furthermore, the legal documents handled by the information processing system 100 may be written in any language. 【0013】 A legal document is, for example, a document that contains legal content that produces a predetermined legal effect, and includes criteria from a predetermined perspective, such as whether the legal content or form is desirable or undesirable. Examples include contracts, application forms, memoranda, internal regulations, policies, etc. In this specification, a document or legal document processed by the information processing system 100 may be referred to as a supported document. A supported document may be a document file uploaded from the user terminal 3, a file newly created in the information processing device 1 using an online editor, or a file that has already been uploaded, saved, and / or edited. 【0014】Furthermore, the information processing system 100 according to this embodiment may include, in addition to a function for evaluating checkpoints related to document review (hereinafter referred to as the checkpoint evaluation function), a function for supporting document viewing (document viewing support function), and a function for supporting document creation (document creation support function). The document viewing support function may include, for example, providing the user with the results of performing document review processing on the supported document (review information) on a web browser, performing document editing in response to user instructions, saving or providing the edited document to the user, editing additional information associated with the supported document (e.g., metadata, comment information, chat information, formatting information, etc.), creating a new document (including a copy), comparing multiple documents (e.g., explicitly showing difference information), automatically proofreading the supported document, performing various processes using a translation display function, adding comments etc. regarding the case to which the supported document belongs, and generating and outputting wording that changes the content of the supported document according to the review information using a large-scale language model (LLM), etc. 【0015】 Furthermore, document review processing is a process that analyzes or evaluates the information contained in a document to be supported based on certain criteria and outputs the results (review information). For example, document review processing includes checking and examining whether the information contained in a document to be supported is appropriate based on certain criteria. It may also include outputting the reasons for determining whether the information contained in the document to be supported is appropriate or not. In particular, the document review processing of this embodiment is preferred when it is performed using checkpoints written in natural language and a trained model, especially a large-scale language model (LLM). 【0016】Large-scale language models (LLMs) allow for relatively free setting of input in natural language, enabling the flexible configuration and modification of checkpoints—the criteria for evaluating documents—in natural language, according to the user's perspective, external environment, document type, etc. However, due to the high degree of input flexibility, it can sometimes be difficult to create appropriate checkpoints for review processing. For example, when a user unfamiliar with using LLMs creates checkpoints and performs review processing, it may be difficult to determine whether the review results are appropriate or to find ways to improve the review results. Furthermore, the appropriate expression of checkpoints may differ with updates or changes to the trained model used. The information processing system 100 of this embodiment can evaluate checkpoints and propose more appropriate expressions of checkpoints, making it suitable for review processing using checkpoints and trained models. 【0017】 Furthermore, the document creation support function provides the ability to edit (modify, add to, delete, etc.) the content of the document to be supported, and to provide information to support the editing. In this specification, editing means, for example, adding, deleting, or changing the information included in the document to be edited. In the following, we will use the case where the information processing system 100 is equipped with a checkpoint evaluation function as an example. 【0018】 Furthermore, checkpoints refer to the review criteria used when processing the supported document for review. While the format can be anything, in this embodiment, checkpoints are written in natural language. For example, if the document under review is a contract, a type of legal document, checkpoints might include the risks contained in the contract and standard responses to those risks. 【0019】The service provided by the checkpoint evaluation function in the information processing system 100 according to this embodiment (hereinafter referred to as the checkpoint evaluation service) provides the user, for example, on a web browser, with an evaluation of checkpoints related to the review of a document (supported document) and suggestions for improving said checkpoints. The checkpoints correspond to the review criteria (sometimes called guidelines) in the review of the supported document. The processing realized by the checkpoint evaluation function (hereinafter referred to as the checkpoint evaluation process) includes, for example, evaluating whether the checkpoints used in the review evaluation of the supported document are appropriate and generating suggestions for improvement. The checkpoint evaluation process may further include outputting the reasons for determining whether the checkpoints are appropriate or not. In other words, the checkpoint evaluation process corresponds to inspecting the checkpoints (checkpoint accuracy evaluation, checkpoint risk detection). The checkpoint evaluation process will be explained later. 【0020】 The user terminal 3 in this embodiment is a client device managed by a user utilizing the checkpoint evaluation service. The user terminal 3, for example, displays a screen related to checkpoint input (hereinafter referred to as the checkpoint input screen) on the display device screen based on information from the information processing device 1. The checkpoint input screen corresponds to a user interface for inputting checkpoints and the like. The user terminal 3 also displays the evaluation of the checkpoints and suggestions for improving them. The information processing device 1 and the external information processing device 5 shown in Figure 1 will be described later. 【0021】In this embodiment, the user performs checkpoint input operations using the checkpoint input screen or the like on the display device of the user terminal 3. Checkpoint input operations can be performed, for example, by entering a string in a predetermined field on the checkpoint input screen, or by selecting a pre-set checkpoint. At this time, the user may also input checkpoints generated by a trained model such as a Large Language Model (LLM) into the predetermined field. In addition, the user may further input at least one of the following: the type of contract to which the document belongs, the category indicating the scope of checkpoint verification in the document, and the importance level of the checkpoint. This allows for a more accurate evaluation of whether the checkpoint is appropriate or not. 【0022】 Next, the information processing device 1 shown in Figure 1 will be described using Figure 2. Figure 2 is a diagram showing an example of the functional block of the information processing device 1 according to this embodiment. The information processing device 1 includes, for example, an acquisition unit 11, a processing unit 12, an output unit 13, and a storage unit 14. The acquisition unit 11, processing unit 12, and output unit 13 shown in the functional block of Figure 2 are functions realized by, for example, the processor of the information processing device 1. The functions realized by the acquisition unit 11, processing unit 12, output unit 13, and storage unit 14 correspond to the acquisition function, processing function, output function, and storage function, respectively. 【0023】 The functions realized by the acquisition unit 11, processing unit 12, output unit 13, and storage unit 14 of this embodiment are each stored as programs in, for example, main memory or auxiliary storage. The processor can realize the functions related to the acquisition unit 11, processing unit 12, output unit 13, and storage unit 14 by reading and executing the programs stored in main memory or auxiliary storage. 【0024】In this embodiment, the acquisition unit 11 acquires, for example, DOCX format document data as a supported document via a network. The acquisition unit 11 also acquires, for example, document data stored in the storage unit 14 as a supported document. Furthermore, the acquisition unit 11 acquires, for example, document data stored in the storage unit of another device via a network as a supported document. 【0025】 In this embodiment, the acquisition unit 11 acquires, for example, checkpoints (strings) related to document review from the user terminal 3 via a network. The acquisition unit 11 also acquires, for example, checkpoints stored in the storage unit 14 as targets for checkpoint evaluation. The acquisition unit 11 also acquires, for example, checkpoints stored in the storage unit of another device via a network as targets for checkpoint evaluation. The acquisition unit 11 may also acquire checkpoints generated by the LLM 7 from the external information processing device 5. The acquisition unit 11 also acquires at least one of the following from the user terminal 3: the type of contract to which the document belongs, the category indicating the scope of checkpoint verification in the document, and the importance level of the checkpoints. Hereinafter, the contract type, category, and importance level will be collectively referred to as check-additional information. The check-additional information may further include the usage scene (usage context) of the document being checked. 【0026】 In this embodiment, the processing unit 12 generates checkpoint evaluation information using a trained model based on the checkpoint received from the acquisition unit 11 and the prompt regarding the evaluation of the checkpoint. Specifically, the processing unit 12 transmits the checkpoint and the prompt to the external information processing unit 5. The checkpoint and the prompt are input to the LLM 7 in the external information processing unit 5. Examples of checkpoints and prompts will be described later. If additional check information is acquired, the processing unit 12 may also transmit the additional check information to the external information processing unit 5 in addition to the checkpoint and prompt. The checkpoint, the prompt, and the additional check information are input to the LLM 7 in the external information processing unit 5. 【0027】The processing unit 12 of this embodiment receives output from the LLM 7. The processing unit 12 may use the output from the LLM 7 to generate checkpoint evaluation information. The checkpoint evaluation information includes the evaluation result for the checkpoint and at least one improvement suggestion for the text indicating the checkpoint. The checkpoint evaluation result includes, for example, one of the first, second, and third perspectives. 【0028】 The first aspect of the evaluation results of the checkpoints in this embodiment is whether or not the text to be supported can be answered in a predetermined format when checked with the checkpoints. The predetermined format corresponds to, for example, Yes / No, or the degree to which the checkpoints are satisfied. The degree to which the checkpoints are satisfied is determined, for example, if multiple items are checked by the checkpoints, by which of the multiple items match the checkpoints. 【0029】 Furthermore, if multiple items have a hierarchical structure, the degree to which the checkpoints are satisfied may be determined by which of the multiple items meet the checkpoints at which level of the hierarchical structure. In the text being supported, the hierarchical structure corresponds, for example, to a stepwise structure from a higher-level concept to a lower-level concept, that is, from abstract content to concrete content. In this case, the degree to which the checkpoints are satisfied can indicate to what stage between abstract content and concrete content the content of the checkpoints has been achieved. 【0030】 The second aspect of the checkpoint evaluation results in this embodiment indicates whether the checkpoint is described specifically and objectively. For example, the second aspect corresponds to specific examples or conditions. Furthermore, the third aspect of the checkpoint evaluation results in this embodiment indicates whether the content of the checkpoint is a single item. For example, the third aspect indicates whether there are specific judgment criteria. It should be noted that the first to third aspects of the checkpoint evaluation results have been described as examples, but the evaluation results are not limited to these and can be changed and set as appropriate by the user or vendor. 【0031】 At least one improvement suggestion for a checkpoint corresponds to, for example, one of the first, second, or third perspectives. Furthermore, the improvement suggestions for checkpoints can be modified and configured as needed based on user instructions via the user terminal 3. 【0032】 The trained model in this embodiment is, for example, LLM7 mounted on the external information processing device 5. The trained model may be the same as or different from the trained model used when performing the review of the supported document. Furthermore, LLM7 may be the same as or different from the trained model used to generate checkpoints. Preferably, the trained model used for checkpoint evaluation processing is the LLM7 used when performing the review of the supported document. Here, "same" corresponds to training the large-scale language model to be learned using the same training data (training data, teacher data) when the trained model (LLM7) is generated, that is, when training the large-scale language model to be learned. For example, LLM7 in this embodiment is generated by training the large-scale language model to be learned using the same training data as the training data used when training the trained model used when performing the review of the supported document and / or the trained model that generates checkpoints. Furthermore, when a model is generally considered to be the same, this includes cases where it is provided as the same model by the developer or provider of the trained model, or where it is provided with the same model number or code name. 【0033】 By using the same trained model that is used during document review to evaluate checkpoints or generate improvement suggestions, it is possible to evaluate checkpoints or generate improvement suggestions more appropriately. This is presumably because the evaluation or generation of improvement suggestions takes into account the accuracy of the actual document review. 【0034】In this embodiment, the output unit 13 outputs, for example, the evaluation information created by the processing unit 12 to the user terminal 3. For example, the output unit 13 converts the checkpoint evaluation information into information for display on the display unit of the user terminal 3 and outputs it. 【0035】 In this embodiment, the storage unit 14 stores, for example, checkpoints 200, prompts 300, check-additional information 400, evaluation information 500, etc. The storage unit 14 may also store document data, including the document to be supported. Furthermore, the storage unit 14 may also store review results for the document to be supported. 【0036】 If the document being supported is a legally binding contract, prompt 300 might be something like, "Determine whether the text given as a checkpoint item for a legally binding contract is a checkpoint that LLM can answer with Yes or No. If there are any areas for improvement, output suggestions for improvement. Output them in bullet points based on the output example. Output multiple examples of improved checkpoint items." Checkpoint 200 might also be something like, "It is not clearly defined which information must be managed as confidential information by our company." 【0037】 Now, let's return to Figure 1 and explain the external information processing device 5. The external information processing device 5 shown in Figure 1 includes pre-trained models such as a large-scale language model (LLM7). The large-scale language model includes, for example, BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), ChatGPT or GPT-4, and GPT-4o. 【0038】 In this embodiment, the information processing device 1 transmits prompts and checkpoints to the LLM 7 provided in the external information processing device 5. The information processing device 1 also receives the information generated by the LLM 7. Based on this information, the information processing device 1 generates checkpoint evaluation information 500. 【0039】The trained model may be a single machine learning model or multiple machine learning models. For example, it may be customized or fine-tuned by additional training according to the field of documents being handled (mainly contracts and legal documents in this embodiment) or the user, or by combining it with another trained model that processes input or output. This makes it possible to generate more appropriate evaluation information 500. 【0040】 Furthermore, the trained model may be stored in the information processing device 1 provided by the vendor providing the information processing system 100, rather than in the external information processing device 5 provided by an external vendor. 【0041】 Next, the details of the user terminal 3 of this embodiment will be described using Figure 3. Figure 3 is a diagram showing an example of a functional block of the user terminal 3 according to this embodiment. For example, the functions are realized by the processor of the user terminal 3. The user terminal 3 includes, for example, an input unit 31, a display unit 32, a storage unit 33, and an output unit 34. 【0042】 The input unit 31 of the user terminal 3 in this embodiment enables the user to select or input information. For example, the input unit 31 allows the user to input and select target documents to be input into the information processing device 1. The input unit 31 also allows the user to input and select checkpoints according to the user's instructions. Furthermore, the input unit 31 allows the user to input and select additional check information according to the user's instructions. Specifically, the input unit 31 is implemented by, for example, a keyboard, mouse, or voice input device. 【0043】 For example, the input unit 31 inputs checkpoints according to the user's instructions using a keyboard, mouse, voice input device, etc. The input unit 31 also selects a checkpoint to be evaluated from a plurality of checkpoints pre-set and stored in the memory unit 33, according to the user's instructions. The input unit 31 may also input and select additional checkpoint information according to the user's instructions using a keyboard, mouse, voice input device, etc. 【0044】The display unit 32 of the user terminal 3 in this embodiment displays a user interface (UI) into which checkpoint and check additional information can be input. For example, the display unit 32 displays a graphical user interface (GUI) as the UI. Further, the display unit 32 displays the evaluation information 500 generated by the information processing apparatus 1. Note that the display unit 32 may display document data or the like for the user. Specifically, the display unit 32 is realized by various display devices or the like. 【0045】 The storage unit 33 of the user terminal 3 in this embodiment stores programs related to various applications that can use the information processing system 100. Further, the storage unit 33 stores, for example, checkpoints, check additional information, and evaluation information. 【0046】 The output unit 34 of the user terminal 3 in this embodiment outputs the information selected by the user to the information processing apparatus 1. For example, the output unit 34 outputs the checkpoint to the information processing apparatus 1. When check additional information is input, the output unit 34 outputs the checkpoint and the check additional information to the information processing apparatus 1. 【0047】 Next, an example of the hardware configuration of a computer 40 that can be used to configure the information processing apparatus 1 and the user terminal 3 will be described using FIG. 4. FIG. 4 is a diagram showing an example of the hardware configuration of the computer 40 according to this embodiment. 【0048】 As shown in FIG. 4, the information processing apparatus 1 and the user terminal 3 in this embodiment are the computer 40. As an example, it includes a processor 41, a main storage device (memory) 42, an auxiliary storage device (memory) 43, an input / output interface 44, and a communication interface 45. These are interconnected via a bus line 46 including an address bus, a data bus, a control bus, and the like. Note that an interface circuit (not shown) may be interposed between the bus line 46 and each hardware resource as appropriate. 【0049】The information processing device 1 and user terminal 3 of this embodiment shown in Figure 4 each have one of each component, but they may also have multiple identical components. Furthermore, although Figures 2 and 3 show one information processing device 1 and one user terminal 3, the software may be installed on multiple computers 40, and each of these computers 40 may execute the same or different parts of the software's processing. In this case, each computer 40 may communicate via a communication interface 45 or the like to execute processing in a distributed computing configuration. In other words, the information processing device 1 and user terminal 3 in this embodiment may be configured as a system that realizes the various functions described later by having one or more computers 40 execute instructions stored in one or more storage devices. 【0050】 The various calculations performed by the information processing device 1 and the user terminal 3 in this embodiment may be executed in parallel using one or more processors 41, or using multiple computers 40 connected via a network. Alternatively, the various calculations may be distributed to multiple processing cores within the processor 41 and executed in parallel. Furthermore, some or all of the processing and means of this disclosure may be executed by at least one of a processor 41 and a storage device located on a cloud that can communicate with the information processing device 1 and the user terminal 3 via a network. Thus, the processing of the information processing system 100 in this embodiment may take the form of parallel computing using one or more computers 40. 【0051】The processor 41 may be an electronic circuit (processing circuit, processing circuit, processing circuitry, CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit), etc.) including a control device and an arithmetic unit. Alternatively, the processor 41 may be a semiconductor device including a dedicated processing circuit. The processor 41 is not limited to an electronic circuit using electronic logic elements, but may also be realized by an optical circuit using optical logic elements. Furthermore, the processor 41 may include arithmetic functions based on quantum computing. 【0052】 The processor 41 performs calculations based on data and software (programs) input from the internal components of the information processing device 1 and the user terminal 3, and can output calculation results and control signals to the respective devices. The processor 41 may also control the components of the information processing device 1 and the user terminal 3 by executing the OS (Operating System) and applications of the information processing device 1 and the user terminal 3. 【0053】 In this embodiment, the information processing device 1 and the user terminal 3 may be implemented by one or more processors 41. Here, the processor 41 may refer to one or more electronic circuits arranged on one chip, or one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, each electronic circuit may communicate by wire or wireless. 【0054】In this embodiment, the main memory 42 is a storage device that stores instructions executed by the processor 41 and various data, and the information stored in the main memory 42 is read by the processor 41. The auxiliary storage device 43 is a storage device other than the main memory 42. These storage devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The storage device for storing various data used in the information processing device 1 and the user terminal 3 according to this embodiment may be implemented by the main memory 42 or the auxiliary storage device 43, or by the built-in memory of the processor 41. For example, the storage unit 14 and the storage unit 33 in this embodiment may be implemented by the main memory 42 or the auxiliary storage device 43. 【0055】 Multiple processors 41 may be connected to (combined with) one memory device, or a single processor 41 may be connected to it. Multiple memory devices may be connected to (combined with) one processor. In this embodiment, if the information processing device 1 and user terminal 3 consist of at least one memory device and multiple processors 41 connected to (combined with) this at least one memory device, the configuration may include at least one of the multiple processors 41 being connected to (combined with) at least one memory device. This configuration may also be realized by memory devices and processors 41 included in multiple computers. Furthermore, the configuration may include a memory device integrated with a processor 41 (for example, a cache memory including an L1 cache and an L2 cache). 【0056】The input / output interface 44 in this embodiment is an interface such as a USB (Universal Serial Bus) that directly connects to an output device such as a display device, an input device, and an external device. The external device may be a storage device such as a memory device, network storage, or HDD. The external device may also be a device that has some of the functions of the components of the information processing device 1 and the user terminal 3 in this embodiment. In other words, the information processing device 1 and the user terminal 3 may transmit or receive some or all of the processing results of the external device. 【0057】 The communication interface 45 in this embodiment is an interface for connecting to a network wirelessly or via a wired connection. The communication interface 45 may be any appropriate interface, such as one that conforms to an existing communication standard. Information may be exchanged with an external device connected via the network through the communication interface 45. The network may be any of the following: WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or a combination thereof, and it is sufficient that information is exchanged between the information processing device 1 and the user terminal 3 or between the information processing device 1 and the external information processing device 5. An example of a WAN is the Internet, an example of a LAN is IEEE 802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication). 【0058】 The overall configuration of the information processing system 100 according to the embodiment has been described above. Under the above configuration, the information processing system 100 executes a checkpoint evaluation function. Specifically, the information processing device 1 works in cooperation with the external information processing device 5 and the user terminal 3 to execute the checkpoint evaluation function. The checkpoint evaluation process executed by the checkpoint evaluation function will be described below with reference to Figure 5. Figure 5 is a flowchart showing an example of the flow of the checkpoint evaluation process. 【0059】The users who perform the checkpoint evaluation process in this embodiment are, for example, business unit personnel, legal personnel, legal managers, general affairs personnel, executives, IT (Information Technology) managers, etc. Furthermore, when the checkpoint evaluation process is used by a team, the users who perform the checkpoint evaluation process are, for example, beginners who have learned the minimum knowledge (approximately Level 3 of the Business Law Proficiency Test) but have no practical experience (less than one year), intermediate personnel who have moderate knowledge (approximately Level 2 of the Business Law Proficiency Test) in addition to 1 to 3 years of legal practice experience, senior personnel who have sufficient knowledge (approximately Level 1 of the Business Law Proficiency Test) in addition to 4 years or more of legal practice experience, managers who manage the legal organization, trainers who are responsible for training junior legal personnel, etc. Alternatively, the process may be performed by a vendor that provides services related to the information processing system 100 to the user. Note that the checkpoint evaluation process is not limited to instructions from the user and may be executed automatically. For example, the process may be initiated when checkpoints are entered and uploaded, when laws and regulations related to the supported document change, when the model (LLM7) used to review the supported document changes, or when data is collected indicating that the review results for the supported document are not very good. 【0060】 (Checkpoint Evaluation Process) (Step S1) When the user instructs the execution of the checkpoint evaluation function via the input unit 31 on the user terminal 3, the display unit 32 displays the checkpoint input screen. For example, the display unit 32 displays the checkpoint input screen as a form for creating company standards related to the user. Checkpoints and the like are entered into the form for creating company standards based on the user's instructions via the input unit 31. Note that the input of checkpoints is not limited to the above. For example, if multiple checkpoints are uploaded together, each of the uploaded checkpoints is entered into the user terminal 3. The input screens for checkpoints and additional checkpoint information in this embodiment will be described below with reference to Figure 6. 【0061】Figure 6 shows an example of the ISC input screen for checkpoints and additional check information according to this embodiment. As shown in Figure 6, the ISC input screen of this embodiment has a checkpoint input field CEF, an additional check information input field AEF, a check field, and various notes. The user inputs checkpoints via the input unit 31 into the checkpoint input field CEF on the ISC input screen displayed on the display unit 32. The input language for checkpoints may be any language. 【0062】 Alternatively, instead of entering a checkpoint string into the checkpoint input field CEF, the user may select a pre-configured checkpoint via the input unit 31. For example, in response to a click operation on the checkpoint input field CEF, the display unit 32 displays a list of pre-configured checkpoints using a pull-down menu or similar method. In this case, the user can select a checkpoint to be evaluated by specifying one checkpoint from the list of displayed checkpoints. The pre-configured checkpoints are uploaded to the information processing device 1 in a known file format, such as CSV (Comma Separated Values), via a user terminal 3, for example, and stored in the storage unit 33. Furthermore, a single checkpoint may be configured by separating it into multiple parts, such as title and content. 【0063】 Furthermore, the user can input additional check information into the input field AEF of the input screen ISC via the input unit 31. Although inputting additional check information in the checkpoint evaluation process is optional, it is preferable to input additional check information to improve the accuracy of the checkpoint evaluation. 【0064】When a checkpoint is entered into the checkpoint input field CEF, and the user presses the execute button CE for evaluating the checkpoint on the input screen ISC, the output unit 34 on the user terminal 3 outputs the checkpoint to the information processing device 1. If additional check information has been entered, the output unit 34 on the user terminal 3 outputs the additional check information in addition to the checkpoint to the information processing device 1. The output unit 34 on the user terminal 3 may also store the checkpoint and the additional check information in the storage unit 33 of the user terminal 3, associating them with each other. 【0065】 (Step S2) In the information processing device 1 of this embodiment, the acquisition unit 11 acquires checkpoints related to the review of the document. Specifically, the acquisition unit 11 acquires checkpoints output from the information processing device 1. At this time, the acquisition unit 11 may store the acquired checkpoints in the storage unit 14. If additional check information is also output from the information processing device 1, the acquisition unit 11 also acquires the additional check information. 【0066】 (Step S3) The processing unit 12 of the information processing device 1 of this embodiment generates a prompt to send to the external information processing device 5 based on the checkpoint. The processing unit 12 generates a prompt that instructs, for example, to generate checkpoint evaluation information 500 using the checkpoint and LLM 7. The prompt may have a string such as, for example, "Please write out the improvement proposal in the language in which the check items (checkpoints, etc.) are written." By including such a string in the prompt, the checkpoint evaluation process can be handled in a variety of languages. 【0067】The prompt may be pre-set and stored in the memory unit 14, such as "Determine whether the text given as a checkpoint item for a legally binding contract is a checkpoint that LLM7 can answer with Yes or No. If there are any areas for improvement, please also output suggestions for improvement. Output them in bullet points based on the output example. Output multiple examples of improved checkpoint items." At this time, the processing unit 12 reads the prompt from the memory unit 14. 【0068】 (Step S4) The output unit 13 in the information processing device 1 of this embodiment outputs (transmits) the checkpoints and prompts to be evaluated to the external information processing device 5. If additional check information has been input, the output unit 13 in the information processing device 1 outputs (transmits) the additional check information to the external information processing device 5 in addition to the checkpoints. 【0069】 Figure 7 shows an example of an ETD (Enterprise Data Table) for evaluation related to checkpoint evaluation. In this embodiment, the ETD for evaluation includes, for example, contract type and category as check-additional information. The ETD for evaluation also includes a string of checkpoints to be evaluated, as shown in Figure 7. The ETD for evaluation may also include an item that sets the answer format for the checkpoints to be evaluated, as shown in Figure 7. 【0070】 (Step S5) In the external information processing device 5 of this embodiment, the LLM7 performs a process (evaluation process) to evaluate checkpoints based on the prompts and checkpoints indicated regarding checkpoint evaluation. That is, when a prompt and a checkpoint are input to the LLM7 in the external information processing device 5, the LLM7 outputs the result of the evaluation process (hereinafter referred to as the evaluation result). 【0071】(Step S6) The external information processing device 5 of this embodiment transmits and outputs the evaluation results to the information processing device 1. For example, the external information processing device 5 transmits and outputs the evaluation results to the information processing device 1 in association with the checkpoints input to the LLM 7. That is, the external information processing device 5 transmits and outputs the evaluation results to the information processing device 1 in association with the checkpoints input to the LLM 7. Note that the association between the checkpoints input to the LLM 7 and the evaluation results is not limited to the checkpoints themselves, and known methods such as metadata may be used. 【0072】 (Step S7) The acquisition unit 11 of the information processing device 1 in this embodiment acquires evaluation results from the external information processing device 5. Specifically, the acquisition unit 11 acquires the checkpoints input to the LLM 7 and the evaluation results from the external information processing device 5. If the evaluation results are associated with metadata, the acquisition unit 11 may use the metadata to acquire the checkpoints corresponding to the evaluation results from the storage unit 14. The processing unit 12 of the information processing device 1 generates evaluation information using the checkpoints and evaluation results. That is, through the processing in steps S5 to S7, the processing unit 12 of the information processing device 1 generates evaluation information using a trained model based on the prompts and checkpoints. The processing unit 12 of the information processing device 1 stores the generated evaluation information in the storage unit 14, associating it with the checkpoints. 【0073】 Figure 8 shows an example of the content of the evaluation information EI corresponding to the data after evaluation by LLM7. The evaluation information EI contains the evaluation results from LLM7 and improvement proposals. Figure 8 shows an example of an improvement proposal. As shown in Figure 8, the details of the improvement proposal are generated as specific suggestions from multiple perspectives (e.g., first perspective, second perspective, third perspective). 【0074】(Step S8) The output unit 13 of the information processing device 1 in this embodiment outputs evaluation information 500 to the user terminal 3. The output unit 13 of the information processing device 1 may output the evaluation information 500 to the user terminal 3 along with checkpoints corresponding to the evaluation information 500. The output unit 13 may also output to a file such as a tabular format. Furthermore, translations according to the language selected by the user may be added to the checkpoints and evaluation information 500 (for example, improvement suggestions). 【0075】 (Step S9) The display unit 32 of the user terminal 3 in this embodiment displays the evaluation information 500 output from the information processing device 1. At this time, the display unit 32 may display a mark indicating that the evaluation result by LLM7 is high (hereinafter referred to as a high evaluation mark) or a mark indicating that the evaluation result by LLM7 is low (hereinafter referred to as a low evaluation mark) together with the evaluation information 500 and checkpoints. The high evaluation mark and the low evaluation mark can be set, for example, by comparing the evaluation result, which has been quantified by a predetermined method, with a threshold. Note that the setting of the high evaluation mark or the low evaluation mark may be implemented by the processing unit 12 of the information processing device 1. At this time, the evaluation information 500 will include the high evaluation mark or the low evaluation mark. 【0076】 Figure 9 shows an example of evaluation information HRT when checkpoint CP receives a high rating. As shown in Figure 9, a high rating mark HRM is displayed in the evaluation information HRT. The high rating mark HRM is, for example, a good mark. Also, as shown in Figure 9, the reason RTR for the evaluation to which the high rating mark HRM was assigned is displayed adjacent to the high rating mark HRM. In addition, below the reason RTR for the evaluation, an example of improvement proposal IP related to the first, second, and third perspectives is displayed. 【0077】Figure 10 shows an example of evaluation information LRT when checkpoint CP receives a low evaluation. As shown in Figure 10, a low evaluation mark LRM is displayed in the evaluation information HRT. Also, as shown in Figure 10, the reason RTR for the evaluation to which the low evaluation mark LRM is assigned is displayed adjacent to the low evaluation mark LRM. In addition, below the reason RTR for the evaluation, an example of improvement proposal IP related to the first, second, and third perspectives is displayed. As shown in Figures 9 and 10, the improvement proposal IP may be displayed on the display unit 32 whether checkpoint CP receives a low or high evaluation. 【0078】 The evaluation information 500 displayed on the display unit 32 of the user terminal 3 is not limited to those shown in Figures 9 and 10. For example, the display unit 32 of the user terminal 3 may display improvement suggestions in an order corresponding to the degree to which the checkpoints are satisfied. Alternatively, the display unit 32 of the user terminal 3 may display improvement suggestions in an order corresponding to the hierarchy in the hierarchical structure. Furthermore, the high-evaluation mark HRM is not limited to the marks shown in Figure 9, and the low-evaluation mark LRM is not limited to the marks shown in Figure 10. 【0079】 When a user selects one of the multiple improvement proposals IP in the evaluation information 500 displayed on the display unit 32 of the user terminal 3, based on instructions from the user via the input unit 31, the input unit 31 inputs the selected improvement proposal as a checkpoint. This allows the selected improvement proposal to be set as a new checkpoint. That is, the selected improvement proposal is reflected in the checkpoint and stored in the storage unit 33. At this point, the checkpoint evaluation process may be executed again for the new checkpoint (checkpoint re-check). 【0080】 Furthermore, the display unit 32 of the user terminal 3 may display a list of multiple checkpoints belonging to the contract type related to checkpoint CP included in the displayed evaluation information 500, along with high-evaluation marks or low-evaluation marks and improvement suggestions for those multiple checkpoints. 【0081】The information processing method according to this embodiment acquires checkpoints related to document review, generates evaluation information using a trained model, which includes evaluation results for the checkpoints and at least one suggested improvement for the text indicating the checkpoints, based on the prompts and checkpoints provided for evaluation of the checkpoints, and displays the generated evaluation information. The trained model in the information processing method according to this embodiment is LLM7. 【0082】 As a result, the information processing method according to this embodiment can provide a technology that supports document review based on checkpoints. For example, the information processing method according to this embodiment can perform checkpoint evaluation (checking) related to document review, and can present the user with the checkpoint evaluation results and suggestions for improving the checkpoints. Therefore, the information processing method according to this embodiment can improve the accuracy of document review. 【0083】 Furthermore, the evaluation result of the checkpoints in the information processing method according to this embodiment includes at least one of the following: a first perspective indicating whether or not the text can be answered in a predetermined format when checked against the checkpoints; a second perspective indicating whether or not the checkpoints are described in a specific and objective manner; and a third perspective indicating whether or not the content of the checkpoints consists of a single item. Furthermore, the predetermined format in the information processing method according to this embodiment corresponds to Yes / No or the degree to which the checkpoints are satisfied. Furthermore, at least one improvement proposal in the information processing method according to this embodiment corresponds to any one of the first, second, and third perspectives. 【0084】 As a result, the information processing method according to this embodiment can present the user with evaluation results corresponding to the first, second, and third perspectives, as well as improvement suggestions corresponding to those evaluation results. Therefore, the information processing method according to this embodiment can improve the checkpoints to make them suitable for document review. 【0085】Furthermore, when an improvement proposal is selected, the information processing method according to this embodiment generates evaluation information based on the prompt and the improvement proposal, including an evaluation result for the improvement proposal and at least one further improvement proposal for the improvement proposal. As a result, according to the information processing method according to this embodiment, checkpoints can be improved in accordance with the improvement proposal and the improved checkpoints can be re-evaluated. Therefore, according to the information processing method according to this embodiment, checkpoint improvements can be easily performed. Therefore, according to the information processing method according to this embodiment, the accuracy of document review can be further improved. 【0086】 Furthermore, the information processing method according to this embodiment acquires at least one of the following: the type of contract to which the document belongs, the category indicating the scope of checkpoint verification in the document, and the importance level of the checkpoint. Based on the type, the category, the importance level, the prompt, and the checkpoint, evaluation information is generated using a trained model. In addition, in the information processing method according to this embodiment, the trained model is the same as the trained model used when performing the review. In this case, the trained model (LLM7) of this embodiment is generated by training a large-scale language model to be trained using the same training data as the training data used when training the trained model used when performing the review of the supported document. As a result, the information processing method according to this embodiment can improve the accuracy of checkpoint evaluation (checking) related to document review. For this reason, the information processing method according to this embodiment can further improve the accuracy of document review. 【0087】When the technical concept in the embodiment is realized in the information processing system 100, the information processing system 100 includes: an acquisition unit 11 that acquires checkpoints related to document review; a processing unit 12 that generates evaluation information, including an evaluation result for the checkpoints and at least one improvement suggestion for the document indicating the checkpoints, using a trained model based on prompts indicating the evaluation of the checkpoints and the checkpoints themselves; and a display unit 32 that displays the evaluation information. The procedure and effects of the checkpoint evaluation process performed by the information processing system 100 are the same as in the embodiment, so a description is omitted. 【0088】 When the technical concept in the embodiment is realized by an information processing program, the information processing program enables the computer to acquire checkpoints related to document review, generate evaluation information using a trained model, which includes evaluation results for the checkpoints and at least one suggested improvement for the text indicating the checkpoints, based on prompts indicating the evaluation of the checkpoints and the checkpoints themselves, and display the evaluation information. 【0089】 For example, checkpoint evaluation processing can also be implemented by installing information processing programs on a computer and loading them into memory. In this case, the information processing programs that enable the computer to execute the checkpoint evaluation processing can also be stored and distributed on storage media such as magnetic disks (hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), and semiconductor memory. The procedure and effects of the checkpoint evaluation processing executed by the information processing program are the same as in the embodiment, so a detailed explanation will be omitted. 【0090】According to the embodiments described above, it is possible to provide a technology that supports document review based on checkpoints. For example, according to this embodiment, it is possible to provide a support technology that enables the evaluation of checkpoints related to document review. Thus, according to at least the embodiments described above, support technology for the evaluation of checkpoints can be provided to users easily and conveniently. 【0091】 Although embodiments of this disclosure have been described in detail above, these embodiments are presented as examples only and are not intended to limit the user to individual embodiments. Each embodiment can be modified in various ways, including additions, changes, substitutions, partial deletions, and combinations, without departing from the technical spirit of the present invention. These embodiments and their variations are included within the scope of the invention described in the claims and its equivalents. 【0092】 1... Information processing device 3... User terminal 5... External information processing device 7... LLM 11... Acquisition unit 12... Processing unit 13, 34... Output unit 14, 33... Storage unit 31... Input unit 32... Display unit 40... Computer 41... Processor 42... Main memory 43... Auxiliary memory 44... Input / output interface 45... Communication interface 100... Information processing system 200... Checkpoint 300... Prompt 400... Check additional information 500... Evaluation information

Claims

1. An information processing method comprising generating evaluation information, which includes an evaluation result for a checkpoint and at least one improvement suggestion for a checkpoint, based on a checkpoint for reviewing a document and a prompt for evaluating the checkpoint, using a trained model.

2. The information processing method according to claim 1, wherein the evaluation result of the checkpoints has at least one of the following: a first viewpoint indicating whether or not the document can be answered in a predetermined format when checked using the checkpoints; a second viewpoint indicating whether or not the checkpoints are specific and objective descriptions; and a third viewpoint indicating whether or not the content of the checkpoints is a single item.

3. The information processing method according to claim 2, wherein the predetermined format corresponds to Yes / No or the degree to which the checkpoints are satisfied.

4. The information processing method according to claim 2, wherein the at least one improvement proposal corresponds to any one of the first, second, and third viewpoints.

5. The information processing method according to claim 1, wherein when the proposed improvement is selected, evaluation information is generated based on the prompt and the proposed improvement, and based on the evaluation result for the proposed improvement and at least one further proposed improvement for the proposed improvement.

6. The information processing method according to any one of claims 1 to 5, further obtaining at least one of the following: the type of contract to which the document belongs, the category indicating the scope of verification by the checkpoint in the document, and the importance of the checkpoint; and generating the evaluation information using the trained model based on the type, the category, the importance, the prompt, and the checkpoint.

7. The information processing method according to claim 1, wherein the trained model is an LLM (Large Language Model).

8. The information processing method according to claim 1, wherein the trained model is the same as the trained model used when performing the review.

9. An information processing system comprising: a processing unit that generates evaluation information, including an evaluation result for the checkpoints and at least one suggestion for improvement to the text indicating the checkpoints, based on checkpoints for reviewing a document and prompts indicating the evaluation of the checkpoints, using a trained model.

10. An information processing program that enables a computer to generate evaluation information, including evaluation results for the checkpoints and at least one suggested improvement for the text representing the checkpoints, based on checkpoints for reviewing a document and prompts indicating the evaluation of the checkpoints, using a trained model.

Images