Document review system and document review method

Abstract

This document review system reviews a review text on the basis of a checklist by a computer, and comprises: a determination criterion extraction unit that creates a determination criterion of review on the basis of the checklist defining check items for the review text and a past review result indicating a past review result by a reviewer; a determination criterion update unit that integrates determination criteria of a plurality of reviewers with respect to the checklist, detects a difference between the determination criteria, and updates the determination criterion; and a review unit that causes an artificial intelligence model to execute a review using the review text and the updated determination criterion.

Description

Document Review System and Document Review Method

[0001] The present invention relates to a document review system and a document review method.

[0002] There are many opportunities to review created materials in daily work. Review operations such as examination, verification, re-investigation, and review of created materials are required to achieve higher efficiency and stable accuracy. For review operations, there is an increasing demand for an automatic review system that reduces the burden on evaluators (humans) with varying review accuracy and stably provides high-precision review using artificial intelligence models such as LLM (Large Language Models).

[0003] Automatic review by LLM can obtain high-precision review result information with good responsiveness in a chat format or the like, and can be implemented by, for example, ChatGPT. However, judgment criteria for obtaining information on review results are defined in advance by experts.

[0004] Patent Document 1 discloses that in the work of checking deliverables in software development, the relevant parts of the deliverables are taken into consideration, the relationship between past checked parts and corrected parts in the current development is grasped, and this relationship is used to more appropriately support the checking work. Patent Document 1 also discloses that the necessity of checking work for check items is indicated by comparing with past check results for the check items.

[0005] Patent Document 2 discloses generating an easy-to-understand check list based on guideline documents. Patent Document 2 also discloses generating a check list (list of requirements) by determining the similarity between requirement sentences and non-requirement sentences (such as explanatory notes corresponding to descriptions of requirement sentences).

[0006] Japanese Patent Application Laid-open No. 2022-006426 Japanese Patent Application Laid-open No. 2021-047779

[0007] Traditionally, the criteria for automated reviews using LLM (Laboratory Review Management) were predefined by experts, and these criteria could vary due to differences in their experience. This resulted in inconsistencies in the criteria used for automated reviews, preventing consistently high-precision reviews. In particular, adding new check items required experts to manually create criteria for the new items, significantly increasing workload.

[0008] Therefore, the present invention aims to reduce the man-hours required to create judgment criteria, enable the formalization of judgment criteria, and provide highly accurate reviews stably.

[0009] To achieve the above objective, one aspect of the present invention is a document review system that uses a computer to review a review document based on a checklist, comprising: a judgment criterion extraction unit that creates judgment criteria for the review based on a checklist defining check items for the review document and past review results showing past review results by reviewers; a judgment criterion update unit that integrates the judgment criteria of multiple reviewers for the checklist, detects differences in the judgment criteria, and updates the judgment criteria; and a review unit that causes an artificial intelligence model to perform a review based on the review document and the updated judgment criteria.

[0010] According to the present invention, the man-hours required to create judgment criteria are reduced, the judgment criteria can be formalized, and highly accurate reviews can be reliably provided.

[0011] This figure shows the system configuration in one embodiment of the present invention. This figure shows the hardware configuration in one embodiment of the present invention. This figure shows the overall processing in one embodiment of the present invention. This figure shows the checklist in one embodiment of the present invention. This figure shows the processing of criterion extraction in one embodiment of the present invention. This figure shows past review results in one embodiment of the present invention. This figure shows review information in one embodiment of the present invention. This figure shows the processing of individual criterion extraction in one embodiment of the present invention. This figure shows the list of individual criterion lists in one embodiment of the present invention. This figure shows the processing of cluster-specific criterion extraction in one embodiment of the present invention. This figure shows the list of cluster-specific criterion lists in one embodiment of the present invention. This figure shows the overall criterion extraction process in one embodiment of the present invention. This figure shows the overall criterion list in one embodiment of the present invention. This figure shows the processing of criterion updates in one embodiment of the present invention. This figure shows the difference between individual criterion differences in one embodiment of the present invention. This figure shows the difference between cluster-specific criterion differences in one embodiment of the present invention. This figure shows the difference between overall criterion differences in one embodiment of the present invention. This figure shows the processing of criterion integration in one embodiment of the present invention. This figure shows the processing of criterion difference detection in one embodiment of the present invention. This figure shows the review process in one embodiment of the present invention. This figure shows the screen for criterion extraction used in one embodiment of the present invention. This figure shows the screen for criterion integration used in one embodiment of the present invention. This figure shows the screen for criterion difference detection used in one embodiment of the present invention. This figure shows the screen for reflecting criterion correction results used in one embodiment of the present invention. This figure shows a review screen used in one embodiment of the present invention.

[0012] Embodiments of the present invention will be described below with reference to the drawings. The following description and drawings are illustrative for illustrating the present invention, and have been omitted and simplified as appropriate for clarity of explanation. The present invention can also be carried out in various other forms. Unless otherwise specified, each component may be singular or plural.

[0013] The positions, sizes, shapes, and ranges of the components shown in the drawings may not represent their actual positions, sizes, shapes, and ranges in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, and ranges disclosed in the drawings.

[0014] In the following explanation, various types of information may be described using terms such as "database," "table," and "list," but these types of information may also be represented using data structures other than these. To indicate independence from data structures, "XX table," "XX list," etc., may be referred to as "XX information." When describing identification information, if terms such as "identification information," "identifier," "name," "ID," and "number" are used, these terms are interchangeable.

[0015] When there are multiple components with the same or similar function, they may be described using the same symbol but with different subscripts. However, if it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description.

[0016] Furthermore, while the following explanation may describe the processes performed by executing a program, the processor (e.g., CPU (Central Processing Unit), GPU (Graphics Processing Unit)) executes the defined processes, using memory resources (e.g., memory) and / or interface devices (e.g., communication ports) as appropriate; therefore, the processor may be the primary driver of the processing. Similarly, the primary driver of the processing performed by executing a program may be a controller, device, system, computer, or node having a processor. The primary driver of the processing performed by executing a program may be an arithmetic unit, and may include dedicated circuits that perform specific processing (e.g., FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)).

[0017] The program may be installed from the program source into a device such as a computer. The program source may be, for example, a program distribution server or a storage medium readable by a computer. If the program source is a program distribution server, the program distribution server includes a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to other computers. In addition, in the following description, two or more programs may be implemented as one program, or one program may be implemented as two or more programs. <Example 1> Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Figure 1 is a diagram showing the system configuration of this embodiment. The review system 1 in Figure 1 performs a review (verification) of the documents of the created materials.

[0018] The review system 1 includes an input / output unit 10, a judgment criterion extraction unit 12, a judgment criterion update unit 15, a review unit 16, and a communication unit 17. The review system 1 also includes a judgment criterion DB (database) 11, a past review results DB 13, a judgment criterion difference DB 14, a judgment criterion correction result DB 18, a review document DB 19, and a review result DB 20.

[0019] The input / output unit 10 receives information from the user 21 and outputs information to the user 21. The judgment criteria DB 11 stores information such as the checklist 110, individual judgment criteria list 111, cluster-specific judgment criteria list 112, and overall judgment criteria list 113.

[0020] The judgment criterion extraction unit 12 includes an individual judgment criterion extraction unit 121, a cluster-specific judgment criterion extraction unit 122, and an overall judgment criterion extraction unit 123.

[0021] The past review results DB13 stores information such as past review results 131 and review information 132. The judgment criterion difference DB14 stores information such as individual judgment criterion differences 141, cluster-specific judgment criterion differences 142, and overall judgment criterion differences 143.

[0022] The judgment criteria update unit 15 includes a judgment criteria integration unit 151, a judgment criteria difference detection unit 152, and a judgment criteria correction result reflection unit 153. The review unit 16 performs a review of the review document, which is the document to be reviewed. The communication unit 17 performs communication with the outside via the network.

[0023] The Judgment Criteria Revision Results DB18 stores information on the results of revising the judgment criteria. The Review Document DB19 stores the documents created for review. The Review Results DB20 stores information on the results of the review.

[0024] Here, the review system 1 for reviewing documents is implemented using a general-purpose computer and has the configuration shown in Figure 2. Figure 2 is a diagram showing the hardware configuration of this embodiment.

[0025] The computer 1600 has a main memory 2 having a CPU 1601 and memory 1602, and an external storage device 1603 that serves as an auxiliary storage device such as an HDD (Hard Disk Drive). The computer also has a reader 1607 for reading and writing information to portable storage media such as CDs (Compact Disks) and USB memory, an input device 1606 such as a scanner, keyboard, and mouse, and an output device 1605 such as a display.

[0026] Furthermore, the computer includes communication devices 1604 such as NICs (Network Interface Cards) for connecting to a communication network, and internal communication lines (referred to as system buses) 1609 such as system buses that connect these.

[0027] The various data stored in or used in the systems and devices shown in the embodiments can be realized by the CPU 1601 reading and using data from memory 1602 or external storage device 1603. Furthermore, each functional unit of each system and device can be realized by the CPU 1601 loading a predetermined program stored in external storage device 1603 into memory 1602 and executing it.

[0028] The aforementioned predetermined program is stored (downloaded) to the external storage device 1603 from the storage medium via the reading device 1607, or from the network via the communication device 1604. The stored (downloaded) program may then be loaded onto the memory 1602 and executed by the CPU 1601.

[0029] Alternatively, the data may be loaded directly onto the memory 1602 via the reading device 1607 from a storage medium, or via the communication device 1604 from a network, and then executed by the CPU 1601.

[0030] In the following, we will illustrate the case where computer 1600 is composed of a single computer. Computer 1600 may achieve similar functions by distributing all or part of these functions across one or more computers, such as a cloud, and communicating with each other via a network.

[0031] Figure 3A is a flowchart illustrating the overall process of review system 1 for reviewing documents.

[0032] In step S1, the judgment criteria extraction unit 12 obtains information for the checklist 110 from the instruction information entered by the user 21 into the input / output unit 10. The checklist 110 is a list-format information consisting of check IDs and check items, as shown in Figure 3B. The checklist 110 is information that defines the check items to be reviewed. The string checklist items are created from knowledge of check items and word groups from the review document, etc., that is to be reviewed.

[0033] Step S2 is the step in which the judgment criteria extraction unit 12 extracts the judgment criteria for the reviewer's review, such as the individual judgment criteria list 111, the cluster-specific judgment criteria list 112, and the overall judgment criteria list 113, and stores them in the judgment criteria DB 11.

[0034] Step S3 is the step in which the judgment criteria update unit 15 updates the judgment criteria. Step S4 is the step in which the review unit 16 reviews the review documents stored in the review document DB 19.

[0035] Figure 4A is a flowchart showing the process of extracting judgment criteria in the review system 1, and is a detailed processing flow of step S2 described above. The process of extracting judgment criteria is performed in the judgment criterion extraction unit 12 in Figure 1.

[0036] Step S21 is the step in which the judgment criteria extraction unit 12 acquires past review results 131 and review information 132 from the past review results DB 13.

[0037] Past review results 131 are information on past review results by experts and other reviewers in a list format, as shown in Figure 4B, consisting of review ID, check ID, judgment result, reason for judgment, location, and content.

[0038] Figure 4B shows, for example, the results of a check on three items identified by check IDs "1" to "3" as past review information of a reviewer identified by review ID "1". Among these, the item identified by check ID "1" shows that the reviewer judged the result to be "A" with the judgment reason "The specifications are described in Chapter 3". Furthermore, it shows that the location and content of the description that formed the basis of this judgment result was "XXX" in "Section 3.1". The judgment result can be a multi-value such as "A, B, C", a numerical value such as 0 to 10, or a binary value such as "OK, NG". In addition, the review information 132 is in list format, consisting of review ID, review date and time, reviewer, and priority, as shown in Figure 4C.

[0039] Review information 132 is information about the review. In Figure 4C, for example, review ID "1" indicates that the review date and time was "2025 / 1 / 10 10:00" and the reviewer was "Reviewer A". It also indicates that the priority is low, as will be explained later, with a priority of "0". In this way, review information 132 defines the date and time the reviewer performed the review and the priority level to indicate which reviewer's review should be given priority.

[0040] Step S22 is a step in which the individual determination criterion extraction unit 121 of the determination criterion extraction unit 12 extracts a check list 110 or the like, and outputs and stores an individual determination criterion list 111 in the determination criterion DB 11.

[0041] Step S23 is a step in which the cluster-based determination criterion extraction unit 122 of the determination criterion extraction unit 12 extracts the individual determination criterion list 111 or the like, and outputs and stores a cluster-based determination criterion list 112 in the determination criterion DB 11.

[0042] Step S24 is a step in which the overall determination criterion extraction unit 123 of the determination criterion extraction unit 12 extracts the cluster-based determination criterion list 112 or the like, and outputs and stores an overall determination criterion list 113 in the determination criterion DB 11. When the determination criterion extraction unit 12 outputs and stores the overall determination criterion 113, it ends the determination criterion extraction process.

[0043] FIG. 5A is a flow diagram showing the individual determination criterion extraction process of the review system 1, which is a detailed processing flow of the above-mentioned step S22. The individual determination criterion extraction process is performed by the individual determination criterion extraction unit 121 in FIG. 1.

[0044] Step S221 is a step in which the individual determination criterion extraction unit 121 acquires the check list 110 in the determination criterion DB 11, the past review result 131 in the past review result DB 13, and review information 132.

[0045] Step S222 is a step in which the individual determination criterion extraction unit 121 performs the processing of step S223 and step S224 for each acquired check ID and review ID.

[0046] Step S223 is a step in which the individual determination criterion extraction unit 121 transmits check items, determination results, determination reasons, description locations, description contents and the like to an artificial intelligence model such as an LLM via the communication unit 17, and the LLM automatically generates individual determination criterion items.

[0047] Step S224 is a step in which the individual determination criterion extraction unit 121 acquires individual determination criterion items from the LLM via the communication unit 17.

[0048] Step S225 is a step in which the individual determination criterion extraction unit 121 outputs and stores each of the acquired individual determination criterion items for each check ID and review ID into the individual determination criterion list 111 of the determination criterion DB 11.

[0049] The individual determination criterion list 111 is information in a list format configured with a check ID, a review ID, and individual determination criterion items as shown in FIG. 5B. The individual determination criterion list 111 is information including individual determination criterion items generated by an LLM based on the check list 110, the review ID of the past review result 131, and the review ID, and is information that defines determination criteria representing how each reviewer made a determination on a check item.

[0050] In FIG. 5B, for example, it shows that the reviewer identified by the review ID "1" made a determination on the check item identified by the check ID "1" based on the criterion that "if the specification is specifically described, it is A".

[0051] FIG. 6A is a flow diagram showing the processing of extracting cluster-specific determination criteria by the review system 1, and is a detailed processing flow of the aforementioned step S23. The processing of extracting cluster-specific determination criteria is performed by the cluster-specific determination criterion extraction unit 122 in FIG. 1.

[0052] Step S231 is a step in which the cluster-specific determination criterion extraction unit 122 acquires the check list 110 and the individual determination criterion list 111 from the determination criterion DB 11.

[0053] Step S232 is the step in which the cluster-specific judgment criterion extraction unit 122 clusters each check item by similarity. In clustering by similarity, the cluster-specific judgment criterion extraction unit 122, for example, vectorizes the strings of the check items, and if the similarity between the vectorized strings of the check items is greater than or equal to a predetermined value, these strings are judged as similar and assigned to the same cluster ID. In this way, the cluster-specific judgment criterion extraction unit 122 clusters similar check items and stores them together as a single item with a belonging check ID, so that even with new check items, check items with high similarity can be clustered as belonging check items.

[0054] Step S233 is a step in which the cluster-specific judgment criterion extraction unit 122 performs the processing of steps S234 and S235 for each assigned cluster ID and review ID.

[0055] Step S234 is a step in which the cluster-specific judgment criterion extraction unit 122 transmits each of the individual judgment criterion items for the same cluster ID to the LLM via the communication unit 17, and the LLM automatically generates cluster-specific judgment criterion items.

[0056] Step S235 is the step in which the cluster-specific judgment criterion extraction unit 122 obtains cluster-specific judgment criterion items from the LLM via the communication unit 17.

[0057] Step S236 is the step in which the cluster-specific judgment criterion extraction unit 122 outputs each of the acquired cluster-specific judgment criterion items to the cluster-specific judgment criterion list 112 of the judgment criterion DB 11 and saves them.

[0058] The cluster-specific judgment criteria list 112 is a list-format information consisting of cluster ID, member check ID, review ID, and cluster-specific judgment criteria item, as shown in Figure 6B. The cluster-specific judgment criteria list 112 is information that includes cluster-specific judgment criteria items generated by LLM based on checklist 110 and individual judgment criteria list 111, and defines the judgment criteria that show how reviewers determined that check items with high similarity belong to clustered groups.

[0059] In Figure 6B, for example, the cluster identified by cluster ID "1" is the check items identified by check IDs "1", "4", and "5", which were judged as "A if specifically described" by the reviewer identified by review ID "1". This indicates that these check items were classified as a single cluster due to their high similarity.

[0060] Figure 7A is a flowchart showing the process of extracting overall judgment criteria in the review system 1, and is a detailed processing flow of step S24 described above. The process of extracting overall judgment criteria is performed in the overall judgment criteria extraction unit 123 in Figure 1.

[0061] Step S241 is the step in which the overall judgment criterion extraction unit 123 obtains the cluster-specific judgment criterion list 112.

[0062] Step S242 is a step in which the overall judgment criteria extraction unit 123 performs the processing of steps S243 and S244 for each review ID.

[0063] Step S243 is a step in which the overall judgment criterion extraction unit 123 transmits the acquired cluster-specific judgment criterion items to the LLM via the communication unit 17, and the LLM automatically generates overall judgment criterion items.

[0064] Step S244 is the step in which the overall judgment criterion extraction unit 123 obtains overall judgment criterion items from the LLM via the communication unit 17.

[0065] Step S245 is the step in which the overall judgment criterion extraction unit 123 outputs each of the acquired overall judgment criterion items to the overall judgment criterion list 113 of the judgment criterion DB 11 and saves them.

[0066] The overall judgment criteria list 113 is a list-format information consisting of an overall criterion ID, a review ID, and an overall judgment criterion item, as shown in Figure 7B. Based on the cluster-specific judgment criteria list 112, the overall judgment criteria list 113 defines the judgment criteria for each reviewer when reviewing a review document as a whole, across all clusters of clustered check items.

[0067] Figure 7B shows, for example, that under overall criterion ID "1", the overall criterion item is stored as "If there is no description, rate it as C" for the reviewer identified by review ID "1", and under overall criterion ID "2", the overall criterion item is stored as "If there is a description such as 'Undecided, see attached document', rate it as B or higher" for the same reviewer.

[0068] Figure 8A is a flowchart showing the process of updating the judgment criteria in the review system 1, and is a detailed processing flow of step S3 described above. The judgment criteria extraction process is performed in the judgment criteria update unit 15 in Figure 1 if an update is necessary. If an update is not necessary, the judgment criteria extraction process is skipped.

[0069] Step S31 is the step in which the judgment criteria integration unit 151 of the judgment criteria update unit 15 integrates the judgment criteria.

[0070] Step S32 is the step in which the judgment criterion difference detection unit 152 of the judgment criterion update unit 15 detects a difference in the judgment criteria.

[0071] Step S33 is a step in which the judgment criterion difference detection unit 152 detects the difference in the judgment criteria and outputs the difference in the judgment criteria stored in the judgment criterion difference DB 14 to the user 21 in a manner that can be visualized via the input / output unit 10. The difference in the judgment criteria may also be transmitted to the LLM via the communication unit 17.

[0072] The differences in judgment criteria output to and saved in the judgment criterion difference DB14 include individual judgment criterion differences 141, cluster-specific judgment criterion differences 142, and overall judgment criterion differences 143.

[0073] The individual judgment criterion difference 141 is a list-format information consisting of a check ID, individual difference ID, difference, and multiple judgment criterion items (ID: 1, ID: 2, etc.), as shown in Figure 8B. The individual judgment criterion difference 141 is information on the difference in individual judgment criterion items between review IDs, as judged by the LLM based on the individual judgment criterion list 111. This difference can be said to be the difference in judgment criteria between reviewers regarding the check items.

[0074] In Figure 8B, for example, check ID "1" indicates that when the individual judgment criterion item for review ID "1" is "A if the specifications are specifically described" and the individual judgment criterion item for review ID "2" is "A if a document containing specific specifications is quoted," the difference is "the judgment when the specifications are described in the quoted source." In other words, the difference in judgment criteria for the check items between these reviewers is "the judgment when the specifications are described in the quoted source."

[0075] The cluster-specific judgment criterion difference 142 is a list-format information consisting of a cluster ID, check ID, cluster-specific difference ID, difference, and multiple judgment criterion items, as shown in Figure 8C. The cluster-specific judgment criterion difference 142 is information on the difference in cluster-specific judgment criterion items between review IDs, as judged by the LLM based on the cluster-specific judgment criterion list 112. This difference can be said to be the difference in judgment criteria among reviewers for clustered check items (groups of check items identified by their affiliated check IDs).

[0076] In Figure 8C, for example, cluster ID "1" corresponds to member check IDs "1, 4, and 5," and the cluster-specific judgment criterion item for review ID "1" is "B if only reference text is included," while the individual judgment criterion item for review ID "2" is "A if reference text is included." This shows that the difference is "judgment regarding the inclusion of reference text." In other words, the difference in judgment criteria for the clustered check items among these reviewers is "judgment regarding the inclusion of reference text."

[0077] The overall judgment criterion difference 143 is a list-format information consisting of an overall difference ID, the difference, and multiple judgment criterion items, as shown in Figure 8D. The overall judgment criterion difference 143 is information on the differences in cluster-specific judgment criterion items between review IDs judged by the LLM based on the overall judgment criterion list. This difference can be said to be the difference in judgment criteria among reviewers for the entire review document being reviewed, across all clusters of clustered check items.

[0078] Figure 8D shows, for example, that when the overall judgment criterion for review ID "1" is "B if there is no specific description" and the overall judgment criterion for review ID "2" is "A if there is a description", the difference is "how to handle cases where there is a description but no specific description". In other words, the difference in the overall judgment criteria for the review text between these reviewers is "how to handle cases where there is a description but no specific description".

[0079] Step S34 is the step in which the judgment criterion correction result reflection unit 153 of the judgment criterion update unit 15 obtains the judgment criterion correction result from the user 21 via the input / output unit 10. Alternatively, this judgment criterion correction result may be stored in the judgment criterion correction result DB 18, and the judgment criterion correction result reflection unit 153 may obtain the stored judgment criterion correction result. Alternatively, the judgment criterion correction result may be obtained from the LLM via the communication unit 17.

[0080] Step S35 is the step in which the judgment criterion correction result reflection unit 153 of the judgment criterion update unit 15 creates a review ID with a review information priority flag in the list of review information 132 in the past review result DB 13 upon acquisition of the judgment criterion correction result. For example, as shown in Figure 4C, the review information 132 is given priority when subsequent reviews are recreated, with "102" added as the review ID and "1" added as the priority flag.

[0081] Step S36 is the step in which the judgment criterion correction result reflection unit 153 adds the judgment criterion correction result as a new judgment criterion to the list of items to be corrected among the individual judgment criterion list 111, the cluster-specific judgment criterion list 112, and the overall judgment criterion list 113 of judgment criterion 11, and saves it. Then, the judgment criterion update unit 15 ends the judgment criterion update process when it saves the new judgment criterion.

[0082] In this way, the judgment criteria update unit 15 can obtain good new judgment criteria that enable stable and highly accurate reviews by avoiding inconsistencies and omissions such as when different reviewers have different judgment criteria, when the judgment criteria have ambiguous operation or rules, or when there are no clear applicable judgment criteria.

[0083] Figure 9 is a flowchart showing the process of integrating the judgment criteria of the review system 1, and is a detailed processing flow of step S31 described above. The judgment criteria integration process is performed in the judgment criteria integration unit 151 in Figure 1 when integration is necessary. Integration means grouping each record of the individual judgment criteria list 111 shown in Figure 5B by check ID, grouping each record of the cluster-specific judgment criteria list 112 shown in Figure 6B by cluster ID, and grouping each record of the overall judgment criteria list 113 shown in Figure 6C by review ID.

[0084] Step S3101 is the step in which the judgment criteria integration unit 151 obtains the individual judgment criteria list 111. Step S3102 is the step in which the judgment criteria integration unit 151 performs the processing of steps S3103 and S3104 for each check ID.

[0085] Step S3103 is the step in which the judgment criteria integration unit 151 transmits the acquired multiple individual judgment criteria items to the LLM via the communication unit 17 for automatic generation. Step S3104 is the step in which the judgment criteria integration unit 151 obtains the integration result of the multiple individual judgment criteria items of the target review ID (integrated individual judgment criteria items) from the LLM via the communication unit 17.

[0086] Step S3105 is the step in which the judgment criteria integration unit 151 outputs the integrated individual judgment criteria items to the individual judgment criteria list 111 and adds them.

[0087] Step S3106 is the step in which the decision criteria integration unit 151 obtains the cluster-specific decision criteria list 112. Step S3107 is the step in which the decision criteria integration unit 151 performs the processing of steps S3108 and S3109 for each cluster ID.

[0088] Step S3108 is the step in which the judgment criteria integration unit 151 transmits the acquired cluster-specific judgment criteria items to the LLM via the communication unit 17 for automatic generation. Step S3109 is the step in which the judgment criteria integration unit 151 obtains the integration result of the multiple cluster-specific judgment criteria items with different review IDs (integrated cluster-specific judgment criteria items) from the LLM via the communication unit 17.

[0089] Step S3110 is the step in which the judgment criteria integration unit 151 outputs the integrated cluster-specific judgment criteria items to the cluster-specific judgment criteria list 112 and adds them.

[0090] Step S3111 is the step in which the judgment criteria integration unit 151 acquires the overall judgment criteria list 113. Step S3112 is the step in which the judgment criteria integration unit 151 transmits the acquired overall judgment criteria items to the LLM via the communication unit 17 for automatic generation. Step S3113 is the step in which the judgment criteria integration unit 151 acquires the integration result of the overall judgment criteria items for the review ID (integrated overall judgment criteria items) from the LLM via the communication unit 17. Step S3114 is the step in which the judgment criteria integration unit 151 outputs the integrated overall judgment criteria items to the overall judgment criteria list and adds them.

[0091] Figure 10 is a flowchart showing the process of detecting the difference in the judgment criteria of the review system 1, and is a detailed processing flow of step S32 described above. The process of detecting the difference in the judgment criteria is performed by the judgment criterion difference detection unit 152 in Figure 1 when difference detection is required.

[0092] Step S3201 is the step in which the judgment criterion difference detection unit 152 obtains the individual judgment criterion list 111. Step S3202 is the step in which the judgment criterion difference detection unit 152 performs the processing of steps S3203 and S3204 for each check ID.

[0093] Step S3203 is the step in which the judgment criterion difference detection unit 152 transmits the acquired multiple individual judgment criterion items to the LLM via the communication unit 17 for automatic generation. Step S3204 is the step in which the judgment criterion difference detection unit 152 acquires the differences between the multiple individual judgment criterion items of the target review ID from the LLM via the communication unit 17.

[0094] Step S3205 is the step in which the judgment criterion difference detection unit 152 outputs the differences of individual judgment criterion items to the list of individual judgment criterion differences 141 in the judgment criterion difference DB 14 (Figure 8B) and saves them.

[0095] Step S3206 is the step in which the judgment criterion difference detection unit 152 obtains the cluster-specific judgment criterion list 112. Step S3207 is the step in which the judgment criterion difference detection unit 152 performs the processing of steps S3208 and S3209 for each cluster ID.

[0096] Step S3208 is the step in which the judgment criterion difference detection unit 152 transmits the acquired cluster-specific judgment criterion items to the LLM via the communication unit 17 for automatic generation. Step S3209 is the step in which the judgment criterion difference detection unit 152 obtains the differences between multiple cluster-specific judgment criterion items with different review IDs from the LLM via the communication unit 17.

[0097] Step S3210 is the step in which the judgment criterion difference detection unit 152 outputs the difference of the cluster-specific judgment criteria to the list of cluster-specific judgment criterion differences 142 in the judgment criterion difference DB 14 (Figure 8C) and saves it.

[0098] Step S3211 is the step in which the judgment criterion difference detection unit 152 acquires the overall judgment criterion list 113. Step S3212 is the step in which the judgment criterion difference detection unit 152 transmits the acquired overall judgment criterion items to the LLM via the communication unit 17 for automatic generation. Step S3213 is the step in which the judgment criterion difference detection unit 152 acquires the difference in the overall judgment criteria of the review ID from the LLM via the communication unit 17. Step S3214 is the step in which the judgment criterion difference detection unit 152 outputs the difference in the overall judgment criteria to the list of overall judgment criterion differences 143 (Figure 8D) in the judgment criterion difference DB 14 and saves it.

[0099] Figure 11 is a flowchart showing the review execution process of the review system 1, and is a detailed processing flow of step S4 in Figure 3A. The review execution process is performed in the review unit 16 in Figure 1.

[0100] Step S401 is the step in which the review unit 16 retrieves a review document from the review document DB 19. Step S402 is the step in which the review unit 16 retrieves a checklist 110 from the judgment criteria DB 11 and review information 132 from the past review results DB 13.

[0101] Step S403 is a step in which the review unit 16 determines whether the information of the priority item in the acquired review information 132 contains a predetermined value (for example, priority "1"). If the information of the priority item contains a predetermined value (step S403; YES), the process proceeds to step S404. If the information of the priority item does not contain a predetermined value (step S403; NO), the process proceeds to step S405.

[0102] Step S404 is the step in which the review unit 16 obtains a list of judgment criteria from the individual judgment criterion list 111, the cluster-specific judgment criterion list 112, and the overall judgment criterion list 113 in the judgment criterion DB, where the information of the priority item corresponds to a predetermined value (for example, the value of the priority item in Figure 4C is "1").

[0103] Step S405 is the step in which the review unit 16 obtains the latest list of judgment criteria from the individual judgment criterion list 111, the cluster-specific judgment criterion list 112, and the overall judgment criterion list 113 in the judgment criterion DB. The latest list of judgment criteria is obtained, for example, based on the latest review date and time information in the review information 132.

[0104] Step S406 is a step in which the review unit 16 performs the processing of steps S407 to S415 for each check ID.

[0105] Step S407 is the step in which the review unit 16 determines whether there is an individual judgment criterion item in the individual judgment criterion list 111 that corresponds to the check ID to be reviewed. If there is an individual judgment criterion item (Step S407; YES), proceed to step S408. If there is no individual judgment criterion item (Step S407; NO), proceed to step S409.

[0106] Step S408 is the step in which the review unit 16 obtains the content of the string of the individual judgment criterion item corresponding to the check ID of the item to be reviewed, and proceeds to step S412.

[0107] Step S409 is the step in which, if there are no individual judgment criterion items, the review unit 16 obtains the check ID that is most similar to the check item in the checklist 110. Obtaining a similar check ID involves, for example, vectorizing the string of the check item and determining strings that are greater than or equal to a predetermined value as similar, thereby obtaining the check ID. Then, step S410 is the step in which the review unit 16 obtains the cluster ID to which the most similar check ID belongs from the cluster judgment criteria 112. Furthermore, step S411 is the step in which the review unit 16 adds the check ID to be reviewed to the member check ID of the cluster ID obtained from the cluster judgment criteria 112, and proceeds to step S412.

[0108] Step S412 is a step in which the review unit 16 obtains a cluster-specific judgment criterion item in the cluster-specific judgment criterion 112 that has a check ID to be reviewed that corresponds to the affiliation check ID.

[0109] Step S413 is the step in which the review unit 16 obtains the overall judgment criterion items from the overall judgment criterion list 113.

[0110] Step S414 is the step in which the review unit 16 transmits the check items in the checklist 110, the review text in the review text DB 19, and the acquired judgment criteria to the LLM via the communication unit 17. The judgment criteria include individual judgment criteria items, cluster-specific judgment criteria items, overall judgment criteria items, etc. Step S415 is the step in which the review unit 16 obtains the automatically generated review results from the LLM via the communication unit 17. Step S416 is the step in which the review unit 16 saves the review results to the review result DB 20 and outputs a visualized version to the user 21 via the input / output unit 10.

[0111] Thus, Review System 1 reduces the effort required to create judgment criteria, enables the formalization of judgment criteria, and allows for the stable provision of highly accurate reviews.

[0112] Furthermore, the review results are added to the past review results DB13. If the visualized review results are incorrect, the review results may be corrected, and task-level and question-level judgment criteria may be added to improve the accuracy of the results in the next review.

[0113] Figure 12 shows an example of the screen of the input / output unit 10 during the judgment criterion extraction process of the review system 1 (Figures 3A and 4A). In the judgment criterion extraction process, the left side of the screen in Figure 12 shows the input from the input / output unit 10, and the right side of the screen shows the output from the input / output unit 10.

[0114] User 21 selects a checklist from the dropdown menu on the left side of the screen, where checklists stored in the judgment criteria DB 11 are displayed, and enters the information. At this time, the "Judgment Criteria Extraction Results" on the right side of the screen is not displayed. If no checklist to select exists, User 21 adds a new checklist using "Add New Checklist". If User 21 selects a checklist using "Select Checklist" and then adds a new checklist using "Add New Checklist", the review system 1 creates a new checklist by combining the two.

[0115] Additionally, if there are any previously reviewed results, User 21 enters the file name of the result in "Add Previous Review Results" on the left side of the screen and adds it. If there are no previously reviewed results, User 21 leaves the field blank. If the field is left blank, the system creates a blank file named Past Review Results 131 and Review Information 132.

[0116] Then, user 21 presses the "Extract Judgment Criteria" button to process the extraction of judgment criteria. Note that in review system 1, "Add New Checklist" is not displayed when "Select Checklist" is selected on the left side of the screen, but "Add New Checklist" may be displayed simultaneously when "Select Checklist" is selected.

[0117] Subsequently, the review system 1 processes the information entered by user 21 via the input / output unit 10 to extract judgment criteria, and sequentially displays the "Individual Judgment Criteria List," "Cluster-Specific Judgment Criteria List," and "Overall Judgment Criteria List" on the right side of the screen under "Judgment Criteria Extraction Results." If the results of the judgment criterion extraction are favorable, user 21 presses the "Register Judgment Criteria" button for each list. If unfavorable, user 21 presses the "Cancel" button to exit without registering the judgment criteria.

[0118] Review System 1 displays the "Individual Judgment Criteria List" but does not display the "Cluster-Specific Judgment Criteria List" or the "Overall Judgment Criteria List." Instead, it displays the "Cluster-Specific Judgment Criteria List" only after the "Judgment Criteria Registration" for the "Individual Judgment Criteria List" has been completed.

[0119] Note that the "Register Judgment Criteria" button in Review System 1 may be pressed after the three types of lists—"Individual Judgment Criteria List," "Cluster-Specific Judgment Criteria List," and "Overall Judgment Criteria List"—have been displayed.

[0120] Figure 13 shows an example of the screen of the input / output unit 10 during the judgment criterion integration process of the review system 1 (Figures 8A and 9). In the judgment criterion integration process, the left side of the screen in Figure 13 shows the input from the input / output unit 10, and the right side of the screen shows the output from the input / output unit 10.

[0121] User 21 selects and enters information for a checklist stored in the Judgment Criteria DB 11 from the "Checklist Selection" section on the left side of the screen, where the checklists are displayed in a dropdown format. Similarly, user 21 selects and enters information for a reviewer stored in the Judgment Criteria DB 11 from the "Review Selection" section on the left side of the screen. Finally, user 21 presses the "Integrate Judgment Criteria" button to proceed with the judgment criteria integration process.

[0122] Subsequently, the review system 1 processes the integration of judgment criteria based on the information input by user 21 via the input / output unit 10, and sequentially displays the "Integrated Judgment Criteria List," "Cluster-Specific Judgment Criteria List," and "Overall Judgment Criteria List" in the "Integrated Judgment Criteria Results" on the right side of the screen.

[0123] Then, if the result of the integrated judgment criteria is good, user 21 presses the "Register Integrated Result" button for each list. If it is bad, they press the "Cancel" button and exit without registering the judgment criteria.

[0124] Review System 1 displays the "Individual Judgment Criteria List" but does not display the "Cluster-Specific Judgment Criteria List" or the "Overall Judgment Criteria List." Instead, it displays the "Cluster-Specific Judgment Criteria List" only after the "Judgment Criteria Registration" for the "Individual Judgment Criteria List" has been completed.

[0125] Note that the "Register Judgment Criteria" button in Review System 1 may be pressed after the three types of lists—"Individual Judgment Criteria List," "Cluster-Specific Judgment Criteria List," and "Overall Judgment Criteria List"—have been displayed.

[0126] Figure 14 shows an example of the screen of the input / output unit 10 during the judgment criterion difference detection process of the review system 1 (Figures 8A and 10). In judgment criterion difference detection, the left side of the screen in Figure 14 shows the input from the input / output unit 10, and the right side of the screen shows the output from the input / output unit 10.

[0127] User 21 selects and enters the desired checklist from the dropdown menu on the left side of the screen, where the checklists stored in the judgment criteria DB 11 are displayed. Similarly, on the left side of the screen, user 21 selects and enters the desired reviewer from the review information stored in the judgment criteria DB 11. Finally, user 21 presses the "Detect Judgment Criteria Differences" button to initiate the judgment criteria difference detection process.

[0128] Subsequently, the review system 1 processes the information entered by user 21 to detect differences in judgment criteria, and sequentially displays "Individual Judgment Criteria Differences," "Cluster-Specific Judgment Criteria Differences," and "Overall Judgment Criteria Differences" in the "Judgment Criteria Difference Detection Results" on the right side of the screen (step S33).

[0129] Then, user 21 presses the "Download Judgment Criteria Difference Detection Results" button to download the judgment criterion difference detection results, and also registers the revised judgment criteria as shown in Figure 15.

[0130] Figure 15 shows an example of the screen of the input / output unit 10 during the processing of the judgment criterion revision results of the review system 1 (Figure 8A). When the judgment criterion revision results are reflected, the left side of the screen in Figure 15 is inherited from the left side of the screen in Figure 14.

[0131] Unlike the right side of the screen in Figure 14, the right side of the screen in Figure 15 displays the "Individual Judgment Criteria List," "Cluster-Specific Judgment Criteria List," and "Overall Judgment Criteria List" in an editable format.

[0132] User 21 modifies the "Individual Judgment Criteria Items," "Cluster-Specific Judgment Criteria Items," and "Overall Judgment Criteria Items" as needed, and presses the "Apply Modification Results" button. The review system 1 then obtains the results of the judgment criterion modification (S34).

[0133] Figure 16 shows an example of the screen of the input / output unit 10 during the review process (Figure 11) of the review system 1. During the review process, the left side of the screen in Figure 16 shows the input from the input / output unit 10, and the right side of the screen shows the output to the input / output unit 10.

[0134] User 21 selects and enters the desired checklist from the dropdown menu on the left side of the screen, where the checklists stored in the judgment criteria DB 11 are displayed. Additionally, on the left side of the screen, User 21 enters the filename of the review document to be reviewed under "Add Review Target File". Finally, User 21 presses the "Execute Review" button to begin the review process.

[0135] Subsequently, the review system 1 executes the review process based on the information input by user 21 via the input / output unit 10, and displays the review results in "Review Results" on the right side of the screen. Then, user 21 downloads the review results by pressing the "Download Review Results" button, and the review system 1 saves the review results to the review results DB 20 (S416).

[0136] Thus, according to this embodiment, checklist items are clustered, judgment criteria are created by summarizing the results of each cluster using LLM, and for new items, only the judgment criteria for the corresponding cluster can be provided.

[0137] The above describes this embodiment, which discloses at least the following configuration.

[0138] (1) A document review system that uses a computer to review a review document based on a checklist, comprising: a judgment criteria extraction unit that creates judgment criteria for the review based on the checklist which defines the items to be checked for the review document and past review results which show past review results by reviewers; a judgment criteria update unit that integrates the judgment criteria of multiple reviewers for the checklist, detects the differences in the judgment criteria and updates the judgment criteria; and a review unit that causes an artificial intelligence model to perform a review based on the review document and the updated judgment criteria.

[0139] (2) A document review system as described in (1) above, wherein the judgment criteria extraction unit has an individual judgment criteria extraction unit that causes an artificial intelligence model to create individual judgment criteria items that define judgment criteria that show how each reviewer judged the check items from the check items of the checklist.

[0140] (3) A document review system as described in (2) above, wherein the judgment criterion extraction unit has a cluster-specific judgment criterion extraction unit that causes an artificial intelligence model to create cluster-specific judgment criterion items that define judgment criteria for clustering highly similar check items among the check items judged by the reviewer from a plurality of individual judgment criterion items.

[0141] (4) A document review system described in (3) above, wherein the cluster-specific judgment criterion extraction unit extracts a plurality of individual judgment criterion items that have similar descriptions from the individual judgment criterion items in the individual judgment criterion list, and causes an artificial intelligence model to create the cluster-specific judgment criterion items from the plurality of extracted individual judgment criterion items.

[0142] (5) A document review system as described in (3) above, wherein the judgment criteria extraction unit has an overall judgment criteria extraction unit that causes an artificial intelligence model to create overall judgment criteria items that define the judgment criteria for each reviewer when reviewing the entire review document, from the cluster-specific judgment criteria items.

[0143] (6) A document review system described in (5) above, wherein the overall judgment criterion extraction unit extracts multiple cluster-specific judgment criterion items with similar descriptions from the cluster-specific judgment criterion items in the cluster-specific judgment criterion list, and causes an artificial intelligence model to create the overall judgment criterion item from the extracted multiple cluster-specific judgment criterion items.

[0144] (7) A document review system as described in (1) above, wherein the judgment criteria update unit has a judgment criteria integration unit that integrates multiple judgment criteria items.

[0145] (8) A document review system as described in (7) above, wherein the judgment criteria update unit includes a judgment criterion difference detection unit that detects the difference between the judgment criteria shown in each judgment criterion item from a plurality of judgment criterion items, and a judgment criterion correction result reflection unit that reflects the result of the judgment criterion correction for the difference in the judgment criteria in the judgment criterion item.

[0146] (9) A document review system as described in (3) above, wherein the review unit, if there is no individual judgment criterion item corresponding to the checklist, obtains the cluster-specific judgment criterion item that is most similar to the check items of the checklist, and causes the artificial intelligence model to perform a review based on the most similar cluster-specific judgment criterion item.

[0147] (10) A document review method in which a computer reviews a review document based on a checklist, wherein a judgment criterion extraction unit creates judgment criteria for the review based on the checklist which defines the check items for the review document and past review results which show past review results by reviewers, a judgment criterion update unit integrates the judgment criteria of multiple reviewers for the checklist, detects the differences in the judgment criteria and updates the judgment criteria, and a review unit has an artificial intelligence model perform a review based on the review document and the updated judgment criteria.

[0148] The present invention is not limited to the embodiments described above, and in the implementation stage, the components can be modified and implemented without departing from the gist of the invention, or the multiple components disclosed in the above embodiments can be appropriately combined.

[0149] 1: Review system, 10: Input / Output unit, 11: Judgment criteria DB (database), 12: Judgment criteria extraction unit, 13: Past review results DB, 14: Judgment criteria difference DB, 15: Judgment criteria update unit, 16: Review unit, 17: Communication unit, 18: Judgment criteria correction result DB, 19: Review text DB, 20: Review result DB, 21: User, 110: Checklist, 111: Individual judgment criteria list, 112: Cluster-specific judgment criteria list, 113: Overall judgment criteria list, 121: Individual judgment criteria extraction unit, 122: Cluster-specific judgment criteria extraction unit, 123: Overall judgment criteria extraction unit, 131: Past review results, 132: Review information, 141: Individual judgment criteria difference, 142: Cluster-specific judgment criteria difference, 143: Overall judgment criterion difference, 151: Judgment criterion integration unit, 152: Judgment criterion difference detection unit, 153: Judgment criterion correction result reflection unit.

Claims

1. A document review system that uses a computer to review a review document based on a checklist, comprising: a judgment criterion extraction unit that creates judgment criteria for the review based on the checklist which defines the items to be checked for the review document and past review results which show past review results by reviewers; a judgment criterion update unit that integrates the judgment criteria of multiple reviewers for the checklist, detects the differences in the judgment criteria, and updates the judgment criteria; and a review unit that causes an artificial intelligence model to perform a review based on the review document and the updated judgment criteria.

2. A document review system according to claim 1, wherein the judgment criteria extraction unit has an individual judgment criteria extraction unit that causes an artificial intelligence model to create individual judgment criteria items that define judgment criteria representing how each reviewer judged the check items from the check items of the checklist.

3. A document review system according to claim 2, wherein the judgment criterion extraction unit has a cluster-specific judgment criterion extraction unit that causes an artificial intelligence model to create cluster-specific judgment criterion items that define judgment criteria for clustering highly similar check items among the check items judged by the reviewer from a plurality of individual judgment criterion items.

4. A document review system according to claim 3, wherein the cluster-specific judgment criterion extraction unit extracts a plurality of individual judgment criterion items with similar descriptions from the individual judgment criterion items in the individual judgment criterion list, and causes an artificial intelligence model to create the cluster-specific judgment criterion items from the plurality of extracted individual judgment criterion items.

5. A document review system according to claim 3, wherein the judgment criteria extraction unit has an overall judgment criteria extraction unit that causes an artificial intelligence model to create overall judgment criteria items that define the judgment criteria for reviewing the entire review document for each reviewer from the cluster-specific judgment criteria items.

6. A document review system according to claim 5, wherein the overall judgment criterion extraction unit extracts a plurality of cluster-specific judgment criterion items with similar descriptions from the cluster-specific judgment criterion items in the cluster-specific judgment criterion list, and causes an artificial intelligence model to create the overall judgment criterion item from the extracted plurality of cluster-specific judgment criterion items.

7. A document review system according to claim 1, wherein the judgment criteria update unit has a judgment criteria integration unit that integrates a plurality of judgment criteria items.

8. A document review system according to claim 7, wherein the judgment criterion update unit includes a judgment criterion difference detection unit that detects the difference in judgment criteria shown in each judgment criterion item from a plurality of judgment criterion items, and a judgment criterion correction result reflection unit that reflects the result of the judgment criterion correction for the difference in judgment criteria in the judgment criterion items.

9. A document review system according to claim 3, wherein, if there are no individual judgment criterion items corresponding to the checklist, the review unit obtains the cluster-specific judgment criterion item that is most similar to the check items of the checklist, and causes the artificial intelligence model to perform a review based on the most similar cluster-specific judgment criterion item.

10. A document review method in which a computer reviews a review document based on a checklist, comprising: a judgment criterion extraction unit creating judgment criteria for the review based on the checklist defining the check items for the review document and past review results showing past review results by reviewers; a judgment criterion update unit integrating the judgment criteria of multiple reviewers for the checklist, detecting differences in the judgment criteria, and updating the judgment criteria; and a review unit having an artificial intelligence model perform a review based on the review document and the updated judgment criteria.

Images