Script creation device and script creation program

Abstract

A script creation device according to one aspect of the present invention comprises an input identification unit, a variable-including variable identification unit, a procedure definition unit, and a script generation unit. The input identification unit identifies input points in operation processes from an operation log, and the variable identification unit identifies reference variables that refer to information about other operations in the operation processes from the relationship between the operation processes at the input points. A variable-including procedure definition unit defines variables related to operation targets and values in the operation processes on the basis of the reference variables, and generates an operation procedure for operation processes including the defined variables. The script generation unit automatically generates a script corresponding to the operation processes on the basis of the operation procedure.

Description

Script Creation Device and Script Creation Program

[0001] One aspect of this invention relates to a script creation device and a script creation program.

[0002] In recent years, the demand for automating operation processes in office work has been increasing.

[0003] In this automation of office work operations, RPA (Robotic Process Automation) products are used. In RPA, a function of automatically generating an automation script by generative AI (Artificial Intelligence) etc. is also realized. For example, in prompt engineering of generative AI, when a prompt written with an operation target, procedures, etc. is created and an instruction is given to the generative AI, a script embedded with assumed variables is automatically generated.

[0004] Conventionally, as one of the RPA products, WinActor (registered trademark) has been provided. As disclosed in Non-Patent Document 1, in version 7.5 of this WinActor, when a prompt for a script to be created is described in natural language, a function of automatically generating a script using generative AI is realized.

[0005] ”WinActor version7.5 Generative AI Linkage”, [Online], [Retrieved on November 21, Reiwa 6], Internet <URL: https: / / winactor.biz / update / 2024 / 07 / 16_6591.html>

[0006] For automating the operation process in office work, it is necessary to accurately identify all “variables” indicating inputs during the operation process. However, conventional RPA products cannot identify the variables required according to the actual operation process. For example, in actual operations, input processing in operations that people perform unconsciously (e.g., inputs during login and form search) is also required as a variable, but conventional RPA products do not handle such inputs as variables.

[0007] Thus, conventional RPA products could not identify the variables required according to the actual operation process, and therefore could not automatically generate scripts that were usable in actual operations—in other words, highly accurate scripts. Consequently, conventionally, it was necessary to manually modify the automatically generated scripts by adding variables. There is a demand for technology that can eliminate this significant manual effort of script modification and automatically generate highly accurate scripts.

[0008] This invention was made in view of the above circumstances, and aims to provide a script generation technology that enables the automatic generation of highly accurate scripts.

[0009] To solve the above problems, a script creation device according to one aspect of this invention comprises an input identification unit, a variable identification unit, a variable-inclusive procedure definition unit, and a script generation unit. The input identification unit identifies input locations in an operation process from an operation log, and the variable identification unit identifies reference variables that refer to information about other operations in an operation process based on the relationships between operation processes at the input locations. The variable-inclusive procedure definition unit defines variables related to the operation target and values ​​in the operation process based on the reference variables and generates an operation procedure for the operation process that includes the defined variables. The script generation unit automatically generates a script according to the operation process based on the operation procedure.

[0010] According to one aspect of this invention, prompts that take into account variables corresponding to the actual operator's operation process can be automatically generated, thereby providing a script generation technology that enables the automatic generation of highly accurate scripts.

[0011] Figure 1 is a block diagram showing an example of the configuration of a business system including a first embodiment of a script creation device according to one embodiment of the present invention. Figure 2 is a block diagram showing an example of the hardware configuration of the script creation device according to the first embodiment. Figure 3 is a block diagram showing an example of the configuration of a business system including a second embodiment of a script creation device according to one embodiment of the present invention. Figure 4 is a block diagram showing an example of the hardware configuration of the script creation device according to the second embodiment. Figure 5 is a block diagram showing an example of the software configuration of a script creation device according to one embodiment of the present invention. Figure 6 is a diagram showing an example of an operation log. Figure 7 is a flowchart showing an example of the processing procedure and processing content of the automatic script creation process executed by the control unit of the script creation device. Figure 8 is a flowchart showing an example of the processing procedure and processing content of the input identification process during the automatic script creation process in Figure 7. Figure 9 is a diagram showing the processing result of the input identification process for the operation log in Figure 6. Figure 10 is a schematic diagram illustrating an example of a method for identifying input locations when there is no label for input operations in the operation log. Figure 11 is a flowchart showing an example of the processing procedure and processing content of the reference input identification process during the automatic script creation process in Figure 7. Figure 12 is a diagram showing the processing result of the reference input identification process for the processing result of the input identification process in Figure 9. Figure 13 is a schematic diagram illustrating an example of a method for determining the match between a screen capture image and an input value. Figure 14 is a flowchart showing an example of the processing procedure and content of the external input identification process during the automatic script creation process in Figure 7. Figure 15 is a diagram showing the processing result of the external input identification process in relation to the processing result of the reference input identification process in Figure 12. Figure 16A is the first part of a series of flowcharts showing an example of the processing procedure and content of the variable-containing procedure definition process during the automatic script creation process in Figure 7. Figure 16B is the second part of a series of flowcharts showing an example of the processing procedure and content of the variable-containing procedure definition process during the automatic script creation process in Figure 7. Figure 17 is a diagram showing an example of the processing result of the variable-containing procedure definition process in relation to the processing result of the external input identification process in Figure 15. Figure 18 is a flowchart showing an example of the processing procedure and content of the script generation process during the automatic script creation process in Figure 7.Figure 19 shows an example of a prompt generated in the script generation process from the processing result of the variable-containing procedure definition process in Figure 15. Figure 20A shows the first part of the script generated in the script generation process from the prompt in Figure 19. Figure 20B shows the second part of the script generated in the script generation process from the prompt in Figure 19.

[0012] Hereinafter, an embodiment of this invention will be described with reference to the drawings.

[0013] (Configuration Example) Figure 1 is a block diagram showing an example of the configuration of a business system including a first embodiment of the script creation device 1 according to one embodiment of the present invention. The script creation device 1 of the first embodiment is configured within a user terminal UT. The user terminal UT is connected to a service provider server SS that provides various services and a cloud storage CS that stores various data files via a network NW such as the Internet or a LAN (Local Area Network). The user terminal UT is an information processing device such as a PC (Personal Computer), smartphone, or tablet terminal. The script creation device 1 acquires an operation log showing the operation process on the user terminal UT and generates a script based on the acquired operation log. The operation process on the user terminal UT includes input operations on GUI (Graphical User Interface) screens provided by various application programs on the user terminal UT and Web screens provided by the service provider server SS. The operation process also includes file upload operations to the cloud storage CS and file download operations from the cloud storage CS.

[0014] Figure 2 is a block diagram showing an example of the hardware configuration of the script creation device 1 in the first embodiment. Note that the script creation device 1 in the first embodiment may use the hardware configuration of the user terminal UT as is. In this case, the script creation device 1 will reside within the user terminal UT by executing the script creation program on the user terminal UT. Therefore, the script creation device 1 in the first embodiment can be rephrased as the user terminal UT.

[0015] The script creation device 1 of the first embodiment includes a control unit 11, to which a storage unit having a program storage unit 12 and a data storage unit 13, a communication interface unit 14, and an input / output interface unit 15 are connected via a bus 16. In Figure 2, "interface" is abbreviated as "IF".

[0016] The control unit 11 is a hardware processor such as a CPU (Central Processing Unit). For example, by using a multi-core and multi-threaded CPU, multiple information processing operations can be performed simultaneously. The control unit 11 may also be equipped with multiple hardware processors.

[0017] The communication interface unit 14, under the control of the control unit 11, transmits and receives information with other devices using a communication protocol defined by the network NW.

[0018] An input device 171 and an output device 172 are connected to the input / output interface unit 15. Note that the script creation device 1 does not necessarily have to have all of these devices; some of them may be connected as external devices.

[0019] The input device 171 is used by the user to input instructions and information necessary for the operation of the user terminal UT. If the user terminal UT is a PC, the input device 171 may include, for example, a keyboard, a mouse, a touchpad or other pointing device. If the user terminal UT is a smartphone or tablet, the input device 171 may include, for example, a touch panel or operation buttons located on the display screen of the output device 172. The input device 171 may also include a reader device for reading information from a recording medium that stores various information such as programs and data. Furthermore, the input device 171 may also include a microphone for acquiring sound, and a camera for capturing images, such as a webcam or a general video camera.

[0020] The output device 172 includes a display, such as an LCD monitor, an OLED (Electroluminescence) monitor, a projector screen, or a head-mounted display, for displaying various information generated by the user terminal UT. The output device 172 may also include a speaker for transmitting various information to the user by sound. Furthermore, the output device 172 may include a writer device for writing the information generated by the user terminal UT to a recording medium.

[0021] The program storage unit 12 is configured by combining, for example, a non-volatile memory that can be written to and read at any time, such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or EEPROM (Registered Trademark) (Electric Erasable Programmable Read-Only Memory), with a non-volatile memory such as ROM (Read Only Memory). In addition to middleware such as an OS (Operating System), the program storage unit 12 stores script creation programs, which are application programs necessary for the user terminal UT to operate as the script creation device 1 of the first type. Hereafter, the OS and each application program will be collectively referred to as a program.

[0022] The data storage unit 13 combines, for example, a non-volatile memory that can be written to and read at any time, such as an HDD, SSD, EEPROM, or memory card, with a volatile memory such as RAM (Random Access Memory), as a storage medium.

[0023] In addition to pre-storing the script creation program in the program storage unit 12, the script creation program may also be read from the storage medium containing the script creation program and stored in the program storage unit 12 when needed. Alternatively, the script creation program may be downloaded from the service provision server SS, cloud storage CS, or other application provisioning device and stored in the program storage unit 12.

[0024] Figure 3 is a block diagram showing an example of the configuration of a business system including a second embodiment of the script creation device 1 according to one embodiment of the present invention. The script creation device 1 in the second embodiment is configured as a server computer connected to a user terminal UT via a network NW. The script creation device 1 in the second embodiment acquires operation logs from the user terminal UT via the network NW, indicating the operation process on the user terminal UT, and generates a script based on the acquired operation logs. The user terminal UT can include information processing devices such as PCs, smartphones, and tablet terminals, as well as on-site work recording devices such as smart glasses. Furthermore, the script creation device 1 in the second embodiment may generate a script based not only on the operation log of a single user terminal UT, but also on an operation log that combines the operation logs of multiple user terminal UTs. However, when combining the operation logs of multiple user terminal UTs, it is necessary to combine them in chronological order of the operations.

[0025] Figure 4 is a block diagram showing an example of the hardware configuration of the script creation device 1 of the second embodiment. The script creation device 1 of the second embodiment has the same hardware configuration as the script creation device 1 of the first embodiment, except that it does not have an input / output interface unit 15, an input device 171, and an output device 172.

[0026] Figure 5 is a block diagram showing an example of the software configuration of a script creation device 1 according to one embodiment of the present invention. In Figure 5, the communication interface unit 14 and the input / output interface unit 15 are collectively shown as the interface (IF) unit.

[0027] The control unit 11, as a processing function unit according to the first embodiment of this invention, includes an operation log acquisition unit 111, an input identification unit 112, a reference input identification unit 113, an external input identification unit 114, a variable-containing procedure definition unit 115, a script generation unit 116, and a script output unit 117. All of these processing function units 111 to 117 are realized by having the hardware processor of the control unit 11 execute a script creation program according to the first embodiment of this invention, which is stored in the program storage unit 12. At least one or at least a part of the processing functions of the above processing function units 111 to 117 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (field-programmable gate array), or GPU (Graphics Processing Unit) instead of being realized by the script creation program and the hardware processor of the control unit 11.

[0028] Furthermore, the data storage unit 13 includes, in its storage area, an operation log storage unit 131, an input identification result storage unit 132, a reference input identification result storage unit 133, an external input identification result storage unit 134, a variable-containing procedure storage unit 135, and a script storage unit 136, as storage units necessary for carrying out the first embodiment.

[0029] In the script creation device 1 of the first embodiment, the operation log acquisition unit 111 acquires an operation log indicating the operation process based on the user operation of the input device 171 acquired via the input / output interface unit 15 and the content presented to the user by the output device 172 via the input / output interface unit 15. In the script creation device 1 of the second embodiment, the operation log acquisition unit 111 acquires an operation log indicating the operation process at the user terminal UT, which is transmitted from the user terminal UT via the network NW and received by the communication interface unit 14. The operation log acquisition unit 111 stores the acquired operation log in the operation log storage unit 131.

[0030] Figure 6 shows an example of an operation log. This example is a PC operation log. The following section describes an example of treating this operation log as operation process information and automatically generating variable information.

[0031] An operation log is data that automatically acquires operation history information at the time of the operator's operation. As shown in Figure 6, an operation log includes, for example, the date, the operating terminal, the operation screen title and / or the screen URL (Uniform Resource Locator), the operation location information on the screen, the operation type, the value, and a screen capture. An operation log is treated as the operation history of one person at a specific time, including operations with multiple records, and input is expected to be one or more operation logs.

[0032] When an input identification unit 112 receives a script creation instruction, it performs an input identification process. In the script creation device 1 of the first embodiment, the script creation instruction is given from the input device 171 via the input / output interface unit 15. In the script creation device 1 of the second embodiment, the script creation instruction is transmitted from the user terminal UT via the network NW and received by the communication interface unit 14. In the input identification process, the input identification unit 112 identifies input locations in the operation process from the operation log stored in the operation log storage unit 131. The input identification unit 112 adds an input mark to the operation log for the record in which the input location has been identified, indicating that the input location has been identified. The input identification unit 112 stores the operation log with this input mark added as the input identification result in the input identification result storage unit 132.

[0033] The reference input identification unit 113 identifies reference inputs that are entered by referencing information from other operations in an operation process, based on the relationship between operation processes at the input location, based on the input identification results stored in the input identification result storage unit 132, and identifies the variables entered in those reference inputs (hereinafter referred to as reference variables). The reference input identification unit 113 adds labels (source record and referenced record) indicating the source and referenced operation processes of the identified reference variable to the source and referenced operation process records in the operation log of the input identification result. The reference input identification unit 113 stores the operation log with these labels added as the reference input identification result in the reference input identification result storage unit 133.

[0034] The external input identification unit 114 determines that input operations other than reference input operations are external input operations based on the reference input identification results stored in the reference input identification result storage unit 133, and identifies the variables entered in these external input operations as external input variables. The external input identification unit 114 adds a label indicating that it is an external input operation to the record of the input operation identified as an external input operation in the operation log of the reference input identification results. The external input identification unit 114 stores the operation log with this label added as an external input identification result in the external input identification result storage unit 134.

[0035] The variable-encoded procedure definition unit 115 defines variables related to the target of operation and values ​​according to the characteristics of the identified information, and generates an operation procedure that includes the defined variables. Specifically, the variable-encoded procedure definition unit 115 defines variables related to external input, reference input values, the screen URL of the target of operation, and operation position information based on the operation log of the external input identification result stored in the external input identification result storage unit 134. Then, the variable-encoded procedure definition unit 115 stores the text data of the procedure with these defined variables embedded in it as a variable-encoded procedure in the variable-encoded procedure storage unit 135.

[0036] The script generation unit 116 generates a prompt based on a variable-containing procedure stored in the variable-containing procedure storage unit 135. Furthermore, the script generation unit 116 inputs this prompt to a text generation AI such as a large-scale language model (hereinafter abbreviated as LLM), causing the LLM to generate a script. The script generation unit 116 retrieves the script generated by the LLM and stores it in the script storage unit 136.

[0037] The script output unit 117 outputs the script stored in the script storage unit 136 when a generated script is stored in the script storage unit 136, or when an output instruction for a script stored in the script storage unit 136 is given. In the first embodiment of the script creation device 1, the script output instruction is given from the input device 171 via the input / output interface unit 15. The script output unit 117 then presents the script stored in the script storage unit 136 to the user via the output device 172 through the input / output interface unit 15. In the second embodiment of the script creation device 1, the script output instruction is transmitted from the user terminal UT via the network NW and received by the communication interface unit 14. The script output unit 117 then transmits the script stored in the script storage unit 136 to the user terminal UT via the network NW using the communication interface unit 14.

[0038] (Example of Operation) Next, an example of the operation of the script creation device 1 according to one embodiment configured as described above will be explained. Figure 7 is a flowchart showing an example of the processing procedure and processing content of the automatic script creation process executed by the control unit 11 of the script creation device 1. Here, it is assumed that the operation log of the user terminal UT has already been acquired and stored in the operation log storage unit 131, and in the following explanation, the stored operation log will be assumed to be the operation log example shown in Figure 6. The automatic script creation process shown in this flowchart is started in response to a script creation instruction. It goes without saying that the control unit 11 can simultaneously execute the operation log acquisition process by the operation log acquisition unit 111 at the same time as this automatic script creation process.

[0039] As shown in FIG. 7, the control unit 11 of the script creation device 1 first operates as an input specifying unit 112 to perform an input specifying process, thereby specifying an input location during the operation process (step S1). Details of this input specifying process will be described later.

[0040] Next, the control unit 11 operates as a reference input specifying unit 113 to perform a reference input specifying process, thereby specifying a variable for referring to information of other operations during the process from the relevance between processes at the input location (step S2). Details of this reference input specifying process will be described later.

[0041] Next, the control unit 11 operates as an external input specifying unit 114 to perform an external input specifying process, thereby determining an input operation that is not a reference input operation as an "external input operation" (step S3). Details of this external input specifying process will be described later.

[0042] Next, the control unit 11 operates as a variable-containing procedure definition unit 115 to perform a variable-containing procedure definition process, thereby generating a variable-containing procedure in which variables regarding external input, values regarding reference input, the screen URL of the operation target, and operation position information are embedded (step S4). Details of this variable-containing procedure definition process will be described later.

[0043] Next, the control unit 11 operates as a script generation unit 116 to perform a script generation process, thereby generating a prompt based on the variable-containing procedure and further generating a script (step S5). Details of this script generation process will be described later.

[0044] Then, the control unit 11 ends the script automatic creation process shown in this flowchart. Note that the control unit 11 can subsequently operate as a script output unit 117 in response to a script output instruction to output the generated script.

[0045] Hereinafter, details of each process will be described. FIG. 8 is a flowchart showing an example of the process procedure and process content of the input specifying process executed in step S1 above.

[0046] First, the control unit 11 reads out the processing target operation log designated as the processing target by the script creation instruction from the operation logs stored in the operation log storage unit 131 (step S11).

[0047] Next, the control unit 11 adds an input field to the read processing target operation log (step S12).

[0048] Then, the control unit 11 saves the operation log after adding the input field in the input identification result storage unit 132 (step S13). As a result, the operation log already stored in the input identification result storage unit 132 is overwritten by this new operation log. Of course, the stored content of the input identification result storage unit 132 may be deleted at the start of this input identification process.

[0049] FIG. 9 is a diagram showing the processing result of this input identification process for the operation log in FIG. 6. As shown in FIG. 9, an input field is added to the operation log shown in FIG. 6 by the processing of steps S11 to S13.

[0050] The control unit 11 performs a process of identifying the input locations during the operation process from the operation log with the input field added, which is the processing target stored in the input identification result storage unit 132.

[0051] Specifically, the control unit 11 first determines the total number L of operation log records, which is the number of records included in the operation log stored in the input identification result storage unit 132 (step S14).

[0052] Thereafter, the control unit 11 resets the value of a counter n (not shown) provided in the control unit 11 or provided in the data storage unit 13 to "1" (step S15).

[0053] The control unit 11 then determines whether the operation process of record n, which is the nth record indicated by the value of counter n, is an input operation (step S16). An input operation can be identified simply by checking whether the value field of the corresponding record n ​​in the operation log contains information, or conversely, whether it is blank. However, depending on the target system, it is quite possible that it may be difficult to determine whether the value is blank or not. Therefore, it is desirable to identify input operations not only from the value but also from the characteristics of the operation process.

[0054] For example, an input operation is identified based on whether or not the operation type is an input operation. Specifically, if the operation type of the corresponding record n ​​in the operation log includes a label indicating an input operation to a GUI component, etc., acquired during the operation, the control unit 11 determines that the operation process of that record is an input operation. For example, in the operation log shown in Figure 9, the operation types of records 2, 5, and 7 include the label "input," which indicates that they are input operations, so the control unit 11 can determine that the operation processes of these records are input operations.

[0055] Furthermore, if the corresponding record n ​​in the operation log does not contain a label for an input operation to a GUI component, etc., acquired during the operation, the control unit 11 may determine whether the operation process is an input operation by using LLM or the like to determine the area and value that appear to be in the input operation area from the image file recorded in the screen capture field of the corresponding record n.

[0056] Figure 10 is a schematic diagram illustrating an example of a method for identifying input locations using LLM, etc., when there is no input operation label in the operation log. As shown in Figure 10, the control unit 11 inputs a screen capture image 100 and a prompt 101 instructing the LLM to determine the location and value of the input operation from the image 100. The control unit 11 then receives an LLM response 102 indicating whether or not an input operation was determined by the LLM, and, if an input operation was performed, the location and value of the input operation.

[0057] Alternatively, if the corresponding record n ​​in the operation log does not contain a label for the input operation, the control unit 11 may, if the information in the screen URL field of the corresponding record n ​​and the information in the screen URL field of the record immediately preceding the corresponding record n ​​(record n-1) are the same, take the difference between the images in the two image files recorded in the screen capture fields of both records, detect whether there has been a change in input, and thereby identify the location and value of the input operation.

[0058] If the operation process for the corresponding record n ​​is an input operation, the control unit 11 adds an input operation mark to the input field of the corresponding record n ​​in the operation log stored in the input identification result storage unit 132 (step S17). For example, in the operation log shown in Figure 9, since the operation processes for records 2, 5, and 7 are determined to be input operations, white circles are added as input operation marks to the input fields of those records. Note that the white circles are just an example, and the input operation mark can of course be any other symbol or any value. Also, if the corresponding record n ​​in the operation log does not contain an input operation label, and the input location and value are determined by LLM or the like, the control unit 11 may add the determined input location to the operation location information field of the corresponding record and add the determined value as the input value to the value field.

[0059] Subsequently, or if it is determined in step S16 that the operation process for the record n ​​is not an input operation, the control unit 11 increments the value of counter n by "+1" (step S18).

[0060] Then, the control unit 11 determines whether the value of counter n has become greater than the total number of operation log records L included in the operation log (step S19). If the value of counter n is less than or equal to the total number of operation log records L, the control unit 11 proceeds to the process in step S16 and repeats the processes in steps S16 to S19 for the next record.

[0061] Then, if the value of counter n becomes greater than the total number of operation log records L, the control unit 11 will terminate the input identification process shown in this flowchart and proceed to the next step S2, the reference input identification process.

[0062] Figure 11 is a flowchart showing an example of the processing procedure and processing content of the reference input identification process performed in step S2 above.

[0063] First, the control unit 11 reads the operation log of the input identification process result stored in the input identification result storage unit 132 (step S21).

[0064] Next, the control unit 11 adds a reference / external input field to the read operation log (step S22).

[0065] Then, the control unit 11 saves the operation log after the addition of the reference / external input field to the reference input identification result storage unit 133 (step S23). As a result, the operation log already stored in the reference input identification result storage unit 133 is overwritten by this new operation log. Of course, the contents stored in the reference input identification result storage unit 133 may also be deleted at the start of this reference input identification process.

[0066] Figure 12 shows the processing result of this reference input identification process applied to the operation log of the input identification process result in Figure 9. As shown in Figure 12, a reference / external input field is added to the operation log shown in Figure 9 by the processing in steps S21 to S23.

[0067] The control unit 11, based on the operation log stored in the reference input identification result storage unit 133, which adds reference / external input fields to the input identification results, determines reference inputs that are entered by referencing information from other operations in the operation process, based on the relationship between operation processes at the input location, and performs a process to identify the reference variable, which is the variable entered in that reference input. This process estimates the relationship between text values ​​between operation processes based on the property that "operations related as reference inputs have the same text value on the screen (for example, a document screen or a report screen)," and treats it as a reference input if a relationship is deemed to exist. Note that the screen of the referenced operation and the screen of the input operation may be the same screen or different screens. Also, the value displayed on the screen and the input value do not necessarily have to be an exact match, and regular expressions or whether they are probabilistically similar may be used as judgment conditions.

[0068] Specifically, the control unit 11 first determines the total number of input operation records E, which is the number of records in the operation log stored in the reference input identification result storage unit 133 that have an input operation mark assigned to the input field (hereinafter referred to as an input operation record) (step S24).

[0069] Subsequently, the control unit 11 resets the value of a counter m (not shown) located within the control unit 11 or within the data storage unit 13 to "1" (step S25).

[0070] Then, the control unit 11 reads the input value entered in the value field of the m-th input operation record, which is indicated by the value of counter m in the record contained in the operation log stored in the reference input identification result storage unit 133 (step S26).

[0071] Furthermore, the control unit 11 reads images from the screen capture fields of x records preceding the m-th input operation record from the records included in the operation log stored in the reference input identification result storage unit 133 (step S27). Here, x is any natural number greater than or equal to 1.

[0072] The control unit 11 then determines whether there are any parts in the screen capture images of the x previous records that have been read that match the text data of the input value of the input operation record (step S28). If applicable, the matching of the screen capture image and the text data of the input value can be determined by obtaining the text data by performing OCR (Optical Character Recognition) processing on the image and matching the acquired text data with the text data of the input value. Alternatively, the matching determination method may be determined by LLM or the like.

[0073] Figure 13 is a schematic diagram illustrating an example of a matching determination method using a screen capture image and an LLM (Language Memory Module) of input values. As shown in Figure 13, the control unit 11 inputs a screen capture image (or object data on the screen of the screen capture) 103 and a prompt 104 instructing the LLM to obtain text data that matches a specified string, which is the text data of the input value, from the image 103. The control unit 11 then receives an LLM response 105 indicating the matching relationship between the text data determined by the LLM and the specified string in that text data.

[0074] If the read screen capture image contains a portion that matches the input value text data, that is, if a screen capture image is found that contains a portion that matches the input value text data of the input operation record, the control unit 11 adds a reference operation label and a source record number to the reference / external field of the mth input operation record in the operation log stored in the reference input identification result storage unit 133 (step S29). Here, the source record is the record that contains the screen capture image in which the matching portion was found. For example, in the operation log shown in Figure 12, if the text data that matches the input value "P200" entered in the value field of record 5 is found in the image file "cap04.png" entered in the screen capture field of record 4, then the reference operation label "Reference" and "(from: 4)" indicating that the source record is record 4 are added to the reference / external field of record 5.

[0075] Furthermore, the control unit 11 adds a reference operation label and the record number of the m-th input operation record, which is the referenced record, to the reference / external field of the reference source record (step S210). For example, in the operation log shown in Figure 12, the reference / external field of record 4, which is the reference source record, has a reference operation label of "Reference" and "(to: 5)" indicating that the referenced record is record 5. It should be noted that the description of the reference operation label and the reference source and referenced records shown in Figure 12 is just one example and is not limited to this.

[0076] Subsequently, or if it is determined that there is no part in the screen capture image read in step S28 that matches the input text data, the control unit 11 increments the value of counter m by "+1" (step S211).

[0077] The control unit 11 then determines whether the value of counter m has become greater than the total number of input operation records E (step S212). If the value of counter m is less than or equal to the total number of input operation records E, the control unit 11 proceeds to the process in step S26 and repeats the processes in steps S26 to S212 for the next record.

[0078] Then, if the value of counter m becomes greater than the total number of input operation records E, the control unit 11 will terminate the reference input identification process shown in this flowchart and proceed to the next step S3, the external input identification process.

[0079] Figure 14 is a flowchart showing an example of the processing procedure and processing content of the external input identification process performed in step S3 above.

[0080] First, the control unit 11 copies the operation log of the reference input identification process result stored in the reference input identification result storage unit 133 to the external input identification result storage unit 134 (step S31). As a result, the operation log already stored in the external input identification result storage unit 134 is overwritten by this new operation log. Of course, the contents stored in the external input identification result storage unit 134 may also be deleted at the start of this external input identification process.

[0081] The control unit 11 determines that input operations other than reference input operations are external input operations based on the operation log corresponding to the reference input identification result stored in the external input identification result storage unit 134, and then performs the process of identifying the variables entered in these external input operations as external input variables.

[0082] Specifically, the control unit 11 extracts records from the operation log stored in the external input identification result storage unit 134 in which an input operation mark is assigned to the input field, but a reference operation label is not assigned to the reference / external field (step S32).

[0083] Then, the control unit 11 assigns an external operation label to the reference / external column of each record corresponding to the extracted record in the operation log stored in the external input identification result storage unit 134 (step S33).

[0084] Figure 15 shows the processing result of this external input identification process for the operation log of the reference input identification process result in Figure 12. As shown in Figure 15, the external operation label "External" is added to the reference / external field of record 2, which is a record in which an input operation mark is assigned to the input field but a reference operation label is not assigned to the reference / external field. It should be noted that the description of the external operation label shown in Figure 15 is just an example and is not limited to this.

[0085] Then, the control unit 11 completes the external input identification process shown in this flowchart and proceeds to the next step S4, the variable-entry procedure definition process.

[0086] Figures 16A and 16B are flowcharts showing an example of the processing steps and content of the variable-containing procedure definition process performed in step S4. This variable-containing procedure definition process defines variables for external inputs, reference inputs, the screen URL of the target, and operation position information based on the operation log of the external input identification result stored in the external input identification result storage unit 134, that is, according to the information of the labeled external operation input and reference operation (reference record and referenced record), and the text data of the procedure into which these defined variables are embedded is made into a variable-containing procedure.

[0087] In this variable-based procedure definition process, screen URLs are defined as the same variable if they are determined to be identical. Similarly, operation position information is defined as the same variable if the screen URLs are determined to be identical and the operation position information is also identical. Input values ​​are not treated as the same variable even if they are the same as those in other operation processes, i.e., other records. However, in the case of reference input, unlike input values, they are treated as the same variable if the data is determined to be identical. In the case of a referenced record, if the operation position information can be identified, the variable definition is performed in the same way as for the input operation of the referenced record.

[0088] If the position of a specific string cannot be obtained in a PDF file or similar, the value obtained by the reference input identification process may be directly expressed as text in the procedure within the script instead of being defined as a variable value. Alternatively, instead of using the application's object information, text information obtained by OCR processing from a screen capture or similar source may be directly written as text in the procedure within the script, or defined as a new variable and expressed as a procedure.

[0089] Furthermore, in this variable-inclusive procedure definition process, the variable names should be different text names for the screen URL, operation position information, and value. While the variable definitions are intended to be text values, you may also define data types that correspond to the data format of the value field.

[0090] In this variable-input procedure definition process, the control unit 11 first deletes the contents stored in the variable-input procedure storage unit 135 (step S41).

[0091] Subsequently, the control unit 11 extracts all records with operations from the operation log stored in the external input identification result storage unit 134 (step S42). Records with operations are those with a label assigned to the reference / external field.

[0092] The control unit 11 classifies the extracted records, that is, classifies the input operations, based on the screen URL entered in the screen URL field of the extracted records (step S43). For example, if the operation log is as shown in Figure 15, the control unit 11 classifies it into three input operations.

[0093] Then, the control unit 11 determines the total number of classifications C, which is the number of classifications of input operations classified in step S43 (step S44). For example, if the operation log is as shown in Figure 15, the control unit 11 determines that C = 3.

[0094] Subsequently, the control unit 11 resets the values ​​of counters i and j (not shown), which are provided within the control unit 11 or within the data storage unit 13, to "1" (step S45).

[0095] Then, the control unit 11 defines a variable for the screen URL for each category where the screen URLs are the same, by defining a variable urli for the screen URL of the i-th category (step S46). For example, in the case of an operation log as shown in Figure 15, the control unit 11 defines a variable url1 as a variable for the first screen URL "http: / / hogehoge.ryokin.html".

[0096] Furthermore, the control unit 11 defines operation position information according to the number of different operation position information in the i-th classification screen URL (step S47). As mentioned above, if there is one operation position information in the same screen URL, they are defined as the same variable. For example, in the case of an operation log as shown in Figure 15, there are four different operation position information in the classification of the first screen URL, so the control unit 11 defines four operation position information variables: loc1, oc2, loc3, and loc4.

[0097] Next, the control unit 11 determines the total number of inputs Ii for the i-th classification screen URL (step S48). For example, in the operation log shown in Figure 15, there are 3 input marks in the input field for the first classification screen URL, so the control unit 11 determines that Ii = 3.

[0098] Subsequently, the control unit 11 resets the value of a counter k (not shown) located within the control unit 11 or within the data storage unit 13 to "1" (step S49).

[0099] Then, the control unit 11 defines the j-th input value, indicated by the value of counter j, as the variable varj among the ii input values ​​for the i-th classification screen URL (step S410).

[0100] Next, the control unit 11 determines whether or not a source record exists (step S411). In other words, it determines whether or not the record corresponding to the j-th input value is the referenced record. This determination can be made based on the presence or absence of a reference flag in the reference / external field of the record in question.

[0101] If a reference record exists, the control unit 11 defines the screen URL of the reference record as a variable (step S412).

[0102] Furthermore, the control unit 11 defines the operation position information of the referenced record as a variable (step S413). If the operation position information of the referenced record cannot be identified, the control unit 11 sets empty data. In this way, if the operation position information can be identified, the variable is defined in the same way as the input operation.

[0103] Then, the control unit 11 makes the value of the referenced record the same as the variable for the j-th operation (step S414). In this way, in the case of reference input, the input value of the referenced record is treated as the same variable as the input value of the referenced record.

[0104] Subsequently, or if it is determined in step S411 that there is no source record, the control unit 11 increments the values ​​of counters j and k by "+1" (step S415).

[0105] Then, the control unit 11 determines whether the value of counter k has become greater than the total number of inputs Ii (step S416). If the value of counter k is less than or equal to the total number of inputs Ii, the control unit 11 proceeds to the process in step S410 and repeats the processes in steps S410 to S416 for the next input value.

[0106] Then, if the value of counter k becomes greater than the total number of inputs Ii, the control unit 11 generates a procedure definition with variables for screen URL, operation position information, input value, and reference input based on the definitions implemented up to this point (step S417). Here, the control unit 11 reads a procedure definition with variables already stored in the procedure storage unit 135 and updates the read procedure definition with variables based on the definitions implemented up to this point to generate a new procedure definition with variables.

[0107] The control unit 11 updates the variable-containing procedure definition stored in the variable-containing procedure storage unit 135 by overwriting the generated variable-containing procedure definition in the variable-containing procedure storage unit 135 (step S418). Figure 17 shows an example of a variable-containing procedure definition, which is the result of the variable-containing procedure definition processing for the processing result of the external input identification processing in Figure 15. This variable-containing procedure definition is generated as text data containing the procedure and variable information. The position information shown in the example of Figure 17 (coordinates of the upper left (x1, y1), coordinates of the lower right (x2, y2)) was not specifically explained, but was obtained in step S417 using a known method. For example, position information can be obtained by performing structural analysis from an image capture of screen information or from the object structure contained in the image information. In the case of an image capture, data containing a specified string is searched for in the data structure structured by document structure analysis, and the coordinate information contained in that data is obtained. When obtaining from the object information of the screen, an object containing a specified string is searched for, and the object information is obtained by API to obtain the position information contained in the object.

[0108] Subsequently, the control unit 11 increments the value of counter i by "+1" (step S419).

[0109] Then, the control unit 11 determines whether the value of counter i has become greater than the total number of classifications C (step S420). If the value of counter i is less than or equal to the total number of classifications C, the control unit 11 proceeds to the process in step S46 and repeats the processes in steps S46 to S420 for the next screen URL, i.e., the i-th classification indicated by the value of counter i.

[0110] Then, if the value of counter i becomes greater than the total number of classifications C, the control unit 11 will terminate the variable-input procedure definition process shown in this flowchart and proceed to the next step S5, the script generation process.

[0111] Figure 18 is a flowchart showing an example of the processing procedure and content of the script generation process during the automatic script creation process executed in step S5 above.

[0112] First, the control unit 11 reads the variable-containing procedure stored in the variable-containing procedure storage unit 135 (step S51).

[0113] Next, the control unit 11 creates a prompt based on the read variable-containing procedure (step S52). Figure 19 shows an example of a prompt generated from the processing result of the variable-containing procedure definition process in Figure 15. As shown in Figure 19, the control unit 11 generates a prompt that instructs the generation of a program to perform automatic operations according to the procedure and variable information described in the variable-containing procedure.

[0114] Then, the control unit 11 generates a script (step S53). Specifically, the control unit 11 inputs the generated prompt to the LLM, etc., and receives a script returned as a response from the LLM, etc. Figures 20A and 20B show a series of scripts generated from the prompt in Figure 19.

[0115] The control unit 11 saves this script to the script storage unit 136 (step S54).

[0116] Then, the control unit 11 terminates the script generation process shown in this flowchart. This completes the automatic script creation process shown in the flowchart of Figure 7.

[0117] (Function and Effects) As described above, the script creation device 1 according to one embodiment uses an input identification unit 112 to identify input locations in the operation process from the operation log, and a variable identification unit to identify reference variables that refer to information of other operations in the operation process based on the relationships between operation processes at the identified input locations. The reference input identification unit 113 and the external input identification unit 114 are examples of this variable identification unit. The script creation device 1 then uses a variable-containing procedure definition unit 115 to define variables related to the operation target and value in the operation process based on the identified reference variables, generates an operation procedure for the operation process that includes these defined variables, and the script generation unit 116 automatically generates a script corresponding to the operation process based on the generated operation procedure. According to the script creation device 1 according to this embodiment, prompts that take into account variables corresponding to the actual operator's operation process can be automatically generated, thus providing a script generation technology that enables the automatic generation of highly accurate scripts.

[0118] In one embodiment of the script creation device 1, the variable identification unit includes a reference input identification unit 113 that identifies a reference input that is entered by referring to information of other operations in an operation process based on the relationship between operation processes at the input location, and identifies a reference variable that is entered by that reference input, and an external input identification unit 114 that identifies variables entered by input operations other than reference inputs in the operation process as external input variables, and a variable-containing procedure definition unit 115 defines variables related to the operation target and value in the operation process based on these reference variables and external input variables. According to this embodiment of the script creation device 1, it becomes possible to automatically generate prompts that take into account conditional branching, reference inputs and external input variables according to the actual operation process.

[0119] Furthermore, in the script creation device 1 according to one embodiment, the variable-containing procedure definition unit 115 generates text data describing the operation procedure of an operation process including the defined variables, and the script generation unit 116 generates prompts for a text generation AI such as LLM based on the text data describing the operation procedure, and obtains a script from the text generation AI by inputting the generated prompts to the text generation AI. With the script creation device 1 according to this embodiment, a script can be easily created by generating a script from the operation procedure of an operation process and using LLM or the like.

[0120] However, this invention is not limited to the embodiments described above.

[0121] The flow of each process described with reference to the flowchart is not limited to the procedures described. For example, the processes in step S14 and step S15 shown in Figure 8 may be performed in reverse order, or they may be performed simultaneously. In this way, the order of some steps may be changed, and some steps may be performed simultaneously. Furthermore, the processing content of some steps may be modified.

[0122] Furthermore, the script creation program stored in the program storage unit 12 may be transferred while stored in the computer constituting the script creation device 1, or it may be transferred while not stored in the computer. In the latter case, the script creation program may be transferred via a network NW, or it may be transferred while recorded on a recording medium. The recording medium is a non-temporary tangible medium. The recording medium is a computer-readable medium. The recording medium can be any medium that is capable of storing programs and is readable by a computer, such as a CD-ROM or memory card, and its form is not limited.

[0123] Although embodiments of this invention have been described in detail above, the above description is merely illustrative in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of this invention. In other words, when implementing this invention, specific configurations may be adopted as appropriate depending on the embodiment.

[0124] In short, this invention is not limited to the embodiments described above, and in the implementation stage, the components can be modified and implemented without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined.

[0125] 1...Script creation device 11...Control unit 12...Program storage unit 13...Data storage unit 14...Communication interface unit 15...Input / output interface unit 16...Bus 100, 103...Image 101, 104...Prompt 102...LLM response 111...Operation log acquisition unit 112...Input identification unit 113...Reference input identification unit 114...External input identification unit 115...Procedure definition unit with variables 116...Script generation unit 117...Script output unit 131...Operation log storage unit 132...Input identification result storage unit 133...Reference input identification result storage unit 134...External input identification result storage unit 135...Procedure storage unit with variables 136...Script storage unit 171...Input device 172...Output device NW...Network SS...Service server UT...User terminal

Claims

1. A script creation device comprising: an input identification unit that identifies input locations in an operation process from an operation log; a variable identification unit that identifies reference variables that refer to information about other operations in the operation process based on the relationships between the operation processes at the input locations; a variable-containing procedure definition unit that defines variables relating to the operation target and value in the operation process based on the reference variables and generates an operation procedure for the operation process that includes the defined variables; and a script generation unit that automatically generates a script corresponding to the operation process based on the operation procedure.

2. The script creation device according to claim 1, wherein the variable identification unit includes: a reference input identification unit that identifies a reference input that is input by referring to information of other operations in the operation process based on the relationship between the operation processes at the input location, and identifies the reference variable which is a variable input by that reference input; and an external input identification unit that identifies a variable that is input by an input operation other than the reference input in the operation process as an external input variable, and the variable-containing procedure definition unit defines the variables relating to the operation target and value in the operation process based on the reference variable and the external input variable.

3. A script creation device according to claim 1 or 2, wherein the variable-containing procedure definition unit generates text data describing the operation procedure of the operation process including the defined variables, the script generation unit generates prompts for a text generation AI based on the text data describing the operation procedure, and obtains the script from the text generation AI by inputting the generated prompts to the text generation AI.

4. A script creation program that causes a processor to execute the following: a process to identify input locations in an operation process from an operation log; a process to identify reference variables that refer to information about other operations in the operation process based on the relationships between the operation processes at the input locations; a process to define variables relating to the operation target and value in the operation process based on the reference variables, and generate an operation procedure for the operation process that includes the defined variables; and a process to automatically generate a script corresponding to the operation process based on the operation procedure.

Images