Information processing device, information processing method, program, and recording medium

Abstract

This information processing device comprises: an acquisition means for acquiring a test specification indicating details of a test and for acquiring, from a test subject, screen configuration information and image information regarding a screen; an extraction means for extracting a test operation on the basis of the details of the test indicated in the test specification; a conversion means for converting information regarding an operation target in the test operation, the screen configuration information regarding each element on the screen, and the image information regarding each element on the screen into respective vectors using a machine learning model that performs embedding; and an identification means for identifying, on the basis of the information regarding the operation target, the screen configuration information, and the image information converted into the vectors, an element corresponding to the operation target in the test operation from among elements on the screen.

Description

Information Processing Apparatus, Information Processing Method, Program, and Recording Medium

[0001] The present invention relates to an information processing apparatus, an information processing method, a program, and a recording medium.

[0002] Conventionally, in the development of systems, applications, etc., tests are conducted to determine whether the developed systems, applications, etc. operate normally. For example, in order to confirm whether a developed Web system operates normally, predetermined inputs are made on the screen according to a test scenario, and tests are conducted to determine whether it operates correctly based on changes in the displayed content and screen transitions. Also, when functions are added after the release of the system, it is similarly necessary to conduct operation confirmation tests. Since it is a huge workload for the person in charge to conduct these tests manually, tools, etc. that automatically perform the work related to testing have been developed.

[0003] Patent Document 1 proposes a technique for acquiring screen information of an application to be tested and test information of the application, deriving correspondence relationship information according to the display positions of the operation target objects and element names included in the acquired screen information based on the acquired screen information, and outputting an execution script for testing the application based on the derived correspondence relationship information and the element names included in the test information.

[0004] Japanese Unexamined Patent Application Publication No. 2023 - 75780

[0005] In Patent Document 1, a concise specification of the operation target in the test is made by the element name, and the operation target may not be uniquely determined by only the element name. In that case, manual correction by a human was required. An object of the present invention is to provide an information processing apparatus, an information processing method, a program, and a recording medium capable of uniquely specifying the operation target in the test.

[0006] The information processing device according to the present invention is characterized by comprising: a test specification document indicating the content of a test; an acquisition means for acquiring screen configuration information and image information relating to a screen from a test target; an extraction means for extracting test operations based on the content of the test indicated in the test specification document; a conversion means for converting information on the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen into vectors using a machine learning model that performs embedding; and an identification means for identifying the element that is the target of operation in the test operation from among the elements on the screen based on the information on the target of operation, the screen configuration information, and the image information converted into vectors.

[0007] According to the present invention, it is possible to provide an information processing device, an information processing method, a program, and a recording medium that can uniquely identify the target of operation in a test.

[0008] Figure 1 is a diagram illustrating an example of the configuration of an information processing system. Figure 2 is a diagram illustrating an example of the hardware configuration of an information processing device. Figure 3 is a diagram illustrating an example of the functional configuration of an information processing device in the first embodiment. Figure 4A is a diagram illustrating an example of processing of an information processing device in the first embodiment. Figure 4B is a diagram illustrating an example of score calculation processing. Figure 5 is a diagram illustrating the division of test operations. Figure 6A is a diagram illustrating the description of test operations. Figure 6B is a diagram illustrating the description of test operations. Figure 6C is a diagram illustrating the description of test operations. Figure 7 is a diagram illustrating the description of test operations. Figure 8 is a diagram illustrating another example of score calculation processing. Figure 9 is a diagram illustrating an example of the functional configuration of an information processing device in the second embodiment. Figure 10 is a diagram illustrating an example of processing of an information processing device in the second embodiment. Figure 11 is a diagram illustrating an example of a test case overview. Figure 12 is a diagram illustrating an example of prompts related to the generation of test operation descriptions. Figure 13A is a diagram illustrating an example of prompts related to the generation of test operation descriptions. Figure 13B is a diagram illustrating an example of prompts related to the generation of test operation descriptions. Figure 14 is a diagram illustrating an example of test operation description data. Figure 15 illustrates an example of prompts related to generating (modifying) test operation descriptions. Figure 16A illustrates an example of prompts related to generating (modifying) test operation descriptions. Figure 16B illustrates an example of prompts related to generating (modifying) test operation descriptions. Figure 17 illustrates an example of prompts related to generating (modifying) test operation descriptions. Figure 18A illustrates an example of prompts related to generating (modifying) test operation descriptions. Figure 18B illustrates an example of prompts related to generating (modifying) test operation descriptions.

[0009] Embodiments of the present invention will be described below with reference to the drawings.

[0010] <First Embodiment> Figure 1 is a diagram showing an example of the configuration of an information processing system in one embodiment of the present invention. The information processing system in this embodiment includes an information processing device 101 and a server device 102. The information processing device 101 and the server device 102 are connected to each other so as to be able to communicate via a network 103. The network 103 only needs to be able to connect the information processing device 101 and the server device 102 so as to be able to communicate, and its type is not particularly limited.

[0011] The information processing device 101 automatically executes tests to verify whether the system or application under test operates correctly, based on a test specification document that outlines the test content. The server device 102 is a server device that provides the system or application under test. Here, as an example, the information processing device 101 automatically executes tests on the system (application) under test provided by the server device 102, based on a test specification document related to those tests.

[0012] The information processing device 101 acquires the test specification, screen configuration information, and image information related to the screen of the system (application) under test, identifies the elements to be operated on the screen based on these, and performs operations according to the test content. The information processing device 101 also verifies the validity of the results obtained from the screen operations, such as whether the results are the expected results shown in the test specification, and captures and records the screen as evidence.

[0013] The following explanation uses a web screen test of the system (application) under test provided by the server device 102 as an example. However, it is not limited to web screens and can be applied to any system or application where operations can be performed on the elements (UI, etc.) displayed on the screen by the system (application) under test. Furthermore, although the example shows the information processing device 101 that performs the test and the server device 102 that provides the system or application under test as separate devices, the information processing device 101 that performs the test may also be configured to have the system or application under test.

[0014] Figure 2 shows an example of the hardware configuration of the information processing device 101 in this embodiment. The information processing device 101 includes a CPU 201, ROM 202, RAM 203, auxiliary storage device 204, output device 205, input device 206, and network I / F 207. The CPU 201, ROM 202, RAM 203, auxiliary storage device 204, output device 205, input device 206, and network I / F 207 are connected to each other via a system bus 208.

[0015] The CPU (Central Processing Unit) 201 is a central processing unit that controls various operations of the information processing device 101. For example, the CPU 201 may control the operation of the entire information processing device 101. The ROM (Read Only Memory) 202 stores control programs, boot programs, etc., that can be executed by the CPU 201. The RAM (Random Access Memory) 203 is the main memory of the CPU 201 and is used as a work area or a temporary storage area for deploying various programs.

[0016] The auxiliary storage device 204 stores various data and programs. The auxiliary storage device 204 is implemented by a storage device capable of temporarily or permanently storing various data, such as an HDD (Hard Disk Drive) or non-volatile memory represented by an SSD (Solid State Drive).

[0017] The output device 205 is a device that outputs various types of information and is used to present various types of information to the user. For example, the output device 205 may be implemented by a display device such as a display. The output device 205 may present information to the user by displaying various types of display information. As another example, the output device 205 may be implemented by an acoustic output device that outputs sounds such as voice or electronic sounds. In this case, the output device 205 may present information to the user by outputting sounds such as voice or electronic sounds. Furthermore, the device to which the output device 205 is applied may be appropriately changed depending on the medium used to present information to the user.

[0018] The input device 206 is used to receive various instructions from the user. For example, the input device 206 may include input devices such as a mouse, keyboard, or touch panel. As another example, the input device 206 may include a sound collection device such as a microphone to collect the voice spoken by the user. In this case, the collected voice may be subjected to various analysis processes such as acoustic analysis and natural language processing so that the content of the voice is recognized as an instruction from the user. Furthermore, the device applied as the input device 206 may be changed as appropriate depending on the method of recognizing the user's instructions. In addition, multiple types of devices may be applied as the input device 206.

[0019] Network I / F 207 is used for communication with external devices via a network. The device used as Network I / F 207 may be changed as appropriate depending on the type of communication path and the applicable communication method.

[0020] The CPU 201 loads the program stored in the ROM 202 or auxiliary storage device 204 into the RAM 203 and executes the program, thereby realizing the various functions and processes of the information processing device 101 described later. The program for the information processing device 101 may be provided to the information processing device 101 by a recording medium such as a CD-ROM, or it may be downloaded via a network or the like. When the program for the information processing device 101 is provided by a recording medium, the program recorded on the recording medium is installed in the auxiliary storage device 204 when the recording medium is set in a predetermined drive device.

[0021] The configuration shown in Figure 2 is merely an example and does not necessarily limit the hardware configuration of the information processing device 101 in this embodiment. For example, some components such as the output device 205 and the input device 206 may be omitted. Another example is that components may be added as appropriate depending on the functions to be realized by the information processing device 101.

[0022] Figure 3 shows an example of the functional configuration of the information processing device 101 in the first embodiment. The information processing device 101 includes a communication unit 301, a control unit 302, an input / output control unit 303, a storage unit 304, an acquisition unit 305, a division unit 306, a conversion unit 307, an arithmetic unit 308, a specification unit 309, and an execution unit 310.

[0023] The communication unit 301 is a communication interface for each component of the information processing device 101 to send and receive information with other devices such as the server device 102 via the network 103. The communication unit 301 can be implemented, for example, by the network I / F 207. In the following description, when each component of the information processing device 101 sends and receives information with other devices, it will be assumed that the information is sent and received via the communication unit 301 unless otherwise specified.

[0024] The control unit 302 is responsible for controlling each component of the information processing device 101. The input / output control unit 303 performs various processes related to presenting various information to the user and receiving information input from the user (e.g., instructions). For example, the input / output control unit 303 may perform processes related to presenting a UI (User Interface) and processes related to receiving input via the UI. As a result, the information processing device 101 can recognize instructions from the user and present the user with the results of processing corresponding to those instructions.

[0025] The memory unit 304 schematically represents a memory area for storing various data and programs. For example, the memory unit 304 may store data and programs for each component of the information processing device 101 to execute processing. The memory unit 304 may also store test specifications, screen configuration information, and image information acquired by the acquisition unit 305, or it may store test operation description data obtained by dividing the test specifications. The memory unit 304 may also store screen images acquired as evidence of executed tests. Furthermore, the memory unit 304 may store machine learning models that embed information about each element and convert it into vector representations (embedded representations, distributed representations).

[0026] The acquisition unit 305 acquires a test specification document that shows the content of the test to be performed. The test specification document shows the operations to be performed (operation instructions) and the confirmation contents (confirmation instructions) to be confirmed during the test. The acquisition unit 305 also acquires screen configuration information and image information related to the screen of the system (application) under test. Screen configuration information is information related to the screen configuration, such as HTML (HyperText Markup Language) information and CSS (Cascade Style Sheet) information. Image information is information related to images displayed on the screen, such as GUI (Graphical User Interface) image information such as icon images.

[0027] The division unit 306 divides the test content, which consists of multiple operation contents and confirmation contents, etc., as shown in the test specification document acquired by the acquisition unit 305, into test operations. For example, the division unit 306 divides the test content shown in the test specification document into multiple test operations such that only one operation contents or one confirmation contents is included in each test operation. The division unit 306 also converts each divided test operation according to a predetermined notation to generate test operation description data. In this embodiment, as an example, BNF notation (Backus-Naur form) is applied to the description of the operation target in the test operation. Note that the test operation description data may also be generated by having a test person or the like describe each divided test operation according to a predetermined notation.

[0028] The conversion unit 307 vectorizes the information about each element involved in the execution of the test using a machine learning model that embeds the information, and converts it into a vector representation (embedding representation, distributed representation). Here, the vector representation is a representation of the information about the elements using multi-dimensional real vectors. The machine learning model is trained so that vector representations of similar elements are placed close together in the vector space (the distance between vectors is small, the similarity is high).

[0029] Specifically, the conversion unit 307 converts the element names of the targets of operation in the test operation, which have been divided by the division unit 306, into vector representations using a machine learning model that performs text embedding. Furthermore, based on the screen configuration information acquired by the acquisition unit 305, the conversion unit 307 converts the screen configuration information for each element (screen component) that may be targeted in the execution of the test operation on the screen of the system (application) under test into vector representations using a machine learning model that performs text embedding. Additionally, based on the image information acquired by the acquisition unit 305, the conversion unit 307 converts the image information for each image element that may be targeted in the execution of the test operation on the screen of the system (application) under test into vector representations using a machine learning model that performs image embedding. For example, the machine learning model that performs text embedding may be a model such as Sentence-BERT. For image embedding, a so-called multimodal embedding model capable of embedding both images and text may be applied; for example, a model such as OpenAI's CLIP may be applied. In addition to these machine learning models, it is also possible to generate a machine learning model by performing machine learning to convert information about elements into vector representations (embedding representations, distributed representations), and then apply the generated machine learning model. Note that the machine learning model for image embedding is pre-trained to match the vector representation of the machine learning model for text embedding. For example, it is pre-trained to match the vector representation of a trash can icon image with the vector representation of the text "trash can". Alternatively, a multimodal embedding model may be applied as the machine learning model for text embedding, and the machine learning model for text embedding may be pre-trained to match the vector representation of the machine learning model for image embedding.

[0030] The calculation unit 308 calculates the similarity between the element to be operated on in the test operation and the screen components and image elements on the screen of the system (application) under test that may be targeted during the execution of the test operation, based on the vector representation of each piece of information obtained by the conversion unit 307. For example, the calculation unit 308 matches the vector representation of the element name to be operated on in the test operation with the vector representations of the screen configuration information and image information of the elements on the screen that may be targeted during the execution of the test operation, and calculates a score indicating the similarity to the element to be operated on in the test operation for each element on the screen that may be targeted during the execution of the test operation.

[0031] The identification unit 309 identifies the element to be operated on when executing a test operation from among the elements on the screen of the system (application) under test, based on the similarity obtained by the calculation unit 308. The identification unit 309 identifies the element with the highest similarity from among the elements on the screen of the system (application) under test as the element to be operated on in the test.

[0032] The execution unit 310 performs tests on the system (application) under test according to the test operation. The execution unit 310 performs the tests by operating the elements to be operated, which have been identified by the identification unit 309, according to the operation instructions (operation type) in the test operation. The execution unit 310 may also acquire evidence of the test execution (for example, a screen captured after the operation) and save it in the storage unit 304 or the like.

[0033] In the example described above, the machine learning model for embedding text or images is configured to be installed in the information processing device 101. However, it may also be configured to be installed in another information processing device that can communicate with the information processing device 101 via a network or the like. In this case, the information processing device 101 can send input to the machine learning model to the other information processing device 101 that is equipped with the machine learning model via a network interface 207 or the like, and receive the processing results for that input.

[0034] The processing in the information processing device 101 in the first embodiment will be described with reference to Figures 4A and 4B. Figures 4A and 4B are diagrams illustrating an example of processing in the information processing device 101 in the first embodiment, with Figure 4A showing the overall processing flow and Figure 4B showing the score calculation processing. In the following description, we will use the case where the screen configuration information for the screen of the system (application) under test is HTML information created in HTML as an example, but it is not limited to this and can also be applied to screen configuration information created using other languages.

[0035] The information processing device 101 acquires a test specification 401 that indicates the contents of the test, such as the operations to be performed (operation instructions) and the contents to be confirmed (confirmation instructions).

[0036] In process 402, the information processing device 101 divides the acquired test specification 401 into test operations. For example, the test specification 401 contains multiple test cases that define a series of operations to be performed in the test, and each test case contains multiple operation contents, verification contents, etc. The information processing device 101 divides the test contents shown in the acquired test specification 401 into test cases, and further divides the divided test cases into multiple test operations such that one operation contents or one verification contents is included in one test operation. Then, for each divided test operation, the information processing device 101 converts each test operation according to a predetermined notation and generates test operation description data 403. Note that the generation of test operation description data 403 may also be performed by a test person or the like describing each test operation according to a predetermined notation for each divided test operation.

[0037] Let's take the following example test case shown in test specification 401. (1) Open "URL / login". (2) Enter "root" in the "Username or email" input area. (3) Enter "password" in the "Password" input area on the right side of the screen. (4) Click the "Sign in" button below the "Password" input area. (5) Confirm that it is the URL screen. ... The information processing device 101 divides the operations (confirmations) of (1), (2), (3), (4), (5), etc. into one test operation each and generates test operation description data 403 as shown in Figure 5. Figure 5 shows an example of the description data for each test operation included in the test case 500 of the example above, and for example, test operation 501 corresponds to the operation of (1) mentioned above. Here, the information processing device 101 selects the operation type from a predefined set according to the operation content. In addition, the input text field contains information other than the target of the operation (for example, the text to be entered) in the input operation (operation type is input), etc., as needed.

[0038] In this embodiment, the object of operation in the test operation description data 403 is described using BNF notation (Backusnaur notation) as an example. For example, the object of operation is described according to the notation shown in Figure 6A, and the element units included in the object of operation are described according to the notation shown in Figure 6B. Furthermore, the elements included in the object of operation and the element units are described according to the notation shown in Figure 6C. Thus, the description of the object of operation in the test operation description data 403 consists of an element name, attributes, and position, and is composed of at least an element name. For example, in the example shown in Figure 5, the object of operation "Password input area on the right side of the screen" is represented by "Password" as the element name, "input area" as the attribute related to the element name "Password", and "right side of the screen" as the position related to the element name "Password". Furthermore, for example, in the operation target "Sign in button below the "Password" input area", "Sign in" corresponds to the element name, "button" corresponds to the attributes related to the element name "Sign in", and "below the "Password" input area" corresponds to the position related to the element name "Sign in". In addition, in the description of this position "below the "Password" input area", "Password" corresponds to the element name, and "input area" corresponds to the attributes related to the element name "Password".

[0039] Furthermore, the names of the elements to be manipulated can be described using absolute or relative positions, or in matrix form. For example, suppose that the screen 701 shown in Figure 7 has a “Username or email” input area 702, a “Password” input area 703, and a “Sign in” button 704. In this case, depending on the description in the test specification, the description of the “Password” input area 703 can be an absolute position description such as “the “Password” input area on the right side of the screen,” or a relative position description such as “the “Password” input area below the “Username or email” input area” or “the “Password” input area above the “Sign in” button.” By enabling descriptions using absolute and relative positions, as well as matrix forms, it becomes possible to uniquely identify elements on the screen, even when there are multiple elements with the same name or when the screen content is complex.

[0040] Returning to Figure 4A, in process 404, the information processing device 101 uses a parser to decompose the description of the target of operation in the test operation description data 403 generated for each test operation into element name 405, attribute 406, and position 407. The element name 405, attribute 406, and position 407 obtained by decomposing the test operation description data 403 by the parser are input to the score calculation process 421.

[0041] Furthermore, the information processing device 101 acquires HTML information (screen configuration information) 412 and image information 415 related to the web screen on which the test is executed from the system (application) 411 under test. The HTML information 412 is information about the screen configuration of the web screen created in HTML, and the image information 415 is information about the images displayed on the web screen.

[0042] In process 413, the information processing device 101 divides the acquired HTML information 412 related to the web screen into elements and outputs HTML information 414 for each element. For example, by dividing the acquired HTML information 412 related to the web screen into elements, the information processing device 101 can determine the value of a certain element (screen component). <input class="form-control gl-form-input bottom" required="required" title="This field is required." data-qa-selector="password_field" type="password" name ="user[password]" id="user_password"> If HTML information is obtained, it is output as the HTML information 414 for that element. The HTML information 414 for each element obtained by process 413 is input to the score calculation process 421.

[0043] In process 416, the information processing device 101 acquires image information for each element based on the acquired image information 415 related to the web screen, and outputs image information 417 for each element. For example, with respect to a certain element (image element), the information processing device 101 outputs an image obtained by cropping it from the acquired image information 415 related to the web screen based on the coordinates of the element, as the image information 417 for that element. The image information 417 for each element obtained by process 413 is input to the score calculation process 421.

[0044] In process 421, the information processing apparatus 101 performs a score calculation process based on information 405 to 407 regarding the operation target in the test operation and information 414 and 417 regarding the Web screen to be tested, and calculates and outputs a score indicating the similarity to the operation target in the test operation for each element on the Web screen. The information processing apparatus 101 uses the element name 405, attribute 406, and position 407 regarding the operation target in the input test operation, and the HTML information 414 for each element (screen component) and the image information 417 for each element (image element) regarding the Web screen to be tested to calculate the similarity between the operation target in the test operation and the elements (screen components and image elements) on the Web screen. The details of the score calculation process in this process 421 will be described later.

[0045] In process 423, the information processing apparatus 101 identifies the element to be operated when executing the test operation from the elements on the Web screen to be tested based on the score 422 for each element on the Web screen to be tested obtained in process 421. The information processing apparatus 101 identifies one element on the Web screen with the highest similarity to the operation target in the test operation as the element to be operated in the test operation based on the score 422 for each element obtained in the score calculation process 421. Here, when the description of the operation target includes a specifying description such as "the nth from the top...", the nth element from the top on the screen may be identified as the element to be operated in the test operation.

[0046] In process 424, the information processing apparatus 101 operates the element to be operated identified in process 423 according to the description data 403 of the test operation. As a result, a test corresponding to the description data 403 of the test operation is executed in the system (application) 411 to be tested.

[0047] In process 425, the information processing apparatus 101 acquires the execution result of the test and stores the acquired execution result 426 as evidence regarding the test. For example, the information processing apparatus 101 acquires (captures) the screen of the system (application) 411 to be tested after the operation is performed according to the description data 403 of the test operation and stores it as the execution result (evidence) 426.

[0048] After performing operations according to the test operation description data 403 for a certain test operation in this way and executing the test of the system (application) 411 to be tested, the information processing apparatus 101 acquires HTML information 412 and image information 415 regarding the Web screen of the system (application) 411 to be tested after performing the operations according to the test operation, and performs processing for the next test operation. That is, the information processing apparatus 101 performs a score calculation process based on the HTML information 412 and the image information 415 regarding the Web screen after the execution of the test operation, and the element name 405, the attribute 406, and the position 407 in the description data 403 of the next test operation, identifies the element to be operated in the next test operation, performs an operation according to the next test operation, and executes the test of the system (application) 411 to be tested. If there is no next test operation, the information processing apparatus 101 ends the test of the system (application) 411 to be tested.

[0049] That is, the information processing apparatus 101 in the present embodiment repeats the processing within the broken line frame shown in FIG. 4A for each test operation, and each time an operation according to the test operation is executed, it acquires the HTML information 412 and the image information 415 regarding the Web screen of the system (application) 411 to be tested after the operation execution, and executes a test regarding the next test operation. As a result, for each test operation, it becomes possible to execute a test on the Web screen of the system (application) 411 to be tested after the display content has changed or the screen has been transitioned by the previous test operation, and it becomes possible to handle a screen that can be dynamically changed by the test operation.

[0050] Next, referring to FIG. 4B, the score calculation process (process 421 shown in FIG. 4A) will be described. In FIG. 4B, the same reference numerals are given to the same components as those shown in FIG. 4A.

[0051] In process 451, the information processing apparatus 101 vectorizes the element name 405 regarding the operation target in the input test operation using a machine learning model for text embedding, and converts it into a vector representation 452 of the element name.

[0052] In process 453, the information processing device 101 filters (filters) the HTML information 414 for each element (screen component) and the image information 417 for each element (image element) of the input Web screen of the test target based on the attributes 406 related to the target of operation in the input test operation, and excludes information of elements whose attributes 406 do not match from the HTML information 414 and image information 417 for each element. The information processing device 101 classifies the attributes of each element on the Web screen based on the HTML information 414 and image information 417 for each element. Then, the information processing device 101 compares the obtained attributes of the elements on the Web screen with the attributes 406 related to the target of operation in the test operation for each element, and excludes the information of the element if the attributes do not match. The attributes of the elements on the Web screen may be classified using a rule-based classification method based on tag information, etc., or they may be classified using a machine learning model.

[0053] In process 454, the information processing device 101 vectorizes the HTML information 414 for each element after filtering in process 453 using a machine learning model that performs text embedding, and converts it into a vector representation 455 of the HTML information for each element. First, the information processing device 101 collects the descriptive information of the elements in the HTML description (including the descriptive information of child elements if there are child elements) from the filtered HTML information 414 for each element, for example in a list format. The descriptive information collected here is descriptive information that can be displayed as text information on the screen and specified as element names. For example, information about "text", "placeholder", "name", "value", "aria-label", "title", "id", etc. for each element in the HTML description is collected as descriptive information. The example shown here is just one example and is not limited to this. Furthermore, for child elements, descriptive information may be appropriately retrieved depending on the distance from the parent element, etc. Also, for inputs, radio buttons, etc., descriptive information of neighboring elements may be collected. For example, if the HTML information 414 for each element is, <input class="form-control gl-form-input bottom" required="required" title="This field is required." data-qa-selector="password_field" type="password" name ="user[password]" id="user_password"> If this is the case, the following information will be collected as descriptive data: ["This field is required.", "user[password]", "user_password"].

[0054] Next, the information processing device 101 converts the collected descriptive information into a vector representation using a machine learning model that performs text embedding. For example, in the example of ["This field is required.", "user[password]", "user_password"] mentioned above, "This field is required.", "user[password]", and "user_password" are each converted into a vector representation.

[0055] In process 456, the information processing device 101 vectorizes the element-specific image information 417 after filtering in process 453 using a machine learning model that performs image embedding, and converts it into a vector representation 457 of the element-specific image information. Here, the machine learning model that performs image embedding used in process 456 is pre-trained to match the vector representation of the machine learning model that performs text embedding.

[0056] In process 458, the information processing device 101 matches the vector representation 452 of the element name with the vector representation 455 of the HTML information for each element and the vector representation 457 of the image information for each element to obtain an element-specific score for each element on the Web screen under test, indicating its similarity to the target of operation in the test operation. The information processing device 101 obtains the score for each element as follows, for example: The information processing device 101 calculates the cosine similarity between the vector representation 452 of the element name and the vector representation 455 of the HTML information of the target element and saves it as a candidate score for the element. The information processing device 101 also calculates the cosine similarity between the vector representation 452 of the element name and the vector representation 457 of the image information of the target element, and saves the obtained cosine similarity corrected using a correction value as a candidate score for the element. Here, the correction value is a correction value that allows the vector representation of the HTML information and the vector representation of the image information to be compared on the same scale, and is assumed to be predetermined. The information processing device 101 then obtains the element with the highest similarity (or the one closest to 1 in the case of cosine similarity) from among the stored element score candidates, and outputs a score of 459 for each element. The above explanation uses cosine similarity, but it is not limited to this, and any index that can evaluate the similarity of each element can be applied. For example, the Levenshtein distance or Jarrow-Winkler distance, which are indices that measure the similarity between strings, can be used, or the Pearson correlation coefficient, Euclidean distance or word carrying distance, which are indices that measure the similarity between vectors, can be used.

[0057] Process 460 is executed when the description of the object to be operated on includes information indicating a relative position, such as "the 'Password' input area above the 'Sign in' button." In process 460, the information processing device 101 obtains and outputs a score 461 for each element (hereinafter also referred to as a relative element) used to indicate the relative position, based on the position 407 of the object to be operated on in the input test operation, in the same manner as in process 458 described above. That is, the information processing device 101 obtains and outputs a score for each element on the web screen for the relative element, based on the vector representation of the element name of the relative element, the vector representation of the HTML information for each element on the web screen, and the vector representation of the image information for each element.

[0058] Process 462 is executed when the description of the object to be operated on includes location information. The information processing device 101 adjusts the element-specific score 459 obtained by process 458 according to the location and outputs the adjusted score as the element-specific score 422. If the description of the object to be operated on does not include location information, the information processing device 101 outputs the element-specific score 459 obtained by process 458 as the element-specific score 422. In process 462, if the description of the object to be operated on includes information indicating an absolute location, the information processing device 101 adjusts the element-specific score 459 according to the location description. For example, if the element-specific score is cosine similarity, the information processing device 101 adjusts the score by subtracting a value that increases as the distance from the described location increases. For example, if it is the "Password" input area on the right side of the screen, the information processing device 101 adjusts the element-specific score 459 by subtracting a value corresponding to the distance between the element and the right edge of the screen. Furthermore, if the description of the object to be manipulated includes information indicating relative position, the information processing device 101 adjusts the element-specific score 459 according to the element-specific score and position description of the relative element. For example, the information processing device 101 adds the element-specific score 461 for the relative element to each of the element-specific scores 459 for the target element, and adjusts the added score based on the distance between elements according to the position description. From the resulting score, the device outputs the score with the highest similarity for each element on the web screen as the element-specific score 422. For example, in the case of the "Password" input area above the "Sign in" button, the information processing device 101 adds the element-specific score 461 for the relative element's element name "Sign in" to each of the element-specific scores 459 for the element name "Password". Furthermore, the information processing device 101 subtracts a value from the summed score that corresponds to the distance (which may be the difference in each coordinate) between the element from which an element-specific score 459 was obtained and the element from which an element-specific score 461 was obtained, and sets the score with the highest similarity for each element on the web screen as the element-specific score.

[0059] As described above, according to this embodiment, for each test operation divided from the test specification, the information of the target of operation in the test operation, the HTML information (screen configuration information) for each element on the screen of the target of test, and the image information for each element are converted into vectors using a machine learning model. The converted vectors are then compared to identify the element on the screen of the target of test that has the highest similarity to the target of operation in the test operation. By performing this processing, it becomes possible to uniquely identify the element of the target of operation in the test operation, and tests corresponding to the test operation can be executed appropriately.

[0060] Furthermore, by obtaining HTML information (screen configuration information) and image information related to the screen under test after each test operation, and then executing the test for the next test operation, it becomes possible to appropriately perform tests in response to screens that may be dynamically changed by test operations. In addition, by referring to the image information related to the screen under test when identifying the elements to be operated on in a test operation, it becomes possible to identify elements that can only be specified visually.

[0061] In the explanation above, an example was described in which the description of the target of operation in the test operation description data 403 consists of an element name 405, an attribute 406, and a position 407. However, the description of the target of operation only needs to include at least an element name 405. For example, if the description of the target of operation does not include an attribute 406, the processing (score calculation processing) 421 shown in Figure 4B can be simplified by performing processes 454 and 456 to vectorize the HTML information (screen configuration information) and image information for each element without performing the filtering in process 453. Also, for example, if the description of the target of operation does not include a position 407, the processing (score calculation processing) 421 shown in Figure 4B can be simplified by omitting processes 460 and 462, and outputting the score for each element obtained in process 458 as the score for each element 422. For example, if the description of the target of operation includes an element name 405 but does not include an attribute 406 and a position 407, the score calculation processing in process 421 can be implemented with the configuration shown in Figure 8. Figure 8 shows another example of the score calculation process, and the same components as those shown in Figure 4B are denoted by the same reference numerals.

[0062] <Second Embodiment> In the second embodiment described below, an overview of each test case is extracted from the test specification document that shows the content of the test, and the extracted overview of the test case is input into a machine learning model to perform inference using the machine learning model and generate a description of the test operation. Then, based on the description of the test operation generated by the machine learning model inference, the test is executed on the system (application) under test in the same manner as in the first embodiment. The configuration of the information processing system and the hardware configuration of the information processing device in the second embodiment are the same as the configuration of the information processing system and the hardware configuration of the information processing device in the first embodiment, so the explanation is omitted.

[0063] Figure 9 shows an example of the functional configuration of the information processing device 101 in the second embodiment. In Figure 9, components having the same function as those shown in Figure 3 are denoted by the same reference numerals, and redundant explanations are omitted. The information processing device 101 in the second embodiment includes a communication unit 301, a control unit 302, an input / output control unit 303, a storage unit 304, an acquisition unit 305, an extraction unit 901, a generation unit 902, a conversion unit 307, an arithmetic unit 308, a specification unit 309, and an execution unit 310.

[0064] The extraction unit 901 extracts an outline of each test case from the test specification document, which shows the content of the test to be performed, for each test case that defines a series of operations to be performed in the test. The extraction unit 901 divides the test content shown in the test specification document acquired by the acquisition unit 305 into test cases, and extracts an outline of each of the divided test cases. The outline of the test case may be modified by the test person or others according to the test operation description data generated based on the outline of the test case.

[0065] The generation unit 902 generates test operation description data for the test operations performed in a test case, based on the test case summary extracted by the extraction unit 901 and the screen configuration information and image information acquired by the acquisition unit 305. For example, the generation unit 902 inputs the test case summary, screen configuration information and image information, and a prompt including instructions to generate a description of the test operations into a machine learning model, performs inference using the machine learning model, and generates the test operation description data.

[0066] Here, the generation unit 902 performs inference using a trained machine learning model, such as a large-scale multimodal model (LMM), which is generated by machine learning (deep learning). A large-scale multimodal model (LMM) is a machine learning model constructed using a large amount of training data and deep learning technology. When data called a prompt, which indicates instructions, is input, it performs inference based on the prompt and generates and outputs data corresponding to the input prompt.

[0067] Furthermore, the generation unit 902 may generate descriptive data for test operations based on the test case overview and screen configuration information. In that case, if the input is only text data, inference may be performed using a Large Language Model (LLM), which is a language model constructed using a large amount of text data (such as a large corpus) and deep learning technology, rather than being limited to a Large Multimodal Model (LMM).

[0068] Similar to the first embodiment, the conversion unit 307 vectorizes information related to each element involved in the execution of the test using a machine learning model that embeds the respective information, and converts it into a vector representation (embedded representation, distributed representation). Specifically, based on the test operation description data generated by the generation unit 902, the conversion unit 307 converts the element names targeted in the test operation into vector representations using a machine learning model that embeds text. Furthermore, based on the screen configuration information acquired by the acquisition unit 305, the conversion unit 307 converts the screen configuration information for each element (screen component) that may be targeted in the execution of the test operation into a vector representation using a machine learning model that embeds text. Furthermore, based on the image information acquired by the acquisition unit 305, the conversion unit 307 converts the image information for each element of the image on the screen that may be targeted in the execution of the test operation into a vector representation using a machine learning model that embeds images.

[0069] In the example described above, the machine learning model may be configured to be installed on another information processing device that can communicate with the information processing device 101 via a network or the like. In this case, the information processing device 101 can send input to the machine learning model to the other information processing device 101 that has the machine learning model installed via a network interface 207 or the like, and receive the processing results for that input.

[0070] Referring to Figure 10, the processing in the information processing device 101 in the second embodiment will be described. Figure 10 is a diagram showing an example of processing in the information processing device 101 in the second embodiment. In Figure 10, the same reference numerals are used for elements that are the same as those shown in Figure 4A. In the following description, we will use the case where the screen configuration information for the screen of the system (application) under test is HTML information created in HTML as an example, but we are not limited to this and the explanation is also applicable to screen configuration information created using other languages.

[0071] The information processing device 101 acquires a test specification 401 that shows the test content, such as the operations to be performed (operation instructions) and the verification content (verification instructions) to be verified on the system (application) under test.

[0072] In process 1001, the information processing device 101 extracts an outline of each test case based on the test content shown in the acquired test specification 401. For example, the test specification 401 contains multiple test cases that define a series of operations to be performed in the test, and each test case contains multiple operation contents, confirmation contents, etc. The information processing device 101 divides the test content shown in the acquired test specification 401 into test cases and extracts a test case outline 1002 for each divided test case. An example of the extracted test case outline 1002 is shown in Figure 11. Figure 11 shows an example of a test case outline 1100 related to a test to confirm that the login process is completed successfully, and includes the screen to be tested, the operation contents and confirmation contents to be performed in the test, etc.

[0073] The process described below is performed for each extracted test case summary 1002. In process 1003, the information processing device 101 generates and outputs test operation description data 1004 to be performed in the test case, based on the extracted test case summary 1002 and the HTML information (screen configuration information) 412 and image information 415 related to the Web screen (screen to be tested) on which the test is performed, obtained from the system (application) 411 under test. Note that the image information 415 is optional, and the information processing device 101 may generate and output test operation description data 1004 based on the extracted test case summary 1002 and the HTML information (screen configuration information) 412 related to the Web screen (screen to be tested) on which the test is performed. In this process 1003, the information processing device 101 refers to the test operation description data 1004 already performed in the test case summary 1002 under processing and generates test operation description data 1004 for the next executable procedure (one test operation). The information processing device 101, for example, inputs the prompt 1200 shown in Figure 12 to the machine learning model, performs inference regarding the generation of a test operation description, and generates and outputs the description data 1004 for the next test operation.

[0074] Figure 12 illustrates an example of a prompt for generating a test operation description. As shown in Figure 12, the prompt 1200 for generating a test operation description includes a description of the test operation generation 1210, a test case summary 1220, a description of previous test operations 1230, constraints for generating the test operation description 1240, a description of the test operation notation 1250, HTML information (screen configuration information) 1260, and image information 1270. The description of the test operation generation 1210 is a description for executing the generation of test operation description data. The test case summary 1220 is a summary of the test case to be processed extracted from the test specification. The description of previous test operations 1230 is the description data of a test operation that has already been executed. The constraints for generating the test operation description 1240 are constraints when generating the test operation description data, and include, for example, generating only one executable test operation or prohibiting the generation of a test operation that has already been executed. The explanation of the test operation notation 1250 is an explanation of the description format for the test operation description data, and includes, for example, describing the operation procedure in the format "operation type "target", describing the target with an element name and a specified attribute, and specifying the element name with text that exists on the screen. The HTML information (screen configuration information) 1260 is HTML information (screen configuration information) related to the web screen (test target screen) on which the test is executed. The image information 1270 is image information related to the web screen (test target screen) on which the test is executed.

[0075] Specific examples of prompts for generating test operation descriptions are shown in Figures 13A and 13B. In Figures 13A and 13B, element 1310 corresponds to the explanation 1210 for generating test operation descriptions, element 1320 corresponds to the test case summary 1220, and element 1330 corresponds to the previous test operation description 1230. Element 1340 corresponds to the constraints 1240 for generating test operation descriptions, element 1350 corresponds to the explanation 1250 for test operation notation, and element 1360 is the part that prompts the machine learning model for output. Note that in the examples shown in Figures 13A and 13B, elements corresponding to HTML information (screen configuration information) 1260 and image information 1270, respectively, are omitted from the illustration.

[0076] Returning to Figure 10, in process 404, the information processing device 101 uses a parser to decompose the description of the target of operation in the test operation description data 1004 generated in process 1003 into element name 405, attribute 406, and position 407. The element name 405, attribute 406, and position 407 obtained by decomposing the test operation description data 1004 with the parser are input to the score calculation process 421. Other processes (processes after the score calculation process) are the same as in the first embodiment, so their explanation is omitted.

[0077] The following description will explain a specific example of the process described with reference to Figure 10, where the information processing device 101 performs processing based on, for example, the test case overview 1100 shown in Figure 11. In the following description, the information processing device 101 will be described as performing inference regarding the generation of a test operation description by inputting the prompt 1200 shown in Figure 12 into a machine learning model, and generating and outputting test operation description data.

[0078] In the first process 1003 performed after starting the processing for test case summary 1100, the information processing device 101 inputs the prompt 1200 shown in Figure 12 into the machine learning model, setting the test case summary 1220 to the test case summary 1100 shown in Figure 11, and setting the description of the test operations so far 1230 to empty data (Null, or information indicating that it has not been performed). This generates, for example, the test operation description data 1401 shown as number 1 in Figure 14, where the operation type is goto and the target of the operation is "URL / login". Based on this test operation description data 1401, the information processing device 101 executes the operation to open the "URL / login" screen as a test operation.

[0079] In the second processing step 1003, after executing an action corresponding to the test action description data 1401, the information processing device 101 inputs a prompt 1200 to the machine learning model, where the test case overview 1220 is set to the test case overview 1100 shown in Figure 11, and the description of the previous test actions 1230 is set to the test action description data 1401. In this prompt 1200, HTML information (screen configuration information) 1260 and image information 1270 are input, respectively, of the HTML information and image information related to the "URL / login" screen obtained from the system (application) under test. As a result, for example, the test action description data 1402 shown as number 2 in Figure 14, where the operation type is input, the target of the operation is the "Username or email" input area, and the input text is root, is generated. Based on the test operation description data 1402, the information processing device 101 performs score calculation processing and processing to identify the element to be operated on, and as a test operation, it executes the operation of entering "root" as the username in the "Username or email" input area of ​​the "URL / login" screen.

[0080] In the third process 1003, after executing an action corresponding to the test action description data 1402, the information processing device 101 inputs a prompt 1200 to the machine learning model, where the test case overview 1220 is set to the test case overview 1100 shown in Figure 11, and the descriptions of the previous test actions 1230 are set to the test action description data 1401 and 1402. In this prompt 1200, HTML information (screen configuration information) 1260 and image information 1270 are input, respectively, of the HTML information and image information related to the "URL / login" screen, which were obtained from the system (application) under test after executing an action corresponding to the test action description data 1402. As a result, for example, the test action description data 1403 shown as number 3 in Figure 14, where the operation type is input, the target of the operation is the "Password" input area, and the input text is password, is generated. Based on the test operation description data 1403, the information processing device 101 performs score calculation processing and processing to identify the element to be operated on, and as a test operation, it executes the operation of entering a password as the password in the "Password" input area of ​​the "URL / login" screen.

[0081] In the processing 1003 after executing the operation corresponding to the test operation description data 1403, the information processing device 101 inputs a prompt 1200 to the machine learning model, where the test case overview 1220 is set to the test case overview 1100 shown in Figure 11, and the descriptions of the previous test operations 1230 are set to the test operation description data 1401 to 1403. In this prompt 1200, HTML information (screen configuration information) 1260 and image information 1270 are input, respectively, of the HTML information and image information related to the "URL / login" screen obtained from the system (application) under test after executing the operation corresponding to the test operation description data 1403. As a result, for example, the test operation description data 1404, shown as number 4 in Figure 14, where the operation type is click and the target of the operation is the "Sign in" button, is generated. Based on the test operation description data 1404, the information processing device 101 performs score calculation processing and processing to identify the element to be operated on, and executes the operation of clicking the "Sign in" button on the "URL / login" screen as a test operation.

[0082] In the processing 1003 after executing an action corresponding to the test action description data 1404, the information processing device 101 inputs a prompt 1200 to the machine learning model, setting the test case overview 1220 to the test case overview 1100 shown in Figure 11, and the descriptions of the previous test actions 1230 to the test action description data 1401 to 1404. In this prompt 1200, HTML information (screen configuration information) 1260 and image information 1270 are input, respectively, relating to the screen obtained from the system (application) under test after executing the action corresponding to the test action description data 1404. As a result, for example, the test action description data 1405, shown as number 5 in Figure 14, with operation type as assert_url and operation target as "URL", is generated. Based on this test action description data 1405, the information processing device 101 confirms that the screen is a URL as a test action.

[0083] In the processing 1003 after executing the operation corresponding to the test operation description data 1405, the information processing device 101 inputs a prompt 1200 to the machine learning model, setting the test case overview 1220 to the test case overview 1100 shown in Figure 11, and the descriptions of the previous test operations 1230 to the test operation description data 1401 to 1405. In this prompt 1200, HTML information (screen configuration information) 1260 and image information 1270 are input, respectively, of the screen information and image information obtained from the system (application) under test after executing the operation corresponding to the test operation description data 1405. As a result, for example, the test operation description data 1406 with operation type "end operations", shown as number 6 in Figure 14, is generated. Based on this test operation description data 1406, the information processing device 101 terminates processing, considering that the test operations to be performed in the test case are complete. If there are other test case overviews that have not yet been performed, processing for those test case overviews is started.

[0084] In the process described above, the test behavior description data generated by the machine learning model based on the test case overview may be inappropriate, potentially causing problems (errors) when attempting to execute the test behavior. For example, possible problems (errors) include: - The test behavior description data cannot be properly parsed by the parser (for example, it should be written as "“Password” input area", but it is written as "“Password” input area", violating the notation) - The target element is unclear (the target score is low, or the element cannot be found) - The target element cannot be uniquely identified, etc.

[0085] When such a problem occurs, the information processing device 101 executes process 1003 by referring to the test operation description data in which the problem (error) occurred and the error message indicating the problem (error) that occurred, and regenerates (corrects) the test operation description data. For example, the information processing device 101 inputs the prompt 1500 shown in Figure 15 into a machine learning model to perform inference regarding the generation of the test operation description, and regenerates (corrects) the test operation description data and outputs it.

[0086] Figure 15 illustrates an example of a prompt related to the generation (modification) of a test operation description. As shown in Figure 15, the prompt 1500 for generating a test operation description includes a description of the test operation description generation 1510, a test case summary 1520, previous test operation descriptions 1530, constraints on generating the test operation description 1540, a description of the test operation notation 1550, the operation, error message, and solution to be modified 1560, HTML information (screen configuration information) 1570, and image information 1580. The description of the test operation description generation 1510 is a description to cause the regeneration (modification) of the test operation description data to be executed. The test case summary 1520, the description of previous test operations 1530, the constraints for generating the test operation description 1540, and the explanation of the test operation notation 1550 are the same as the test case summary 1220, the description of previous test operations 1230, the constraints for generating the test operation description 1240, and the explanation of the test operation notation 1250 for prompt 1200 shown in Figure 12. The operation to be corrected, error message, and solution 1560 are the description data of the test operation in which the problem (error) occurred, the error message indicating the problem (error) that occurred, and the suggestion of a method to resolve the problem (error). The HTML information (screen configuration information) 1570 and image information 1580 are the HTML information (screen configuration information) and image information related to the screen under test.

[0087] Specific examples of prompts for generating (correcting) test operation descriptions are shown in Figures 16A and 16B. Figures 16A and 16B show examples of prompts for generating (correcting) test operation descriptions when the parser cannot properly decompose the test operation description data and a problem (error) occurs during parsing. In Figures 16A and 16B, element 1610 corresponds to the explanation of test operation description generation 1510, element 1620 corresponds to the test case summary 1520, and element 1630 corresponds to the previous test operation description 1530. Element 1640 corresponds to the constraints 1540 when generating test operation descriptions, element 1650 corresponds to the explanation of test operation notation 1550, and element 1660 corresponds to the operation, error message, and solution to be corrected 1560. Element 1570 is the part that prompts the machine learning model for output. In the examples shown in Figures 16A and 16B, the elements corresponding to the HTML information (screen configuration information) 1570 and the image information 1580, respectively, are omitted from the illustration.

[0088] If the element to be operated on is unknown and a problem (error) occurs, the element 1660 enclosed by the dashed line in Figure 16B (corresponding to the operation to be corrected, error message, and solution 1560) should be changed to the element 1760 shown in Figure 17. Furthermore, if the element to be operated on cannot be uniquely identified and a problem (error) occurs, the element 1660 enclosed by the dashed line in Figure 16B (corresponding to the operation to be corrected, error message, and solution 1560) should be changed to the element 1860 shown in Figures 18A and 18B.

[0089] If a problem (error) occurs when attempting to execute a test based on the generated test operation description data, the test operation description data can be regenerated (corrected) in this way, making it possible to generate highly accurate test operation description data. Furthermore, by switching prompts according to the problem (error) that occurred and regenerating (correcting) the test operation description data, it is possible to appropriately correct the test operation description data according to the problem (error) that occurred. Note that the pre-correction test operation description data that caused the problem (error) will be deleted, but the pre-correction test operation description data may be saved separately to be useful in generating test operation descriptions.

[0090] According to the second embodiment, similar to the first embodiment, it becomes possible to uniquely identify the element to be operated on in the test operation, and to appropriately execute tests corresponding to the test operation. Furthermore, by acquiring information on the screen after the operation is executed and executing tests related to the next test operation, it becomes possible to appropriately execute tests in response to screens that may be dynamically changed by the test operation.

[0091] <Other Embodiments> The present invention can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various recording media; and having the computer (or CPU or MPU, etc.) of the system or device read and execute the program. Computer-readable recording media on which the program is recorded and computer program products such as the program can also be applied as embodiments of the present invention. Examples of recording media that can be used include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0092] It should be noted that the embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0093] The following devices and methods are also included in the disclosure of this embodiment: (1) An information processing device comprising: a test specification showing the content of a test; an acquisition means for acquiring screen configuration information and image information relating to a screen from a test target; an extraction means for extracting test operations based on the content of the test shown in the test specification; a conversion means for converting information on the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen into vectors using a machine learning model that performs embedding; and an identification means for identifying the element of the target of operation in the test operation from among the elements on the screen based on the information on the target of operation, the screen configuration information, and the image information converted into vectors. (2) The information processing device according to (1), wherein the extraction means divides the content of the test shown in the test specification into a plurality of test operations, and generates a description data of the test operation converted according to a predetermined notation for each of the divided test operations. (3) The information processing device according to (1), wherein the extraction means extracts an outline of a test for each test content shown in the test specification, and uses the extracted outline of a test to perform inference by a machine learning model to generate descriptive data of a test operation. (4) The information processing device according to any one of (1) to (3), further comprising an execution means for executing the test operation on the identified element of the target of operation. (5) The information processing device according to any one of (1) to (4), wherein the acquisition means acquires the screen configuration information and the image information each time the test operation is executed. (6) The information processing device according to any one of (1) to (5), wherein the identification means acquires the similarity to the target of operation for each element on the screen based on the information of the target of operation converted into a vector, the screen configuration information, and the image information, and identifies the element with the highest similarity as the element of operation in the test operation. (7) The information processing apparatus according to (2) or (3), wherein the description of the object to be operated in the description data of the test operation includes an element name, and the conversion means converts the element name into a vector using the machine learning model.(8) The information processing apparatus according to (7), wherein the description of the object to be operated in the description data of the test operation further includes a description indicating the position of the object to be operated, and the identification means identifies an element of the object to be operated in the test operation based on the information of the object to be operated converted into a vector, the screen configuration information, and the image information, and the described position of the object to be operated. (9) The information processing apparatus according to (8), wherein the identification means obtains a similarity to the object to be operated for each element on the screen based on the information of the object to be operated converted into a vector, the screen configuration information, and the image information, adjusts the obtained similarity of the elements on the screen according to the described position of the object to be operated, and identifies the element with the highest adjusted similarity as the element of the object to be operated in the test operation. (10) The information processing apparatus according to (8) or (9), wherein the position of the object to be operated is an absolute position on the screen. (11) The information processing apparatus according to (8) or (9), wherein the position of the object to be operated is a relative position to an element on the screen. (12) The information processing device according to any one of (7) to (11), wherein the description of the target to be operated in the description data of the test operation further includes a description indicating the attributes of the target to be operated, and the conversion means converts the screen configuration information for each element on the screen and the image information for each element on the screen into vectors, excluding elements whose attributes do not match those of the target to be operated. (13) The information processing device according to (3), wherein if a problem occurs when executing a test operation based on the generated description data of the test operation, the extraction means corrects the description data of the test operation by performing inference using a machine learning model by referring to the problem that occurred.(14) An information processing method to be executed by an information processing device, comprising: an acquisition step of acquiring a test specification that shows the content of a test, screen configuration information and image information relating to a screen from a test target; an extraction step of extracting test operations based on the content of the test shown in the test specification; a conversion step of converting information of the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen into vectors using a machine learning model that performs embedding; and an identification step of identifying the element of the target of operation in the test operation from among the elements on the screen based on the information of the target of operation, the screen configuration information, and the image information converted into vectors. (15) A program (computer program product) that causes a computer to execute: an acquisition step of obtaining a test specification that shows the content of the test, screen configuration information and image information related to the screen from the test target; an extraction step of extracting test operations based on the content of the test shown in the test specification; a conversion step of converting the information of the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen into vectors using an embedding machine learning model; and an identification step of identifying the element of the target of operation in the test operation from among the elements on the screen based on the information of the target of operation, the screen configuration information, and the image information converted into vectors. (16) A computer-readable recording medium characterized by recording a program for causing a computer to execute: an acquisition step of acquiring a test specification that shows the content of the test, screen configuration information and image information relating to the screen from the test target; an extraction step of extracting test operations based on the content of the test shown in the test specification; a conversion step of converting the information of the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen into vectors using a machine learning model that performs embedding; and an identification step of identifying the element of the target of operation in the test operation from among the elements on the screen based on the information of the target of operation, the screen configuration information, and the image information converted into vectors.

[0094] According to the present invention, it becomes possible to uniquely identify the object to be operated on during testing.

Claims

1. An information processing device comprising: a test specification document showing the content of the test; an acquisition means for acquiring screen configuration information and image information related to the screen from the test target; an extraction means for extracting test operations based on the content of the test shown in the test specification document; a conversion means for converting information on the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen into vectors using a machine learning model that performs embedding; and an identification means for identifying the element that is the target of operation in the test operation from among the elements on the screen based on the information on the target of operation, the screen configuration information, and the image information converted into vectors.

2. The information processing apparatus according to claim 1, characterized in that the extraction means divides the content of the test shown in the test specification into a plurality of test operations, and generates test operation description data for each of the divided test operations by converting the test operation according to a predetermined notation.

3. The information processing apparatus according to claim 1, characterized in that the extraction means extracts an outline of the test for each test content shown in the test specification, and uses the extracted outline of the test to perform inference using a machine learning model to generate descriptive data of the test operation.

4. The information processing apparatus according to any one of claims 1 to 3, further comprising execution means for performing the test operation on the specified element to be operated on.

5. The information processing apparatus according to any one of claims 1 to 4, characterized in that the acquisition means acquires the screen configuration information and the image information each time the test operation is performed.

6. The information processing apparatus according to any one of claims 1 to 5, characterized in that the identifying means obtains a similarity to the target of operation for each element on the screen based on the information of the target of operation converted into a vector, the screen configuration information, and the image information, and identifies the element with the highest similarity as the target of operation in the test operation.

7. The information processing apparatus according to claim 2 or 3, wherein the description of the object to be operated on in the description data of the test operation includes element names, and the conversion means converts the element names into vectors using the machine learning model.

8. The information processing apparatus according to claim 7, wherein the description of the object to be operated in the description data of the test operation further includes a description indicating the position of the object to be operated, and the identifying means identifies the elements of the object to be operated in the test operation based on the information of the object to be operated converted into a vector, the screen configuration information, and the image information, and the described position of the object to be operated.

9. The information processing apparatus according to claim 8, characterized in that the identifying means obtains a similarity to the target for each element on the screen based on the information of the target to be operated on converted into a vector, the screen configuration information, and the image information, adjusts the obtained similarity of the elements on the screen according to the position of the target to be operated on described, and identifies the element with the highest adjusted similarity as the target to be operated on in the test operation.

10. The information processing apparatus according to claim 8 or 9, characterized in that the position of the object to be operated is an absolute position on the screen.

11. The information processing apparatus according to claim 8 or 9, characterized in that the position of the object to be operated on is a relative position to an element on the screen.

12. The information processing apparatus according to any one of claims 7 to 11, wherein the description of the object to be operated in the description data of the test operation further includes a description indicating the attributes of the object to be operated, and the conversion means converts the screen configuration information for each element on the screen and the image information for each element on the screen into vectors, excluding elements whose attributes do not match those of the attribute of the object to be operated.

13. The information processing apparatus according to claim 3, characterized in that, if a problem occurs when executing a test operation based on the generated test operation description data, the extraction means refers to the problem that occurred, performs inference using a machine learning model, and modifies the test operation description data.

14. An information processing method executed by an information processing device, comprising: an acquisition step of acquiring a test specification document indicating the content of a test, screen configuration information and image information relating to a screen from a test target; an extraction step of extracting test operations based on the content of the test indicated in the test specification document; a conversion step of converting information on the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen into vectors using a machine learning model that performs embedding; and an identification step of identifying the element that is the target of operation in the test operation from among the elements on the screen based on the information on the target of operation, the screen configuration information, and the image information converted into vectors.

15. A program for causing a computer to execute the following steps: an acquisition step of obtaining a test specification document showing the content of the test, screen configuration information and image information related to the screen from the test target; an extraction step of extracting test operations based on the content of the test shown in the test specification document; a conversion step of converting the information of the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen into vectors using a machine learning model that performs embedding; and an identification step of identifying the element of the target of operation in the test operation from among the elements on the screen based on the information of the target of operation, the screen configuration information, and the image information converted into vectors.

16. A computer-readable recording medium characterized by recording a program for causing a computer to execute: an acquisition step of acquiring a test specification document showing the content of the test, screen configuration information and image information related to the screen from the test target; an extraction step of extracting test operations based on the content of the test shown in the test specification document; a conversion step of converting the information of the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen into vectors using a machine learning model that performs embedding; and an identification step of identifying the element of the target of operation in the test operation from among the elements on the screen based on the information of the target of operation, the screen configuration information, and the image information converted into vectors.

Images