Evaluation device, evaluation method, and evaluation program

Abstract

This evaluation device, which performs software evaluation tests: stores, in association with each other in a storage device, a failure state occurring in the evaluation test and a recovery script including an operating procedure for at least one of equipment and software for successfully restarting the evaluation test from the occurrence of the failure; when a failure occurs in the evaluation test, identifies the failure state that has occurred, searches the storage device for the recovery script associated with the identified failure state, and automatically operates at least one of the equipment and the software on the basis of the operating procedure of the recovery script that was successfully retrieved.

Description

Evaluation apparatus, evaluation method, and evaluation program

[0001] The present disclosure relates to an evaluation apparatus, an evaluation method, and an evaluation program.

[0002] Conventionally, in device development, there is a technology for controlling experimental equipment by executing commands from a command line to perform an evaluation test on an evaluation target. Further, there is a technology for extracting a procedure capable of resolving an error when the error occurred during command execution is resolved. For example, Patent Document 1 discloses a CAD apparatus that, when an error occurring during command execution is resolved, re-executes all combinations of commands executed in the error resolution processing to extract the minimum number of processing procedures capable of resolving the error.

[0003] Japanese Unexamined Patent Publication No. 2000-3375

[0004] In device development, there are situations where evaluation tests on an evaluation target such as embedded software are repeatedly executed. Further, when an evaluation test is executed, in addition to defects caused by the evaluation target, defects may occur caused by the evaluation environment in which the evaluation target is provided, for example, peripheral equipment such as hardware in which software is embedded or experimental equipment.

[0005] When there are a wide variety of defect types, it is not practical to comprehensively cover defect resolution procedures in advance. However, due to the characteristic that evaluation tests are repeatedly executed, defects caused by similar causes may recur. Therefore, when a defect that has occurred in the past recurs, it is required that the operating procedure for resolving the defect be automatically executed, so that the defect can be resolved without requiring a large amount of man-hours. Even if the technology of Patent Document 1 can extract a procedure capable of resolving a defect, it cannot automatically execute the procedure when a similar defect recurs.

[0006] The present disclosure has been devised in view of the above-described conventional situation, and an object of the present disclosure is to automatically restore an evaluation test when a defect occurs during the evaluation test.

[0007] This disclosure provides an evaluation apparatus for performing software evaluation tests using at least one piece of equipment, comprising a processing unit and a storage device, wherein the storage device stores a malfunction state occurring during the evaluation test and a recovery script that includes operating procedures for at least one of the equipment and the software to successfully restart the evaluation test after the malfunction occurs, and the processing unit, when a malfunction occurs during the evaluation test, identifies the state of the malfunction, searches the storage device for the recovery script associated with the identified malfunction state, and automatically operates at least one of the equipment and the software based on the operating procedures of the successfully retrieved recovery script.

[0008] Furthermore, this disclosure provides an evaluation method for performing an evaluation test of software operating on predetermined hardware, wherein a predetermined storage device stores, linked to the state of a malfunction occurring in the evaluation test and a recovery script that includes operating procedures for at least one of the hardware and software to successfully restart the evaluation test after the occurrence of the malfunction, and the processing device, when a malfunction occurs in the evaluation test, identifies the state of the malfunction that occurred, searches the storage device for the recovery script linked to the identified malfunction state, and automatically operates at least one of the hardware and software based on the operating procedures of the recovery script that was successfully found.

[0009] Furthermore, this disclosure provides an evaluation program for performing evaluation tests on software operating on predetermined hardware, wherein a predetermined storage device stores a recovery script that includes the state of a malfunction occurring during the evaluation test and the operating procedure for at least one of the hardware and software that successfully restarts the evaluation test after the malfunction occurs, and the computer, when a malfunction occurs during the evaluation test, identifies the state of the malfunction that occurred, searches the storage device for the recovery script associated with the identified malfunction state, and automatically operates at least one of the hardware and software based on the operating procedure of the recovery script that was successfully found.

[0010] Furthermore, any combination of the above components, as well as any conversion of the expressions of this disclosure between methods, apparatus, systems, storage media, computer programs, etc., are also valid as aspects of this disclosure.

[0011] According to this disclosure, if a malfunction occurs during evaluation testing, the system can automatically recover to the evaluation test.

[0012] Block diagram showing an example configuration of the evaluation system according to Embodiment 1. Flowchart showing the recovery process of the evaluation device according to Embodiment 1. Schematic diagram for explaining the malfunction state and recovery script according to Embodiment 1. Schematic diagram for explaining the scripting of the recovery work of the evaluation device according to Embodiment 1. Schematic diagram for explaining the scripting of the recovery work of the evaluation device according to Embodiment 1. Schematic diagram for explaining the recovery script according to Embodiment 1.

[0013] The following description will detail embodiments specifically illustrating the evaluation apparatus, evaluation method, and evaluation program described herein, with reference to the drawings as appropriate. However, unnecessarily detailed explanations may be omitted. For example, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily verbose and to facilitate understanding by those skilled in the art. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand this disclosure and are not intended to limit the subject matter described in the claims.

[0014] (Embodiment 1) Figure 1 is a block diagram showing an example configuration of the evaluation system 10 according to Embodiment 1. The evaluation system 10 includes an evaluation environment 100, an evaluation device 200, and a user terminal 300. The evaluation system 10 is a system that allows a user, such as a software developer, to perform an evaluation test of an evaluation target 110 provided in the evaluation environment 100 by operating the evaluation device 200 via the user terminal 300.

[0015] In this embodiment, it is assumed that the evaluation target 110 of the evaluation test is software and is incorporated into the hardware 120. Furthermore, in this embodiment, it is assumed that peripheral equipment 130 is connected to the hardware 120 for the purpose of the evaluation test. The evaluation environment 100 in which the evaluation target 110, hardware 120, and peripheral equipment 130 are provided may be, for example, a room or facility for testing, experimentation, or research.

[0016] In the example shown in Figure 1, two peripheral devices 130 are connected to the hardware 120, but the number of peripheral devices 130 is not limited to this. For example, one peripheral device 130 may be connected to the hardware 120, or three or more peripheral devices 130 may be connected. Alternatively, the peripheral devices 130 may not even be provided in the evaluation environment 100.

[0017] The software, hardware 120, and peripheral devices 130 of the evaluation target 110 are not particularly limited, but for example, the evaluation target 110 may be an In-Vehicle Information (hereinafter referred to as "IVI") or car navigation system installed in a vehicle. In this case, the hardware 120 may be a head unit capable of running the IVI or car navigation system, or the vehicle itself. The peripheral devices 130 may be various sensors such as temperature sensors and acceleration sensors, communication equipment, audio equipment, or input / output devices.

[0018] The evaluation device 200 is an evaluation device that performs evaluation tests on the product under evaluation 110. The evaluation device 200 can control the product under evaluation 110, hardware 120, or peripheral devices 130 by executing commands entered by the user. Alternatively, the evaluation device 200 may control the product under evaluation 110, hardware 120, or peripheral devices 130 by executing a script.

[0019] In this embodiment, the evaluation device 200 is described as being configured using a general-purpose computer device such as a personal computer or server computer at a location where the evaluation environment 100 is set up, such as a research institute. For example, if the evaluation environment 100 and the evaluation device 200 are each set up in a research institute, the evaluation environment 100 and the evaluation device 200 may be set up in the same laboratory of the research institute, or they may be set up in separate laboratories. The evaluation device 200 is connected to the hardware 120 in a communicative manner so that evaluation tests of the object to be evaluated 110 can be performed. Although not shown in the figure, the evaluation device 200 may also be connected to peripheral devices 130 in a communicative manner.

[0020] The evaluation device 200 comprises a processing unit 201, a storage device 202, an input device 203, a display device 204, a communication device 205, and an external interface device 206. Each device included in the evaluation device 200 is configured to communicate with each other via an internal interface 207.

[0021] The processing unit 201 may be configured using, for example, a Central Processing Unit (hereinafter referred to as "CPU"), a Graphical Processing Unit (hereinafter referred to as "GPU"), a Micro Processing Unit (hereinafter referred to as "MPU"), a Digital Signal Processor (hereinafter referred to as "DSP"), or a Field Programmable Gate Array (hereinafter referred to as "FPGA"). The processing unit 201 realizes various functions, for example, by referring to various data stored in the storage device 202 or by reading programs.

[0022] The storage device 202 stores various data and programs. The storage device 202 may consist of volatile / non-volatile storage devices such as Random Access Memory (hereinafter referred to as "RAM"), Read Only Memory (hereinafter referred to as "ROM"), Hard Disk Drive (hereinafter referred to as "HDD"), and Solid State Drive (hereinafter referred to as "SSD").

[0023] The input device 203 accepts operations and instructions from users such as the developers of the product under evaluation 110. The input device 203 may consist of a mouse, keyboard, touch panel display, etc.

[0024] The display device 204 displays various user interfaces to the user. The display device 204 may consist of a liquid crystal display, a touch panel display, a lamp, etc.

[0025] The communication device 205 is an interface for communicating with external devices via a network. The communication standards supported by the communication device 205 are not particularly limited and may support either wired or wireless communication standards. Furthermore, the communication device 205 may support multiple communication standards. Therefore, the network used by the communication device 205 may be configured by combining networks using multiple communication standards.

[0026] The external interface device 206 is an interface for communicating with an external device.

[0027] Furthermore, the evaluation device 200 is not limited to being configured as an on-premise system, but may also be configured as a cloud-based system on a network. Alternatively, the evaluation device 200 may be configured as an on-premise system at a location different from the site where the evaluation environment 100 is located.

[0028] The user terminal 300 may be, for example, a stationary information processing device such as a personal computer, or a mobile terminal such as a tablet or smartphone. The hardware configuration of the user terminal 300 may be the same as that of the evaluation device 200 described above. The user terminal 300 and the evaluation device 200 are connected via a network so as to be able to communicate with each other.

[0029] In this embodiment, users such as software developers can operate the evaluation device 200 via the user terminal 300. For example, even if the user terminal 300 and the evaluation device 200 are located at different locations, the user can remotely operate the evaluation device 200 via the user terminal 300. The user can then perform evaluation tests on the evaluation target 110 using the evaluation device 200. If the user performs evaluation tests on the evaluation target 110 by directly operating the evaluation device 200, the user terminal 300 can be omitted in the evaluation system 10.

[0030] The evaluation tests of the evaluation target 110 may be performed repeatedly. Furthermore, during the execution of the evaluation tests, in addition to defects caused by the evaluation target 110, which is software, defects caused by the hardware 120 or peripheral devices 130 may occur. For example, when performing the evaluation tests, wiring errors or omissions in the pre-configuration of the hardware 120 or peripheral devices 130 may occur. For example, defects caused by the evaluation environment 100 other than the evaluation target 110 may occur more frequently when multiple users use the same evaluation environment 100. This is because the settings of the hardware 120 or peripheral devices 130 may differ for each evaluation test performed by each of the multiple users.

[0031] In addition to defects caused by the software under evaluation 110, defects caused by hardware 120 or peripheral devices 130 may occur, resulting in a wide variety of defects. Therefore, it is not practical to comprehensively cover all defect resolution procedures in advance. However, given that evaluation tests are repeatedly performed, defects caused by similar factors may recur. Therefore, it is desirable that, when a previously occurring defect recurs, the operation procedure used to resolve that defect is automatically executed, allowing the defect to be resolved without requiring significant effort.

[0032] The evaluation device 200 can automatically resume the evaluation test if a malfunction occurs during the evaluation test. Referring to Figure 2, the processing flow when the evaluation device 200 detects a malfunction during the evaluation test will be explained. Figure 2 is a flowchart of the recovery process of the evaluation device 200 according to Embodiment 1. At the start of the flowchart shown in Figure 2, it is assumed that a malfunction has occurred in the evaluation test and the evaluation test has been interrupted. Furthermore, each process in the flowchart shown in Figure 2 is performed by the processing unit 201 of the evaluation device 200.

[0033] The processing unit 201 detects a defect that occurred during the evaluation test (step S1000). The method by which the processing unit 201 detects the defect is not particularly limited. The processing unit 201 may detect the defect by, for example, monitoring the processing time, performance data, error log of the evaluation device 200, or the status of the evaluation target 110, hardware 120, or peripheral equipment 130.

[0034] The processing unit 201 acquires the status of the evaluation target 110 and each device, and identifies the state of the malfunction detected in step S1000 (step S1001). In this embodiment, the status of the evaluation target 110 and each device when the malfunction occurs is collectively referred to as the state of the malfunction. The above-mentioned devices refer to the hardware 120 and peripheral devices 130.

[0035] The processing unit 201 searches the storage device 202 for a recovery script associated with the malfunction identified in step S1001 (step S1002). This will be explained in more detail with reference to Figure 3.

[0036] Figure 3 is a schematic diagram illustrating the malfunction state and recovery script according to Embodiment 1. As shown in Figure 3, the storage device 202 stores the malfunction state and the scripted recovery procedure linked together.

[0037] In this embodiment, the scripted recovery procedure is referred to as a recovery script. The recovery script may include operating procedures for at least one of the equipment and the evaluation target 110 that successfully resume the evaluation test after a malfunction occurs. Here, the equipment may be hardware 120 or peripheral equipment 130. The processing unit 201 can automatically operate at least one of the equipment and the evaluation target 110 based on the operating procedures included in the recovery script. Hereinafter, for the sake of simplicity, the automatic operation of at least one of the equipment and the evaluation target 110 by the processing unit 201 based on the operating procedures included in the recovery script may be referred to as executing the recovery script. By executing the recovery script, the processing unit 201 can resolve the malfunction associated with the recovery script. In this embodiment, the evaluation test is resumed when the malfunction is resolved. The malfunction state and the recovery script may, for example, be stored in the storage device 202 in advance by the user, or they may be stored in the storage device 202 by the processing from steps S1007 to S1011 described later.

[0038] For example, the storage device 202 stores the status ST1 and the recovery script RS1 in association. The malfunction status ST1 indicates that the status of the software, device A, and device B are "OK," and the status of device C is "Error X." Here, a status of "OK" for the software or device means that no malfunction has been detected with respect to that software or device. A status of "Error X" for the software or device means that a malfunction called "Error X" has been detected with respect to that software or device. For illustrative purposes, in Figure 3, the software represents the evaluation target 110, device A represents the hardware 120, and device B and device C represent peripheral devices 130, respectively. If there are further software or devices used in the evaluation test, the malfunction status stored in the storage device 202, such as status ST1, may also include the status of those software or devices.

[0039] In the storage device 202, the recovery script RS1 is associated with state ST1. By executing the recovery script RS1, the malfunction caused by state ST1 can be resolved.

[0040] Let's return to the explanation of Figure 2. The states ST2, recovery script RS2, state ST50, and recovery script RS50 shown in Figure 3 will be described later.

[0041] As described above, the processing unit 201 searches the storage device 202, which stores the malfunction state and the recovery script associated with it, for the recovery script associated with the malfunction state identified in step S1001 (step S1002). More precisely, the processing unit 201 searches the storage device 202 for a recovery script associated with a malfunction state similar to the one identified in step S1001.

[0042] The processing unit 201 determines whether or not there is a recovery script in the storage device 202 that is associated with the malfunction state identified in step S1001 (step S1003).

[0043] First, a description will be given of the case where the storage device 202 has a recovery script associated with the failure state identified in step S1001. If the processing device 201 determines that the storage device 202 has a recovery script associated with the failure state identified in step S1001 (step S1003: YES), the processing device 201 advances the process to step S1004. If the processing device 201 determines that there is a recovery script associated with the failure state identified in step S1001, the processing device 201 may also determine that the search in step S1002 was successful.

[0044] For example, a description will be given of the case where the failure state identified by the processing device 201 in step S1001 is the same as the state ST1. In the storage device 202, the state ST1 and the recovery script RS1 are stored in association with each other. Therefore, based on the state ST1 and the recovery script RS1, the processing device 201 can determine that the storage device 202 has a recovery script associated with the failure state identified in step S1001.

[0045] The processing device 201 executes the recovery script searched in step S1002, which is associated with the failure state identified in step S1001 (step S1004).

[0046] The processing device 201 determines whether the failure detected in step S1000 has been resolved (step S1005). Here, the method by which the processing device 201 determines whether the failure has been resolved is not particularly limited. The processing device 201 may be able to determine whether the failure has been resolved, for example, by monitoring the processing time, performance data, or error logs of the evaluation device 200, or monitoring the states of the evaluation target 110, the hardware 120, or the peripheral device 130.

[0047] If the processing device 201 determines that the failure detected in step S1000 has been resolved (step S1005: YES), the processing device 201 terminates this processing flow. As described above, when a failure occurs during an evaluation test and the evaluation test is interrupted, the evaluation device 200 can automatically recover the evaluation test by executing a recovery script capable of resolving the failure.

[0048] If the processing device 201 determines that the failure detected in step S1000 has not been resolved (step S1005: NO), the processing device 201 proceeds the process to step S1006 described later.

[0049] Next, a case will be described where the storage device 202 does not have a recovery script associated with the failure state identified in step S1001.

[0050] If the processing device 201 determines that the storage device 202 does not have a recovery script associated with the failure state identified in step S1001 (step S1003: NO), the processing device 201 proceeds the process to step S1006. If the processing device 201 determines that there is no recovery script associated with the failure state identified in step S1001, the processing device 201 may determine that the search in step S1002 has failed.

[0051] The processing device 201 accepts a recovery operation performed by a user operation (step S1006). Then, the processing device 201 optimizes the recovery operation accepted in step S1006 (step S1007). A more detailed description will be given with reference to FIG. 4 and FIG. 5.

[0052] FIG. 4 is a schematic diagram for explaining scripting of recovery work of the evaluation apparatus 200 according to the first embodiment. In the present embodiment, the evaluation apparatus 200 activates evaluation software for an evaluation test during the evaluation test of the evaluation target 110. FIG. 4 shows a screen G1 displayed on the display device 204 when the evaluation apparatus 200 activates the evaluation software. Note that the screen G1 may also be displayed on a display device of the user terminal 300. A user can execute various functions of the evaluation software by operating the user terminal 300, thereby enabling an evaluation test to be performed even from a remote location away from the evaluation environment 100 or the evaluation apparatus 200.

[0053] The display content of the screen G1 includes information I1 and information I2. The information I1 indicates information of the evaluation target 110. The information I1 may indicate, for example, the name or reference number of the evaluation target 110. The information I2 indicates user information. The information I2 may indicate, for example, the user's name or affiliation.

[0054] Furthermore, the display content of screen G1 includes areas R1, R2, and R3. Area R1 may display, for example, the operation screen of the evaluation target 110. For example, if the evaluation target 110 is a car navigation system, area R1 may display the operation screen of the car navigation system. The user can perform operations such as checking the operation of the car navigation system via area R1, for example, by touch operation.

[0055] Area R2 displays content corresponding to the selected tab from among tabs T1, T2, T3, T4, T5, T6, T7, and T8. In the example in Figure 4, tab T1 is selected. Tab T1 may be a tab for displaying operation buttons, etc., in area R2 for controlling peripheral devices 130 and controlling signals of hardware 120. When tab T1 is selected, area R2 may display operation buttons or tabs, etc., for controlling peripheral devices 130 or controlling signals of hardware 120. The user can perform the desired operation by selecting tab T1 when controlling peripheral devices 130 or controlling signals of hardware 120.

[0056] An example of the case where tab T5 is selected will be explained with reference to Figure 5. Figure 5 is a schematic diagram for explaining the scripting of the recovery operation of the evaluation device 200 according to Embodiment 1. In the example in Figure 5, tab T5 is selected. Tab T5 may be a tab for displaying the history of various user operations in area R2, such as signal control operations of hardware 120 or user touch operations in area R1. When tab T5 is selected, the history of various user operations may be displayed in area R2. If a user wants to check the history of various operations performed in the evaluation test, they can check the operation history so far by selecting tab T5.

[0057] Area R3 displays content corresponding to the selected tab from among tabs T9, T10, T11, T12, T13, T14, T15, T16, and T17. In the example in Figure 4 or Figure 5, tab T9 is selected. Tab T9 may be, for example, a tab for displaying logs output from the evaluation target 110 in area R3. When tab T9 is selected, logs output from the evaluation target 110 may be displayed in real time in area R3. By checking the logs output from the evaluation target 110 in area R3, the user can, for example, verify the operation of the evaluation target 110.

[0058] The user can control the evaluation target 110, hardware 120, and peripheral devices 130 by operating each part displayed on screen G1. If a malfunction occurs and the evaluation test is interrupted, the user can perform recovery work by operating each part displayed on screen G1 as described above.

[0059] Furthermore, the example configuration of screen G1 of the evaluation software for evaluation testing shown in Figures 4 and 5, and the example of user operation explained with reference to Figures 4 and 5, are merely illustrative examples. In other words, Figures 4 and 5 are examples of screen configurations and specific examples of user operation for evaluation software for evaluation testing or recovery work, and are not intended to be interpreted as being limited to these examples.

[0060] For the sake of explanation, we will assume that the malfunction detected in step S1000 has been resolved by the recovery work received by the processing unit 201 in step S1006 of the flowchart shown in Figure 2. The user may resolve the malfunction through trial and error during the recovery work. In other words, the user may not necessarily take the shortest path to resolve the malfunction. Therefore, the recovery work received by the processing unit 201 in step S1006 may include work unrelated to resolving the malfunction, or work that is performed in many steps and can be shortened to fewer steps. In step S1007, the processing unit 201 may remove the steps of work unrelated to resolving the malfunction from the series of steps included in the recovery work received in step S1006, or combine some of the steps in that series into fewer steps. In this way, the processing unit 201 may optimize the steps of the recovery work received in step S1006. Note that the recovery procedure received by the processing unit 201 in step S1006 may be the same before and after the processing in step S1007.

[0061] The processing unit 201 receives the data in step S1006, scripts the optimized recovery procedure in step S1007, and generates a recovery script (step S1008). A specific example will be explained with reference to Figure 6.

[0062] Figure 6 is a schematic diagram illustrating a recovery script according to Embodiment 1. The processing unit 201 generates a recovery script RS3, for example, by scripting the steps of a recovery operation performed by a user. The method for scripting the recovery operation is not particularly limited, and known techniques may be used. The recovery script generated by the processing unit 201 may include scripts for each step. For example, the recovery script RS3 shown in Figure 6 includes scripts 30, 31, and 32. The recovery script RS3 may include titles that show an overview of the scripts included in the recovery script RS3. For example, title 40 shows an overview of script 31, and title 41 shows an overview of script 32. Including titles in the recovery script can improve the efficiency of verification when a user checks the contents of the recovery script, for example.

[0063] Note that the recovery script RS3 shown in Figure 6 is an example for illustrative purposes only, and the recovery scripts generated by the processing unit 201 are not limited to this example.

[0064] Returning to the explanation of Figure 2, if the processing unit 201 has a recovery script in the storage device 202 that is associated with the malfunction identified in step S1001 (step S1009: YES), it proceeds to step S1010. In other words, if the processing unit 201 proceeds from step S1005 to step S1006, it executes the process in step S1010 after the process in step S1008.

[0065] The processing unit 201 updates the recovery script already stored in the storage device 202 with the recovery script generated in step S1008 (step S1010). This will be explained in more detail with reference to Figure 3.

[0066] For example, if the malfunction identified by the processing unit 201 in step S1001 is state ST2 as shown in Figure 3, the processing unit 201 executes the recovery script RS2 in step S1004. However, since the malfunction was not resolved by the recovery script RS2, the processing unit 201 generates a new recovery script in step S1008 based on the recovery work performed by the user. In step S1008, the processing unit 201 updates the recovery script RS2 already stored in the storage device 202 with the newly generated recovery script capable of resolving the malfunction in state ST2. As a result, in the storage device 202, state ST2 and the recovery script generated in step S1008, i.e., the updated recovery script RS2, are linked and stored together. Furthermore, if a malfunction occurs in a future evaluation test that is the same as state ST2, the processing unit 201 can resolve the malfunction by executing the recovery script RS2 linked to state ST2.

[0067] Returning to the explanation of Figure 2, if the processing unit 201 does not have a recovery script associated with the malfunction identified in step S1001 in the storage device 202 (step S1009: NO), it proceeds to step S1011. In other words, if the processing unit 201 proceeds from step S1003 to step S1006, it executes the process in step S1011 after the process in step S1008.

[0068] The processing unit 201 stores the malfunction status identified in step S1001 and the recovery script generated in step S1008 in the storage device 202, associating them (step S1011). This will be explained in more detail with reference to Figure 3.

[0069] For example, if the malfunction identified by the processing unit 201 in step S1001 is state ST50 as shown in Figure 3, the processing unit 201 proceeds to step S1003 through step S1006. This is because the storage device 202 does not store a state similar to state ST50, and therefore the processing unit 201 fails to search for the recovery script in step S1002.

[0070] Then, based on the recovery operation performed by the user, the processing unit 201 generates a recovery script RS50 in step S1008 that can resolve the malfunction in state ST50. Subsequently, the processing unit 201 links the malfunction state ST50 identified in step S1001 with the recovery script RS50 generated in step S1008 and stores it in the storage device 202. As a result, if a malfunction that occurs in a future evaluation test is in the same state as state ST50, the processing unit 201 can resolve the malfunction by executing the recovery script RS50 linked to state ST50.

[0071] Returning to the explanation of Figure 2, the processing unit 201 terminates this processing flow after processing in step S1010 or step S1011. In this way, the evaluation device 200 accepts recovery work by user operation when a malfunction occurs during the evaluation test and the evaluation test is interrupted. The evaluation device 200 can then generate a recovery script that can resolve the malfunction by scripting the procedure of the accepted recovery work.

[0072] In this way, the evaluation device 200 can perform an evaluation test of the software under evaluation 110 using at least one device, for example, hardware 120. The evaluation device 200 may also use peripheral devices 130 in addition to hardware 120 as equipment for the evaluation test. The storage device 202 of the evaluation device 200 stores a link between the state of a malfunction that occurs during the evaluation test and a recovery script that includes the operating procedure for at least one of the devices and software that successfully restarts the evaluation test after the malfunction occurs. The processing unit 201 of the evaluation device 200 can identify the state of the malfunction that occurred during the evaluation test. The processing unit 201 can then search the storage device 202 for the recovery script linked to the identified malfunction state. Furthermore, the processing unit 201 automatically operates at least one of the devices and software based on the operating procedure of the successfully retrieved recovery script. As a result, the evaluation device 200 can resolve the malfunction and automatically recover to the evaluation test.

[0073] In this embodiment, an example is shown in which the malfunction state and a recovery script capable of resolving the malfunction are linked and stored in the storage device 202 of the evaluation device 200. However, it is not limited to this, and the malfunction state and the recovery script capable of resolving the malfunction may also be linked and stored in an external device (not shown) capable of data communication with the evaluation device 200.

[0074] (Summary of Embodiment 1) The above description of Embodiment 1 discloses at least the following technologies. The components etc. in parentheses are examples of those corresponding in Embodiment 1, but are not limited to these.

[0075] (Technology 1) An evaluation device (e.g., evaluation device 200) that performs an evaluation test of software (e.g., software under evaluation 110) using at least one piece of equipment (e.g., hardware 120) comprises a processing unit (e.g., processing unit 201) and a storage device (storage device 202). The storage device stores, in association with the state of a malfunction that occurs during the evaluation test (e.g., state ST1) and a recovery script (e.g., recovery script RS1) that includes the operating procedure for at least one of the equipment and software that successfully restarts the evaluation test after the malfunction occurs. When a malfunction occurs during the evaluation test, the processing unit identifies the state of the malfunction that occurred, searches the storage device for a recovery script associated with the identified malfunction state, and automatically operates at least one of the equipment and software based on the operating procedure of the successfully retrieved recovery script.

[0076] This allows the evaluation device to identify the state of a malfunction if one occurs during the evaluation test. The evaluation device can then retrieve a recovery script linked to the identified malfunction from a storage device that stores the malfunction state and a recovery script containing the operating procedures for resolving the malfunction and returning to the evaluation test. If the retrieval is successful, the evaluation device can automatically operate at least one of the equipment and software based on the operating procedures contained in the successfully retrieved recovery script. This enables the evaluation device to automatically return to the evaluation test if a malfunction occurs during the evaluation test.

[0077] (Technology 2) In the evaluation apparatus described in Technology 1, the apparatus includes predetermined hardware (e.g., hardware 120) and peripheral devices connected to the hardware (e.g., peripheral devices 130), the software operates on the hardware, and the recovery script may include operating procedures for the hardware and peripheral devices.

[0078] This allows the evaluation device to automatically operate at least one of the hardware, peripherals, and software based on a recovery script, thereby resolving the malfunction and returning to evaluation testing.

[0079] (Technology 3) In the evaluation apparatus described in Technology 1 or 2, if the processing device fails to search for a recovery script associated with the identified defect state, it may accept a recovery operation to restart the evaluation test, which includes operation of at least one of the equipment and software by the user, and in response to the acceptance of the recovery operation, it may script the recovery operation to generate a recovery script that includes operation procedures for at least one of the equipment and software that will successfully restart the evaluation test, and store the generated recovery script in a storage device in association with the identified defect state.

[0080] This allows the evaluation device to accept recovery operations from the user even if there is no recovery script linked to the state of the malfunction that occurred in the storage device. The evaluation device can then script the accepted recovery operation to generate a recovery script, link the generated recovery script to the state of the malfunction, and store it in the storage device. As a result, the next time a similar malfunction occurs, the evaluation device can operate the equipment or software based on the recovery script linked to the state of that malfunction and resolve the malfunction.

[0081] (Technology 4) In the evaluation apparatus described in Technology 1, 2, or 3, the processing device determines whether the malfunction that occurred has been resolved by operating at least one of the equipment and software based on the operating procedure of the recovery script. If it determines that the malfunction has not been resolved, the processing device accepts a recovery operation to restart the evaluation test, which includes operating at least one of the equipment and software by the user. In response to the acceptance of the recovery operation, the processing device scripts the recovery operation to generate a recovery script that includes the operating procedure of at least one of the equipment and software that will successfully restart the evaluation test. The generated recovery script and the state of the identified malfunction may be linked and stored in the storage device.

[0082] This allows the evaluation device to accept user-initiated recovery work even if it cannot resume the evaluation test by operating the equipment or software based on the recovery script. The evaluation device can then script the accepted recovery work to generate a recovery script, link the generated recovery script to the state of the malfunction that occurred, and store it in a memory device.

[0083] (Technical 5) In the evaluation apparatus described in Technical 4, the processing device may update the recovery script that was successfully retrieved and stored in the storage device with a recovery script generated based on the recovery work received when it is determined that the malfunction that occurred has not been resolved.

[0084] This allows the evaluation device to update the recovery script, which includes the operating procedures that failed to restart the evaluation test, with a recovery script that includes the operating procedures that succeeded in restarting the evaluation test.

[0085] (Technical 6) In the evaluation apparatus described in any one of Technical 3 to 5, the processing device may script the recovery work after excluding the operating procedures unrelated to resolving the malfunction from the operating procedures included in the recovery work by the user.

[0086] This allows the evaluation device to exclude from scripting any operations performed by the user on the equipment and / or software during the recovery process that are unrelated to resolving the malfunction. As a result, the evaluation device can generate a more concise recovery script.

[0087] (Technical 7) In the evaluation apparatus described in any one of Technical 3 to 6, the processing device may, after consolidating several operating procedures included in the user's recovery work into fewer operating procedures that produce the same result as the result of those multiple operating procedures, script the recovery work.

[0088] This allows the evaluation device to consolidate multiple operations performed by the user on the equipment and / or software during the recovery process into fewer steps, and then script those operations into fewer steps. As a result, the evaluation device can generate a more concise recovery script.

[0089] (Technical 8) In the evaluation apparatus described in any one of Technical 3 to 7, the processing device may accept recovery work performed remotely by the user.

[0090] This allows the evaluation device to script recovery operations performed remotely by the user, even when the user is in a remote location and cannot directly operate the equipment used for evaluation testing.

[0091] (Technical 9) An evaluation method for performing evaluation tests on software operating on predetermined hardware is provided, in which a predetermined storage device stores a record of the state of a malfunction that occurs during the evaluation test and a recovery script that includes the operating procedure for at least one of the hardware and software that successfully restarts the evaluation test after the malfunction occurs. When a malfunction occurs during the evaluation test, the processing device identifies the state of the malfunction that occurred, searches the storage device for the recovery script associated with the identified malfunction state, and automatically operates at least one of the hardware and software based on the operating procedure of the recovery script that was successfully found.

[0092] As a result, the evaluation method can achieve the same effect as Technique 1.

[0093] (Technical 10) If the evaluation method described in Technical 9 fails to find a recovery script associated with the identified defect state, it may accept a recovery operation to restart the evaluation test, which includes at least one operation of hardware and software by the user, and generate a recovery script that includes at least one operation procedure of hardware and software that will successfully restart the evaluation test by scripting the recovery operation in response to the acceptance of the recovery operation, and store the generated recovery script in a storage device in association with the identified defect state.

[0094] As a result, the evaluation method can achieve the same effect as technology 3.

[0095] (Technical 11) The evaluation method described in Technical 9 or 10 determines whether the malfunction has been resolved by operating at least one of the hardware and software based on the operating procedure of the recovery script. If it is determined that the malfunction has not been resolved, it accepts a recovery operation to restart the evaluation test, which includes operating at least one of the hardware and software by the user. In response to the acceptance of the recovery operation, it scripts the recovery operation to generate a recovery script that includes the operating procedure of at least one of the hardware and software that will successfully restart the evaluation test. The generated recovery script and the state of the identified malfunction may be linked and stored in a storage device.

[0096] As a result, the evaluation method can achieve the same effect as technology 4.

[0097] (Technical 12) The evaluation method described in Technical 11 may update the recovery script that was successfully found in the storage device with a recovery script generated based on the recovery work received when it is determined that the problem that occurred has not been resolved.

[0098] As a result, the evaluation method can achieve the same effect as technology 5.

[0099] (Technical 13) The evaluation method described in any one of Technical 10 to 12 may involve scripting the recovery work after excluding operational procedures unrelated to resolving the problem from the operational procedures included in the recovery work performed by the user.

[0100] As a result, the evaluation method can achieve the same effect as technology 6.

[0101] (Technical 14) The evaluation method described in any one of Technical 10 to 13 may involve combining multiple operating procedures included in the user's recovery work into fewer operating procedures that achieve the same result as the result of those multiple operating procedures, and then scripting the recovery work.

[0102] As a result, the evaluation method can achieve the same effect as technology 7.

[0103] (Technical 15) The evaluation method described in any one of Technical 10 to 14 may accept recovery work performed remotely by the user.

[0104] As a result, the evaluation method can achieve the same effect as technology 8.

[0105] (Technical 16) An evaluation program that performs evaluation tests on software operating on predetermined hardware stores in a predetermined storage device the state of a malfunction that occurs during the evaluation test and a recovery script that includes the operating procedure for at least one of the hardware and software that successfully restarts the evaluation test after the malfunction occurs. When a malfunction occurs during the evaluation test, the computer identifies the state of the malfunction that occurred, searches the storage device for the recovery script associated with the identified state of the malfunction, and automatically operates at least one of the hardware and software based on the operating procedure of the recovery script that was successfully found.

[0106] This allows the program to achieve the same effect as Technique 1.

[0107] The functions of the above-described embodiment can also be realized by supplying programs and applications for realizing the functions of the above-described embodiment to a system or device using a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the programs.

[0108] Furthermore, the functions of the above-described embodiment may be implemented by a circuit that implements one or more functions, such as an Application Specific Integrated Circuit (hereinafter referred to as "ASIC") or an FPGA.

[0109] While embodiments of this disclosure have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these are also understood to fall within the technical scope of this disclosure. Furthermore, the components of the embodiments described above can be combined in any way without departing from the spirit of the invention.

[0110] This application is based on a Japanese patent application (JP 2025-042631) filed on March 17, 2025, the contents of which are incorporated herein by reference.

[0111] This disclosure is useful as an evaluation device, evaluation method, and evaluation program.

[0112] 10 Evaluation System 100 Evaluation Environment 110 Evaluation Target 120 Hardware 130 Peripherals 200 Evaluation Device 201 Processing Unit 202 Storage Device 203 Input Device 204 Display Device 205 Communication Device 206 External Interface Device 207 Internal Interface 300 User Terminals ST1, ST2, ST50 Status RS1, RS2, RS3, RS50 Recovery Script

Claims

1. An evaluation device for performing software evaluation tests using at least one piece of equipment, comprising a processing unit and a storage device, wherein the storage device stores, in association with the state of a malfunction occurring in the evaluation test and a recovery script that includes the operation procedure of at least one of the piece of equipment and the software for successfully restarting the evaluation test after the malfunction occurs, and the processing unit, when a malfunction occurs in the evaluation test, identifies the state of the malfunction that occurred, searches the storage device for the recovery script associated with the identified malfunction state, and automatically operates at least one of the piece of equipment and the software based on the operation procedure of the recovery script that was successfully found.

2. The evaluation apparatus according to claim 1, wherein the apparatus includes predetermined hardware and peripheral devices connected to the hardware, the software operates on the hardware, and the recovery script includes operating procedures for the hardware and the peripheral devices.

3. The evaluation apparatus according to claim 1, wherein if the processing device fails to search for the recovery script associated with the identified defect state, it accepts a recovery operation to restart the evaluation test, which includes operations on at least one of the equipment and the software by the user; in response to the acceptance of the recovery operation, it scripts the recovery operation to generate a recovery script which includes the operation procedure for at least one of the equipment and the software that successfully restarts the evaluation test; and it stores the generated recovery script and the identified defect state in the storage device.

4. The evaluation apparatus according to claim 1, wherein the processing device determines whether the malfunction that occurred has been resolved by operating at least one of the equipment and the software based on the operating procedure of the recovery script, and if it determines that the malfunction has not been resolved, it accepts a recovery operation to restart the evaluation test, which includes operating at least one of the equipment and the software by the user, and generates the recovery script which includes operating procedure for at least one of the equipment and the software that successfully restarts the evaluation test by scripting the recovery operation in response to the acceptance of the recovery operation, and stores the generated recovery script and the state of the identified malfunction in the storage device.

5. The evaluation device according to claim 4, wherein the processing device updates the recovery script that was successfully retrieved and stored in the storage device with the recovery script generated based on the recovery work received when it is determined that the malfunction that occurred has not been resolved.

6. The evaluation apparatus according to claim 3 or 4, wherein the processing apparatus scripts the recovery work after removing from the operating procedures included in the recovery work performed by the user any operating procedures unrelated to resolving the malfunction.

7. The evaluation apparatus according to claim 3 or 4, wherein the processing apparatus consolidates a plurality of operation procedures among the operation procedures included in the recovery work performed by the user into fewer operation procedures that achieve the same operation result as the operation result of the plurality of operation procedures, and then scripts the recovery work.

8. The evaluation apparatus according to claim 3 or 4, wherein the processing apparatus accepts the recovery work performed remotely by the user.

9. An evaluation method for performing an evaluation test of software operating on predetermined hardware, wherein a predetermined storage device stores, in association with the state of a malfunction occurring in the evaluation test and a recovery script that includes the operation procedure of at least one of the hardware and the software to successfully restart the evaluation test after the occurrence of the malfunction, and the processing device, when a malfunction occurs in the evaluation test, identifies the state of the malfunction that occurred, searches the storage device for the recovery script associated with the identified malfunction state, and automatically operates at least one of the hardware and the software based on the operation procedure of the recovery script that was successfully found.

10. The evaluation method according to claim 9, wherein if the search for the recovery script associated with the identified defect state fails, the evaluation method accepts a recovery operation by the user to restart the evaluation test, which includes operations on at least one of the hardware and software; generates a recovery script that includes operations on at least one of the hardware and software that successfully restart the evaluation test by scripting the recovery operation in response to the acceptance of the recovery operation; and stores the generated recovery script and the identified defect state in the storage device.

11. The evaluation method according to claim 9, comprising: determining whether the malfunction that occurred has been resolved by operating at least one of the hardware and software based on the operating procedure of the recovery script; if it is determined that the malfunction has not been resolved, accepting a recovery operation by the user to restart the evaluation test, which includes operating at least one of the hardware and software; generating the recovery script which includes operating procedure for at least one of the hardware and software that successfully restarts the evaluation test by scripting the recovery operation in response to the acceptance of the recovery operation; and storing the generated recovery script and the state of the identified malfunction in the storage device.

12. The evaluation method according to claim 11, wherein the recovery script that was successfully searched and stored in the storage device is updated with the recovery script generated based on the recovery work received when it is determined that the malfunction that occurred has not been resolved.

13. The evaluation method according to claim 10 or 11, wherein the recovery work is scripted after removing from the operation procedures included in the recovery work performed by the user any operation procedures unrelated to resolving the malfunction.

14. The evaluation method according to claim 10 or 11, wherein, among the operation procedures included in the recovery work performed by the user, multiple operation procedures are combined into fewer operation procedures that achieve the same operation result as the operation result of the multiple operation procedures, and then the recovery work is scripted.

15. The evaluation method according to claim 10 or 11, which accepts the recovery work performed remotely by the user.

16. An evaluation program for performing evaluation tests on software operating on predetermined hardware, wherein a predetermined storage device stores, in association with the state of a malfunction occurring during the evaluation test and a recovery script that includes operating procedures for at least one of the hardware and software to successfully resume the evaluation test after the malfunction occurs, and the evaluation program causes a computer to identify the state of the malfunction that occurred when a malfunction occurs during the evaluation test, to search the storage device for the recovery script associated with the identified malfunction state, and to automatically operate at least one of the hardware and software based on the operating procedures of the recovery script that was successfully found.

Images