Binary fuzzing device, binary fuzzing method, and program

Abstract

A binary fuzzing device (1) comprises: an input generation unit (11) that generates one or more fuzzes by mutating a seed; a test target execution unit (12) that inputs the fuzzes into test target software (800) for execution thereof; a system information collection unit (13) that collects system information of an execution environment (700) in which the test target software (800) is operated, when the test target software (800) is executed; and a coverage estimation unit (14) that estimates whether or not coverage has increased using the collected system information.

Description

Binary Fuzzing Device, Binary Fuzzing Method, and Program 【0001】 The present invention relates to a binary fuzzing device, a binary fuzzing method, and a program. 【0002】 One of the techniques for detecting software vulnerabilities is a technique called "fuzzing". Fuzzing is a test technique for automatically / semi-automatically finding software vulnerabilities and defects. A large amount of "fuzz" is given to the software to be tested, and its behavior is confirmed to find vulnerabilities and defects. "Fuzz" is data that is unexpected for that software, data that is likely to cause an exception, and data that is likely to crash. 【0003】 FIG. 22 is an explanatory diagram of a test by fuzzing (referred to as a "fuzz test"). A fuzzing device 9 equipped with a fuzzing tool 90 inputs fuzz to the software 800 to be tested and monitors its response. Here, for example, when the normal input is "GET / HTTP / 1.1\r\n", input data such as inserting unexpected data "???" such as "GET???HTTP / 1.1\r\n" and "GET / HTTP???1.1\r\n" as fuzz, or replacing it with an unexpected numerical value such as "GET / HTTP / 100\r\n" is input to the software 800 to be tested and executed. Thereby, when there is no response from the software 800 to be tested or the response is abnormal, vulnerabilities and defects of the software 800 to be tested can be detected. 【0004】A well-known conventional fuzzing technique is AFL (American Fuzzy Lop), which efficiently performs fuzzing by using the coverage of 800 pieces of software under test (see Non-Patent Documents 1 and 2). Here, "coverage" is an indicator of the test scope of a program, identifying which parts of the program (execution paths) are being tested and indicating the extent to which testing has been performed as a percentage. The higher the coverage, the more extensive the testing, making it easier to find and fix vulnerabilities and defects. The technique described in Non-Patent Documents 1 and 2 (AFL) is called coverage-based fuzzing. 【0005】 Figure 23 is a diagram illustrating the overview of AFL, a conventional fuzzing technique. Figure 24 is a diagram illustrating the fuzz generation and coverage determination processes of AFL, a conventional fuzzing technique. 【0006】 As shown in Figure 23, the fuzzing device 9 using AFL technology is equipped with a fuzzing tool 90 that generates fuzz and performs tests on the software under test 800. The fuzzing tool 90 calculates coverage in block units separated by conditional branches by instrumenting during the compilation of the software under test 800. Here, "instrumentation" is a technique used as part of fuzzing, which involves inserting code to monitor and record the internal operation of the software. Furthermore, if the software under test 800 cannot be compiled using only a binary file, the emulator 850 dynamically performs instrumentation. 【0007】 Next, we will explain the fuzz generation and coverage determination processes performed by AFL. In AFL, as shown in Figure 24, initial test cases are prepared in accordance with the input of the software under test 800 and stored in the seed queue (step S1). The data that forms the basis of the test cases is also called the "seed". 【0008】Then, the fuzzing tool 90 (Figure 23) is used to select a test case from the seed queue, performing seed selection (step S2). This seed selection may be done in a round-robin manner, or seeds may be selected with priority given to them. Here, we assume that test case A is selected as the seed, as shown in Figure 24. 【0009】 Next, mutation processing is performed on the selected seed (step S3) to generate fuzz. Here, mutation processing is performed on test case A, and test case A １ , A ２ A ３ ...is generated as a fuzz. 【0010】 Next, the fuzzing tool 90 (Figure 23) generates the fuzz (Test Case A) １ , A ２ , A ３ The following are sequentially input to the software under test 800 (step S4), and the execution path is observed (step S5) to confirm whether it is a new execution path and to determine whether the coverage has increased (coverage determination). Then, the test case that is determined to have branched to a new execution path (for example, test case A) １ ) is added to the seed queue (step S6). In this way, AFL performs a fuzz test by repeatedly generating fuzz and determining coverage. 【0011】 “american fuzzy lop,” Github, [online], [retrieved November 1, 2020], Internet <URL: https: / / github.com / google / AFL> Andrea Fioraldi, et.al., “AFL++: Combining Incremental Steps of Fuzzing Research,” USENIX, [online], [searched on November 1, 2020], Internet <URL: https: / / www.usenix.org / system / files / woot20-paper-fioraldi.pdf> 【0012】Using AFL described in Non-Patent Documents 1 and 2 as an example, in "coverage-based fuzzing," if the software under test 800 cannot be compiled using only a binary file, the emulator 850 (Figure 23) dynamically performs instrumentation. However, dynamic instrumentation by the emulator 850 is 2 to 5 times slower than when instrumentation is performed during compilation. In fuzzing, a decrease in execution speed leads to a decrease in the number of test attempts per unit of time, thus reducing the opportunities to detect vulnerabilities and defects. 【0013】 In view of these points, the present invention has been made, and the object of the present invention is to perform coverage-based fuzzing without reducing execution speed, even when the software under test is a binary file and cannot be compiled. 【0014】 To solve the above-mentioned problems, the binary fuzzing device according to the present invention is a binary fuzzing device that performs fuzzing tests on software to be tested, which is composed of binary files, and is characterized by comprising: an input generation unit that generates one or more fuzzes by changing a seed that indicates a test case; a test target execution unit that inputs the generated fuzzes into the software to be tested and causes it to execute; a system information collection unit that collects system information indicating the operating status of the execution environment on which the software to be tested is run when the software to be tested is executed; and a coverage estimation unit that estimates whether or not the coverage has increased using the collected system information. 【0015】 According to the present invention, coverage-based fuzzing can be performed without reducing execution speed, even when the software under test consists only of binary files and cannot be compiled. 【0016】This is a functional block diagram showing the configuration of a binary fuzzing device according to the first embodiment of the present invention. This is a diagram illustrating the processing of the input generation unit of the binary fuzzing device according to the first embodiment of the present invention. This is a sequence diagram showing the overall processing flow of the binary fuzzing device according to the first embodiment of the present invention. This is a flowchart showing the processing performed by the coverage estimation unit of the binary fuzzing device according to the first embodiment of the present invention when the first coverage estimation is performed. This is a flowchart showing the processing performed by the coverage estimation unit of the binary fuzzing device according to the first embodiment of the present invention when the second and subsequent nth coverage estimation processes are performed. This is a diagram showing an image of the coverage estimation process according to this embodiment. This is a functional block diagram showing the configuration of a binary fuzzing device according to the second embodiment of the present invention. This is a sequence diagram showing the overall processing flow of a binary fuzzing device according to the second embodiment of the present invention. This is a flowchart showing the processing performed by the coverage estimation unit of the binary fuzzing device according to the second embodiment of the present invention when the first coverage estimation is performed. This is a flowchart showing the processing performed by the coverage estimation unit of the binary fuzzing device according to the second embodiment of the present invention when the second and subsequent nth coverage estimation processes are performed. This is a functional block diagram showing the configuration of a binary fuzzing device according to the third embodiment of the present invention. This is a flowchart showing the processing performed by the coverage estimation unit of the binary fuzzing device according to the third embodiment of the present invention during the execution of the second and subsequent nth coverage estimation processes. This is a functional block diagram showing the configuration of the binary fuzzing device according to the fourth embodiment of the present invention. This is a sequence diagram showing the overall processing flow of the binary fuzzing device according to the fourth embodiment of the present invention. This is a flowchart showing the execution of the seed before modification processing performed by the coverage estimation unit of the binary fuzzing device according to the fourth embodiment of the present invention. This is a flowchart showing the execution of the seed (fuzz) after mutation processing performed by the coverage estimation unit of the binary fuzzing device according to the fourth embodiment of the present invention. This is a functional block diagram showing the configuration of the binary fuzzing device according to the fifth embodiment of the present invention. This is a sequence diagram showing the overall processing flow of the binary fuzzing device according to the fifth embodiment of the present invention.This is a flowchart of the execution of the seed before modification by the coverage estimation unit of the binary fuzzing device according to the fifth embodiment of the present invention. This is a flowchart of the execution of the seed (fuzz) after mutation by the coverage estimation unit of the binary fuzzing device according to the fifth embodiment of the present invention. This is a functional block diagram showing the configuration of the binary fuzzing device according to the sixth embodiment of the present invention. This is a functional block diagram showing the configuration of the binary fuzzing device according to the seventh embodiment of the present invention. This is a sequence diagram showing the overall processing flow of the binary fuzzing device according to the seventh embodiment of the present invention. This is a flowchart of the execution of the seed before modification by the coverage estimation unit of the binary fuzzing device according to the seventh embodiment of the present invention after a predetermined number of executions. This is a flowchart of the execution of the seed (fuzz) after mutation by the coverage estimation unit of the binary fuzzing device according to the seventh embodiment of the present invention. This is a hardware configuration diagram showing an example of a computer that realizes the functions of the binary fuzzing device according to this embodiment. This is an explanatory diagram of testing by fuzzing. This is a diagram for explaining the overview of conventional fuzzing technology. This is a diagram for explaining the fuzz generation and coverage determination processes in conventional fuzzing technology. 【0017】 Next, embodiments for carrying out the present invention (hereinafter referred to as "this embodiment") will be described. The binary fuzzing device according to this embodiment performs coverage-based fuzzing without reducing execution speed, even when the software under test cannot be compiled using only binary files, that is, when instrumentation cannot be performed at the time of compilation as in the prior art, without performing dynamic instrumentation by an emulator. Each embodiment will be described in detail below. 【0018】 <First Embodiment> Figure 1 is a functional block diagram showing the configuration of a binary fuzzing device 1 according to the first embodiment of the present invention. 【0019】The binary fuzzing device 1 according to the first embodiment generates fuzz, which is unexpected data, and inputs it into the software under test 800 for execution. At the same time, it collects system information from the execution environment 700 (OS / kernel / CPU) while the software under test 800 is running. The binary fuzzing device 1 then estimates whether or not coverage has increased from the collected system information. In this way, the binary fuzzing device 1 can reduce overhead and prevent a decrease in execution speed compared to when dynamic instrumentation is performed using an emulator. 【0020】 The binary fuzzing device 1 is implemented by a computer consisting of a control unit 10, a storage unit (not shown), and an input / output unit (not shown). As shown in Figure 1, the binary fuzzing device 1 stores the software 800 to be tested and uses the CPU and the execution environment 700, such as the OS and kernel on the CPU, to perform fuzzing tests (fuzz tests) according to the instructions of the control unit 10 (fuzzing tool). Note that the OS / kernel is not an essential component of the execution environment 700. 【0021】 As shown in Figure 1, the control unit 10 (fuzzing tool) comprises an input generation unit 11, a test target execution unit 12, a system information collection unit 13, and a coverage estimation unit 14. 【0022】 The input generation unit 11 selects one seed from a seed queue containing test cases (seeds) and generates input information for the software under test. Specifically, the input generation unit 11 modifies the selected seed to generate one or more fuzzes. The functions of this input generation unit 11 can be the same as those of the AFL described in Non-Patent Documents 1 and 2. In terms of detailed processing, as shown in Figure 2, the input generation unit 11 performs seed selection processing, modification processing, and processing to store the modified seed, which is determined to be a new coverage, in the seed queue. 【0023】The input generation unit 11, at the start of the test, selects seeds from those stored in the seed queue as initial test cases, for example, using a round-robin method (11a: seed selection). After the test has started, it selects seeds, including test cases stored as new seeds. The input generation unit 11 may also set up logic to determine the priority of seed selection in this seed selection process, such as giving higher priority to seeds that have not yet been selected than seeds that have already been selected. 【0024】 In the mutation process (11b), the input generation unit 11 may perform a process to reduce the size of the input (trimming) before the mutation process in order to shorten the execution time. The input generation unit 11 removes the contiguous region of the seed (the part where the same process is repeated), and if the hash value of the execution coverage is the same, it determines that the execution result is the same and uses the trimmed version. Furthermore, mutation processes include deterministic processes and havoc processes. Deterministic mutation processes include bit inversion, addition or subtraction of values, and replacement with a specific value such as the maximum value. Havoc mutation processes include randomly adding or deleting multiple times, and splicing which combines two test cases. The input generation unit 11 performs this mutation process automatically or semi-automatically. 【0025】 Furthermore, the input generation unit 11 checks whether the seeds after mutation processing have been determined to have new coverage (increased coverage) by the coverage estimation unit 14, which will be described later. If a seed has been determined to have new coverage, it performs a process (11c) to store it in the seed queue. 【0026】 Returning to Figure 1, the test target execution unit 12 inputs the test case (seed) into the test target software 800 and executes it. The test target execution unit 12 also inputs the seed (fuzz) after mutation processing into the test target software 800 and executes it. In detail, the test target execution unit 12 performs the following processing. 【0027】The test target execution unit 12 notifies the system information collection unit 13, etc., of the start of the test. At this time, the test target execution unit 12 may also notify the system information collection unit 13 of the start of the test (test start notification) along with the identifier of the test target (process ID). After the system information collection unit 13 starts collecting system information, the test target execution unit 12 inputs the input (fuzz) received from the input generation unit 11 to the test target software 800 and instructs it to execute. The test target execution unit 12 also confirms the completion of the execution of the test target software 800 and notifies the system information collection unit 13, etc. (execution completion notification). 【0028】 The system information collection unit 13 collects system information from the execution environment 700 (OS / kernel / CPU) from the test start notification to the execution end notification from the test target execution unit 12. If the test start notification is received from the test target execution unit 12, including the identifier of the test target (process ID), the system information collection unit 13 focuses on collecting system information for that test target identifier (process ID). The system information collection unit 13 outputs the collected system information to the coverage estimation unit 14. 【0029】 The system information collected by the system information collection unit 13 is information indicating the operating status of the execution environment 700 (OS / kernel / CPU), such as the number of CPU cycles, the number of CPU instruction executions, the number of cache lookups, the number of cache misses, the number of CPU branch instruction executions, the number of CPU branch instruction execution failures, the number of bus cycles, the number of clocks during task execution, the number of page faults (total, minor, major), context switches, the number of CPU migrations, the number of alignment faults, the number of context switches between cgroups, the number of system call invocations, the number of specific system call invocations, the number of specific CPU instruction executions, the number of specific kernel function invocations, storage I / O, network I / O, memory access, memory usage, and the addresses of accessed memory and storage. 【0030】System information that is particularly highly correlated with coverage (strong correlation with increased coverage) includes CPU cycle count, CPU instruction execution count, CPU branch instruction execution count, specific CPU instruction execution count, or combinations thereof. Mechanisms for collecting this system information include, for example, performance analysis tools such as perf or eBPF (extended Berkley Packet Filter). 【0031】 The coverage estimation unit 14 uses the system information obtained from the system information collection unit 13 to estimate whether there has been an increase in coverage, that is, whether it is new coverage (whether a new execution path has been observed). 【0032】 The coverage estimation unit 14 uses, for example, one of the following algorithms to determine whether the tested fuzz is on the same execution path as an already observed execution path: - Distance-based algorithms in anomaly detection methods: k-NN (k-Nearest Neighbor), LOF (Local Outlier Factor), etc. - Probability-based algorithms in anomaly detection methods: Hotelling's T2 method, GMM (Gaussian Mixture Model), KDE (Kernel Density Estimation), etc. - Clustering-based algorithms in anomaly detection methods: OC-SVM, etc. - Estimation using deep learning Then, using one of the above pre-set algorithms, the coverage estimation unit 14 determines that there is new coverage, that is, that the coverage has increased, and outputs information to that effect to the input generation unit 11. 【0033】 [Processing of the First Embodiment] Next, the processing flow executed by the binary fuzzing device 1 according to the first embodiment will be described. Figure 3 is a sequence diagram showing the overall processing flow of the binary fuzzing device 1 according to the first embodiment of the present invention. 【0034】As shown in Figure 3, first, the input generation unit 11 of the binary fuzzing device 1 generates fuzz to be newly input to the software under test 800 from the seed queue where test cases (seeds) are stored (step S101). Specifically, the input generation unit 11 selects a test case (seed) stored in the seed queue and generates fuzz to be input data by performing a mutation process on the selected seed. Then, the input generation unit 11 outputs the generated fuzz to the execution unit 12 under test (step S102). 【0035】 When the test target execution unit 12 receives the generated fuzz, it outputs a test start notification to the system information collection unit 13 (step S103). In addition to this test start notification, the test target execution unit 12 may also include the identifier of the test target (process ID) in the notification. 【0036】 Next, when the system information collection unit 13 receives a test start notification, it begins collecting system information from the execution environment 700 (OS / kernel / CPU) (step S104). 【0037】 Next, the test target execution unit 12 inputs the generated fuzz to the test target software 800 and executes it (step S105: test execution instruction). Then, when the test target execution unit 12 confirms that the execution of the test target software 800 has finished (step S106), it outputs an execution completion notification to the system information collection unit 13 (step S107). When the system information collection unit 13 receives the execution completion notification, it terminates the collection of system information (step S108). Then, the system information collection unit 13 outputs the collected system information to the coverage estimation unit 14 (step S109). 【0038】 Then, the coverage estimation unit 14 uses the system information acquired from the system information collection unit 13 to estimate whether or not the coverage has increased, that is, whether it is new coverage (whether a new execution path has been observed) (Step S110: Coverage estimation process). 【0039】Subsequently, the coverage estimation unit 14 outputs the estimation result of the presence or absence of coverage increase to the input generation unit 11 (step S111). The input generation unit 11 stores the mutated seed determined to be a new coverage in the seed queue (step S112). Then, it returns to step 101, selects the next seed, and continues the process. 【0040】 (Coverage Estimation Process) Next, a detailed explanation will be given of the coverage estimation process executed by the coverage estimation unit 14 to estimate the presence or absence of coverage increase. This process is the process executed in step S110 of FIG. 3. 【0041】 FIG. 4 is a flowchart showing the flow of the coverage estimation process executed by the coverage estimation unit 14 of the binary fuzzing device 1 according to the first embodiment to estimate the presence or absence of coverage increase. FIG. 4A is a flowchart showing the process when the collection of system information for each execution path has not yet been performed, that is, the process at the time of the first execution of coverage estimation. FIG. 4B is a flowchart showing the process at the time of execution of the second to nth coverage estimation processes after the collection of system information for each execution path has been performed. 【0042】 The process at the time of the first execution of coverage estimation will be described with reference to FIG. 4A. First, the coverage estimation unit 14 of the binary fuzzing device 1 receives the system information acquired by the system information collection unit 13 at the first execution of the fuzz input to the test target software 800 (step S121). 【0043】 Next, the coverage estimation unit 14 stores the system information obtained by the execution of the first input test target software 800 as the system information of the execution path "1" (step S122). 【0044】 Subsequently, the coverage estimation unit 14 notifies the input generation unit 11 of "new coverage present (coverage increase)" (step S123). Then, the process ends. 【0045】Next, the processing during the second and subsequent nth coverage estimation executions will be explained with reference to Figure 4B. First, the coverage estimation unit 14 receives system information acquired by the system information collection unit 13 during the nth execution of the fuzz input to the software under test 800 (hereinafter referred to as "input n execution" in each figure) (step S131). 【0046】 Next, the coverage estimation unit 14 repeats the following process for each execution path (i = 1, 2, ...) (steps S132 to S135). Here, assuming there are m execution paths observed at the present time, the process in steps S133 to S134 is performed for all m execution paths (i = 1; i <m+1 ; i++;)。 【0047】 In step S133, the coverage estimation unit 14 determines whether the system information for the nth execution is the same as one of the execution paths i. As described above, the determination of whether they are the same execution paths is made using distance-based algorithms (k-NN, LOF, etc.) or probability-based algorithms (Hotelling's T2 method, GMM, KDE, etc.) used in anomaly detection methods. If the system information for the nth execution is the same as execution path i (the selected i-th execution path) (step S133 → Yes), the process proceeds to step S134. 【0048】 In step S134, the coverage estimation unit 14 adds the system information for the nth execution to the group of execution paths i. Then, if the system information for the nth execution is the same as one of the current m execution paths, the coverage estimation unit 14 notifies the input generation unit 11 that there is "no new coverage," that is, "no increase in coverage" (step S136), and ends the process. 【0049】 On the other hand, in step S133, if the system information for the nth execution is not the same as any of the execution paths i (step S133 → No), the coverage estimation unit 14 uses the system information for the nth execution as the system information for execution path m+1 (step S137). In other words, it uses the system information for a new execution path. 【0050】 Then, the coverage estimation unit 14 notifies the input generation unit 11 that "new coverage exists," that is, "coverage has increased" (step S138), and ends the process. 【0051】 Figure 5 is an illustrative diagram of the coverage estimation process shown in Figures 4A and 4B. The coverage estimation unit 14 first receives the system information for the first execution (reference numeral 141) and stores it as execution path "1". Subsequently, when the coverage estimation unit 14 receives the system information for the second execution, it determines whether the execution path obtained from the system information for the second input is the same as the execution path "1" that has already been recorded (reference numeral 142). 【0052】 Then, if the execution path obtained from the second input system information is the same as execution path "1", the coverage estimation unit 14 adds the system information from the second input execution to the execution path "1" group (reference numeral 143). On the other hand, if the execution path obtained from the second input system information is not the same as execution path "1", the coverage estimation unit 14 creates a new execution path "2" group and stores the system information from the second input execution (reference numeral 144). 【0053】 In this way, the coverage estimation unit 14 determines whether coverage has increased by determining whether the system information for the nth execution is the same as the m execution paths observed up to that point. 【0054】 <Second Embodiment> Next, a binary fuzzing device 1A according to the second embodiment will be described. In the binary fuzzing device 1 according to the first embodiment, coverage estimation was performed by collecting system information from the execution environment 700 when the fuzz was executed on the software under test 800. In the binary fuzzing device 1A according to the second embodiment, in addition to system information, response signals from the software under test 800 are collected, and coverage estimation is performed including these response signals. In this way, the estimation accuracy can be improved by adding response signals from the software under test 800 as an information source for the coverage estimation process. 【0055】 Figure 6 is a functional block diagram showing the configuration of a binary fuzzing device 1A according to a second embodiment of the present invention. Components having the same functions as the binary fuzzing device 1 of the first embodiment shown in Figure 1 are given the same names and reference numerals and their descriptions are omitted. 【0056】 The control unit 10 (fuzzing tool) of the binary fuzzing device 1A comprises an input generation unit 11, a test target execution unit 12, a system information collection unit 13, a coverage estimation unit 14A, and a response signal collection unit 15. The binary fuzzing device 1 shown in Figure 1 differs from this device in that it has a coverage estimation unit 14A and a response signal collection unit 15. 【0057】 The response signal acquisition unit 15 collects response signals from the software under test 800 and outputs them to the coverage estimation unit 14A. Specifically, the response signal acquisition unit 15 receives a test start notification from the execution unit 12 under test and starts collecting response signals from the software under test 800. Then, the response signal acquisition unit 15 receives an execution end notification from the execution unit 12 under test and terminates the collection of response signals. The response signal acquisition unit 15 collects response signals, for example, by performing a probe to the kernel network stack using eBPF. 【0058】 The coverage estimation unit 14A uses the system information acquired from the system information acquisition unit 13 and the response signals acquired by the response signal acquisition unit 15 to estimate whether there has been an increase in coverage, that is, whether it is new coverage (whether a new execution path has been observed). Specifically, the coverage estimation unit 14A determines whether the system information and response signals during execution of the test software 800 for the input fuzz are the same execution path as previously observed. The coverage estimation unit 14A then estimates whether there has been an increase in coverage, that is, whether it is new coverage (whether a new execution path has been observed), and outputs the estimation result to the input generation unit 11. 【0059】[Processing of the Second Embodiment] Next, the processing flow executed by the binary fuzzing device 1A according to the second embodiment will be described. Figure 7 is a sequence diagram showing the overall processing flow of the binary fuzzing device 1A according to the second embodiment of the present invention. Note that the same processing as the binary fuzzing device 1 according to the first embodiment shown in Figure 3 is given the same step number and its explanation is omitted. 【0060】 When the test target execution unit 12 receives the generated fuzz from the input generation unit 11, it outputs a test start notification to the system information collection unit 13 (step S103) and also outputs a test start notification to the response signal collection unit 15 (step S201). In addition to this test start notification, the test target execution unit 12 may also include the identifier of the test target (process ID) in the notification. 【0061】 Next, upon receiving a test start notification, the response signal acquisition unit 15 begins acquiring response signals from the software under test 800 (step S202). Upon receiving an execution end notification (step S203), the response signal acquisition unit 15 terminates the acquisition of response signals from the software under test 800 (step S204). Finally, the response signal acquisition unit 15 outputs the acquired response signals to the coverage estimation unit 14A (step S205). 【0062】 Then, the coverage estimation unit 14A uses the system information acquired from the system information acquisition unit 13 and the response signals collected by the response signal acquisition unit 15 to estimate whether there has been an increase in coverage, that is, whether it is new coverage (whether a new execution path has been observed) (step S206: coverage estimation process). Thereafter, the same processing as in steps S111 and S112 in Figure 3 is performed. 【0063】 (Coverage Estimation Process) Next, the coverage estimation process performed by the coverage estimation unit 14A to estimate whether or not coverage has increased will be explained in detail. This process is performed in step S206 in Figure 7. 【0064】Figure 8 is a flowchart showing the flow of the coverage estimation process performed by the coverage estimation unit 14A of the binary fuzzing device 1A according to the second embodiment, which estimates whether or not coverage has increased. Figure 8A is a flowchart showing the process when system information and response signals for each execution path have not yet been collected, that is, when the first coverage estimation is performed. Figure 8B is a flowchart showing the process when system information and response signals for each execution path have been collected, that is, when the second and subsequent nth coverage estimation processes are performed. 【0065】 The process during the initial coverage estimation will be explained with reference to Figure 8A. First, the coverage estimation unit 14A of the binary fuzzing device 1A receives the system information acquired by the system information acquisition unit 13 and the response signal acquired by the response signal acquisition unit 15 during the first execution of fuzz input to the software under test 800 (step S221). 【0066】 Next, the coverage estimation unit 14A stores the system information and response signals obtained from the execution of the first input test target software 800 as system information for execution path "1" (step S222). 【0067】 Next, the coverage estimation unit 14A notifies the input generation unit 11 that "new coverage is available (coverage has increased)" (step S223). Then, the process ends. 【0068】 Next, the processing during the second and subsequent nth coverage estimation executions will be explained with reference to Figure 8B. First, the coverage estimation unit 14A receives the system information acquired by the system information acquisition unit 13 and the response signals collected by the response signal acquisition unit 15 when the nth fuzz input is executed to the software 800 under test (input n execution time) (step S231). 【0069】 Next, the coverage estimation unit 14A repeats the following process for each execution path (i = 1, 2, ...) (steps S232 to S235). Here, assuming there are m execution paths observed at the present time, the process in steps S233 to S234 is performed for all m execution paths (i = 1; i <m+1 ; i++;)。 【0070】 In step S233, the coverage estimation unit 14A determines whether the system information and response signal from the nth execution input are the same as one of the execution paths i. The determination of whether they are the same execution path is made using distance-based algorithms (k-NN, LOF, etc.) or probability-based algorithms (Hotelling's T2 method, GMM, KDE, etc.) used in anomaly detection methods, as described above. If the system information and response signal from the nth execution input are the same as execution path i (the selected i-th execution path) (step S233 → Yes), the process proceeds to step S234. 【0071】 In step S234, the coverage estimation unit 14A adds the system information and response signal of the nth execution input to the group of execution paths i. Then, if the system information and response signal of the nth execution input is the same as one of the current m execution paths, the coverage estimation unit 14 notifies the input generation unit 11 that there is "no new coverage," that is, "no increase in coverage" (step S236), and ends the process. 【0072】 On the other hand, in step S233, if the system information and response signal for the nth execution input are not the same as any of the execution paths i (step S233 → No), the coverage estimation unit 14A uses the system information and response signal for the nth execution input as the system information and response signal for execution path m+1 (step S237). In other words, it uses the system information and response signal for a new execution path. 【0073】 Then, the coverage estimation unit 14A notifies the input generation unit 11 that "new coverage exists," that is, "coverage has increased" (step S238), and ends the process. 【0074】 By doing so, the binary fuzzing device 1A according to the second embodiment can improve the estimation accuracy by adding the response signal from the software under test 800 to the system information obtained from the execution environment 700 as an information source for the coverage estimation process. 【0075】 <Third Embodiment> Next, a binary fuzzing device 1B according to the third embodiment will be described. In the binary fuzzing device 1 according to the first embodiment and the binary fuzzing device 1A according to the second embodiment, when the coverage estimation units 14 and 14A perform coverage estimation, they perform a determination process in one step to determine whether the execution path is the same as an existing execution path. In contrast, the binary fuzzing device 1B according to the third embodiment is characterized by performing coverage estimation in multiple stages. In this way, the binary fuzzing device 1B can reduce the time required for coverage estimation processing and improve execution speed by performing coverage estimation in multiple stages. 【0076】 Figure 9 is a functional block diagram showing the configuration of a binary fuzzing device 1B according to the third embodiment of the present invention. Components having the same functions as the binary fuzzing device 1 of the first embodiment shown in Figure 1 and the binary fuzzing device 1A of the second embodiment shown in Figure 6 are given the same names and reference numerals and their descriptions are omitted. 【0077】 The control unit 10 (fuzzing tool) of the binary fuzzing device 1B includes an input generation unit 11, a test target execution unit 12, a system information collection unit 13, and a coverage estimation unit 14B. The control unit 10 may also include a response signal collection unit 15. The binary fuzzing device 1, 1A shown in Figures 1 and 6 differs in that it has a coverage estimation unit 14B. 【0078】 The coverage estimation unit 14B does not perform a single, one-step determination of whether the execution path is the same as an existing one using system information and response signals. Instead, it performs determination processing for each piece of system information (e.g., number of CPU cycles, number of CPU instructions executed, etc.) and for each response signal. In other words, the coverage estimation unit 14B performs coverage estimation processing in multiple stages. In the following description, the control unit 10 is equipped with a response signal collection unit 15, and the determination of whether the execution path is the same as an existing one is performed in two stages: using system signals and response signals. 【0079】(Coverage Estimation Process of the Third Embodiment) The coverage estimation process performed by the coverage estimation unit 14B of the binary fuzzing device 1B according to the third embodiment will be described in detail. Note that the processing performed by the coverage estimation unit 14B of the binary fuzzing device 1B according to the third embodiment when the collection of system information for each execution path has not yet been performed, that is, the processing at the time of the first coverage estimation, is the same as in Figure 8A, so the explanation will be omitted. 【0080】 Figure 10 is a flowchart showing the processing after the collection of system information and response signals for each execution path, that is, the processing during the execution of the second and subsequent nth coverage estimation processes. Here, we will explain an example in which the coverage estimation unit 14B determines the system information and response signals in two stages. Note that the same processing as in Figure 8B is given the same step number and its explanation is omitted. 【0081】 The coverage estimation unit 14B executes the process in step S231. Next, the coverage estimation unit 14B repeats the following process for each execution path (i = 1, 2, ...) (steps S232 to S235). Here, assuming there are m execution paths observed at the present time, the process in steps S301 to S234 is performed for all m execution paths (i = 1; i <m+1 ; i++;)。 【0082】 In step S301, the coverage estimation unit 14B determines whether the response signal for the nth execution input is the same as any of the execution paths i. If the response signal for the nth execution input is not the same as any of the execution paths i (step S301 → No), the coverage estimation unit 14B determines that there is an increase in coverage and proceeds with the process (steps S237, S238). On the other hand, if the response signal for the nth execution input is the same as any of the execution paths i (step S301 → Yes), the process proceeds to step S302. 【0083】In step S302, the coverage estimation unit 14B determines whether the system information for the nth execution is the same as any of the execution paths i. If the system information for the nth execution is not the same as any of the execution paths i (step S302 → No), the coverage estimation unit 14B determines that there is an increase in coverage and proceeds with processing (steps S237, S238). On the other hand, if the system information for the nth execution is the same as any of the execution paths i (step S302 → Yes), the process proceeds to steps S234 to S236. 【0084】 In this way, the coverage estimation unit 14B performs a multi-stage determination of whether the execution path is the same as an existing execution path during the coverage estimation process. For example, if it is determined in the first stage that the execution path is not the same as an existing execution path, there is no need to perform the determination process from the second stage onward. By performing the determination of whether the execution path is the same in multiple stages in this way, it is possible to eliminate the need to perform the subsequent processing if it is determined in the previous stage that the execution path is not the same. Therefore, according to the binary fuzzing device 1B of the third embodiment, the time required for the coverage estimation process can be reduced and the execution speed can be improved. 【0085】<Fourth Embodiment> Next, the binary fuzzing device 1C according to the fourth embodiment will be described. The binary fuzzing device 1 according to the first embodiment holds all execution path information (information on the execution path from execution path "1" to execution path "m" observed at that time) when the fuzz is executed on the software under test 800, and compares whether all the execution path information is the same as the execution path obtained by inputting the fuzz into the software under test 800. In contrast, the binary fuzzing device 1C according to the fourth embodiment estimates whether there has been an increase in coverage by comparing the system information obtained when the fuzz is executed on the software under test 800 with some of the system information obtained by executing the software under test 800 immediately before the execution of the fuzz. Specifically, the binary fuzzing device 1C is characterized by comparing the system information at the time of input execution before the variation of the seed (test case) selected for input into the software under test 800 with the system information obtained when the fuzz with the variation of that seed is input, and determining whether it is new coverage, that is, whether there has been an increase in coverage. By doing so, the binary fuzzing device 1C can reduce the number of comparison targets and quickly determine whether or not coverage has increased. 【0086】 Figure 11 is a functional block diagram showing the configuration of a binary fuzzing device 1C according to a fourth embodiment of the present invention. Components having the same functions as the binary fuzzing device 1 of the first embodiment shown in Figure 1 are given the same names and reference numerals and their descriptions are omitted. 【0087】 The control unit 10 (fuzzing tool) of the binary fuzzing device 1C comprises an input generation unit 11C, a test target execution unit 12, a system information collection unit 13, and a coverage estimation unit 14C. This differs from the binary fuzzing device 1 shown in Figure 1 in that it includes an input generation unit 11C and a coverage estimation unit 14C. 【0088】When the input generation unit 11C selects a seed (test case) stored in the seed queue for fuzz generation, it first outputs it to the test target execution unit 12 without performing mutation processing. As a result, the test target execution unit 12 inputs the seed before mutation into the test target software 800, and the system information collection unit 13 collects the system information before mutation and outputs it to the coverage estimation unit 14C. 【0089】 Next, the input generation unit 11C performs mutation processing on the selected seed to generate fuzz. Then, the input generation unit 11C outputs the generated fuzz to the test target execution unit 12. As a result, the test target execution unit 12 inputs the mutated seed, which is the fuzz, into the test target software 800, and the system information collection unit 13 collects system information when the fuzz is executed and outputs it to the coverage estimation unit 14C. 【0090】 The coverage estimation unit 14C determines whether the execution paths are the same by comparing the system information obtained from the input of the seed before mutation with the system information obtained from the input of the mutated seed, Fuzz. If the coverage estimation unit 14C determines that the execution paths are the same, it outputs "no new coverage" (no increase in coverage) to the input generation unit 11C. If the coverage estimation unit 14C determines that the execution paths are not the same, it outputs "new coverage" (increase in coverage) to the input generation unit 11C. 【0091】 [Processing of the Fourth Embodiment] Next, the processing flow executed by the binary fuzzing device 1C according to the fourth embodiment will be described. Figure 12 is a sequence diagram showing the overall processing flow of the binary fuzzing device 1C according to the fourth embodiment of the present invention. Note that the same processing as the binary fuzzing device 1 according to the first embodiment shown in Figure 3 is given the same step number and its explanation is omitted. 【0092】As shown in Figure 12, the input generation unit 11C selects one seed (test case) from among the seeds (test cases) stored in the seed queue (step S401). Then, the input generation unit 11C outputs the selected seed (seed before mutation) to the test target execution unit 12 (step S402). 【0093】 From that point onward, the processes from steps S103 to S111 are performed on the "pre-mutation seed" instead of the "fuzz," and the coverage estimation unit 14C acquires and stores the system information obtained from the system information collection unit 13 based on the input of the pre-mutation seed. In the coverage estimation process of step S110, the coverage estimation unit 14C estimates that there is no new coverage (no increase in coverage) because no mutation processing has been performed on the seed, and outputs this estimation result to the input generation unit 11 (step S111). 【0094】 Next, the input generation unit 11C performs mutation processing on the seed selected in step S401 to generate fuzz (step S101). After that, it performs the processes from steps S102 to S109, and the coverage estimation unit 14C acquires system information obtained from the system information collection unit 13 using the input of the fuzz. 【0095】 Then, the coverage estimation unit 14C compares the system information obtained from the input of the seed before mutation with the system information obtained from the input of the mutated seed, Fuzz, to determine whether or not they are the same execution path (Step S403: Coverage estimation process). 【0096】The coverage estimation unit 14C then outputs the estimation result of whether or not there is an increase in coverage to the input generation unit 11 (step S404). Specifically, if the coverage estimation unit 14C determines that it is the same execution path, it outputs an estimation result of no new coverage (no increase in coverage) to the input generation unit 11C. Also, if the coverage estimation unit 14C determines that it is not the same execution path, it outputs an estimation result of new coverage (increase in coverage) to the input generation unit 11C. After that, the input generation unit 11C stores the seed after mutation processing that has been determined to be new coverage in the seed queue (step S112), returns to step 401, selects the next seed, and continues processing. 【0097】 (Coverage Estimation Process) Next, the coverage estimation process performed by the coverage estimation unit 14C to estimate whether or not coverage has increased will be described in detail. This process is performed in steps S110 and S403 of Figure 12. 【0098】 Figure 13 is a flowchart showing the flow of the coverage estimation process performed by the coverage estimation unit 14C of the binary fuzzing device 1C according to the fourth embodiment, which estimates whether or not coverage has increased. Figure 13A is a flowchart showing the execution of the seed when seed is selected, that is, when the seed is executed before the modification process. Figure 13B is a flowchart showing the execution of the seed (fuzz) after the mutation process. 【0099】 The coverage estimation process during seed selection will be explained with reference to Figure 13A. First, the coverage estimation unit 14C of the binary fuzzing device 1C receives system information acquired by the system information collection unit 13 when the seed is executed before mutation into the software under test 800 (step S411). 【0100】 Next, the coverage estimation unit 14C stores the obtained system information as the system information of the seed before mutation (step S412). 【0101】Next, the coverage estimation unit 14C estimates that there is no new coverage (no increase in coverage) because no mutation processing has been performed on the seed, and outputs this estimation result to the input generation unit 11 (step S413). 【0102】 Next, the coverage estimation process during the execution of the seed (fuzz) after mutation processing will be explained with reference to Figure 13B. First, the coverage estimation unit 14C receives system information collected during the execution of the seed (fuzz) after mutation processing from the system information collection unit 13 (step S421). 【0103】 Next, the coverage estimation unit 14C determines whether the system information of the seed (fuzz) after mutation processing is the same as the system information of the seed before mutation (step S422). As described above, the determination of whether the execution paths are the same is made using distance-based algorithms (k-NN, LOF, etc.) or probability-based algorithms (Hotelling's T2 method, GMM, KDE, etc.) used in anomaly detection methods. 【0104】 Here, if the system information during the execution of the fuzz follows the same execution path as the system information of the seed before mutation (step S422 → Yes), the coverage estimation unit 14C notifies the input generation unit 11C that there is "no new coverage," that is, "no increase in coverage" (step S423), and ends the process. 【0105】 On the other hand, if the system information during the execution of the fuzz is not the same execution path as the system information of the seed before mutation (step S422 → No), the coverage estimation unit 14C notifies the input generation unit 11 that "new coverage exists," that is, "coverage has increased" (step S424), and ends the process. 【0106】 In this way, the binary fuzzing device 1C can compare the system information at the time of inputting the seed before mutation with the system information at the time of inputting the fuzz that has been mutated from that seed, and determine whether there is new coverage, that is, whether there is an increase in coverage. As a result, the binary fuzzing device 1C can reduce the number of comparison targets and can determine whether there is an increase in coverage at high speed. 【0107】 <Fifth Embodiment> Next, a binary fuzzing device 1D according to the fifth embodiment will be described. The binary fuzzing device 1D according to the fifth embodiment estimates whether or not coverage has increased by comparing system information obtained when the fuzz is executed on the software under test 800 with some of the system information obtained when the software under test 800 was executed immediately before the execution of the fuzz. Specifically, the binary fuzzing device 1D is characterized in that, in determining whether or not there is new coverage by the coverage estimation unit, it estimates whether or not coverage has increased by comparing system information obtained from the most recent N inputs to the software under test 800 with system information obtained when the fuzz is executed this time. In this way, the binary fuzzing device 1D can reduce the number of comparison targets compared to the binary fuzzing device 1 according to the first embodiment, and can determine whether or not coverage has increased at high speed. 【0108】 Figure 14 is a functional block diagram showing the configuration of a binary fuzzing device 1D according to a fifth embodiment of the present invention. Components having the same functions as the binary fuzzing device 1 of the first embodiment shown in Figure 1 are given the same names and reference numerals and their descriptions are omitted. 【0109】 The control unit 10 (fuzzing tool) of the binary fuzzing device 1D comprises an input generation unit 11D, a test target execution unit 12, a system information collection unit 13, and a coverage estimation unit 14D. This differs from the binary fuzzing device 1 shown in Figure 1 in that it includes an input generation unit 11D and a coverage estimation unit 14D. 【0110】 Similar to the input generation unit 11C of the binary fuzzing device 1C in Figure 11, the input generation unit 11D, upon selecting a seed (test case) stored in the seed queue for fuzz generation, first outputs it to the test target execution unit 12 without performing mutation processing. As a result, the test target execution unit 12 inputs the pre-mutation seed into the test target software 800, and the system information collection unit 13 collects the pre-mutation system information and outputs it to the coverage estimation unit 14D. 【0111】Next, the input generation unit 11D performs mutation processing on the selected seed to generate fuzz. Then, the input generation unit 11D outputs the generated fuzz to the test target execution unit 12. As a result, the test target execution unit 12 inputs the mutated seed, which is the fuzz, into the test target software 800, and the system information collection unit 13 collects system information when the fuzz is executed and outputs it to the coverage estimation unit 14D. 【0112】 The coverage estimation unit 14D compares the system information obtained from the most recent N inputs with the system information obtained by executing the fuzz input to the software 800 under test in the current process to determine whether the execution paths are the same. If the coverage estimation unit 14D determines that the execution paths are the same, it outputs "no new coverage" (no increase in coverage) to the input generation unit 11D. If the coverage estimation unit 14D determines that the execution paths are not the same, it outputs "new coverage" (increase in coverage) to the input generation unit 11D. 【0113】 [Processing of the Fifth Embodiment] Next, the processing flow executed by the binary fuzzing device 1D according to the fifth embodiment will be described. Figure 15 is a sequence diagram showing the overall processing flow of the binary fuzzing device 1D according to the fifth embodiment of the present invention. Note that the same processing as the binary fuzzing device 1 according to the first embodiment shown in Figure 3 is given the same step number and its explanation is omitted. 【0114】 As shown in Figure 15, the input generation unit 11D selects one seed (test case) from among the seeds (test cases) stored in the seed queue (step S501). Then, the input generation unit 11D outputs the selected seed (seed before mutation) to the test target execution unit 12 (step S502). 【0115】From that point onward, the processes from steps S103 to S111 are performed on the "pre-mutation seed" instead of the "fuzz," and the coverage estimation unit 14D acquires system information obtained from the system information collection unit 13 based on the input of the pre-mutation seed and stores it as one of the N items. In the coverage estimation process of step S110, the coverage estimation unit 14D estimates that there is no new coverage (no increase in coverage) because no mutation processing has been performed on the seed, and outputs this estimation result to the input generation unit 11 (step S111). 【0116】 Next, the input generation unit 11D performs mutation processing on the seed selected in step S501 to generate a fuzz (step S101). After that, the processing from steps S102 to S109 is performed, and the coverage estimation unit 14D acquires system information obtained from the system information collection unit 13 through the input of the fuzz. The input generation unit 11D generates one or more fuzzes, and the test target execution unit 12 inputs each of the generated fuzzes into the test target software 800 and executes them. The coverage estimation unit 14D then acquires system information from the system information collection unit 13 when each fuzz is executed. At this time, the coverage estimation unit 14D stores the system information acquired from the most recent N inputs, compares the system information acquired from the most recent N inputs with the system information obtained by executing the fuzz input into the test target software 800 in the current process, and determines whether or not they follow the same execution path (step S503: coverage estimation process). 【0117】The coverage estimation unit 14D then outputs the estimation result of whether or not there is an increase in coverage to the input generation unit 11 (step S504). Specifically, if the coverage estimation unit 14D determines that it is the same execution path, it outputs an estimation result of no new coverage (no increase in coverage) to the input generation unit 11D. Also, if the coverage estimation unit 14D determines that it is not the same execution path, it outputs an estimation result of new coverage (increase in coverage) to the input generation unit 11D. After that, the input generation unit 11D stores the seed after mutation processing that has been determined to be new coverage in the seed queue (step S112), returns to step 501, selects the next seed, and continues processing. 【0118】 (Coverage Estimation Process) Next, the coverage estimation process performed by the coverage estimation unit 14D to estimate whether or not coverage has increased will be explained in detail. This process is performed in steps S110 and S503 of Figure 15. 【0119】 Figure 16 is a flowchart showing the flow of the coverage estimation process performed by the coverage estimation unit 14D of the binary fuzzing device 1D according to the fifth embodiment, which estimates whether or not coverage has increased. Figure 16A is a flowchart showing the execution of the seed before the modification process, i.e., when the seed is selected. Figure 16B is a flowchart showing the execution of the seed (fuzz) after the mutation process. Here, it is explained that the coverage estimation unit 14D determines whether or not the execution path is the same using system information obtained from the most recent N=1 inputs. 【0120】 The coverage estimation process during seed selection will be explained with reference to Figure 16A. First, the coverage estimation unit 14D of the binary fuzzing device 1D receives system information acquired by the system information collection unit 13 when the seed is executed before mutation into the software under test 800 (step S511). 【0121】Next, the coverage estimation unit 14D stores the obtained system information (in this case, the information obtained as the most recent N=1 system information) as the system information from the previous input execution (the system information of the seed before mutation) (step S512). 【0122】 Next, the coverage estimation unit 14C estimates that there is no new coverage (no increase in coverage) because no mutation processing has been performed on the seed, and outputs this estimation result to the input generation unit 11 (step S513). 【0123】 Next, the coverage estimation process during the execution of the seed (fuzz) after mutation processing will be explained with reference to Figure 16B. First, the coverage estimation unit 14D receives system information collected during the execution of the fuzz input in the current process from the system information collection unit 13 (step S521). 【0124】 Next, the coverage estimation unit 14D determines whether the system information obtained in the current process (during fuzz execution) follows the same execution path as the most recent N (N=1) system information obtained from the previous input (step S522). 【0125】 If the system information for the current process follows the same execution path as the most recent (N=1) system information obtained during the previous input execution (step S522 → Yes), then the process proceeds to step S523. 【0126】 In step S523, the coverage estimation unit 14D stores the system information obtained from the current process (the current fuzz process) as the system information (N=1) obtained during the input execution immediately preceding the current process. Then, the coverage estimation unit 14D notifies the input generation unit 11D that there is "no new coverage," that is, "no increase in coverage" (step S524), and ends the process. 【0127】 On the other hand, if the system information of the current process does not follow the same execution path as the system information obtained most recently (N=1), that is, during the previous input execution (step S522 → No), the coverage estimation unit 14D proceeds to step S525. 【0128】In step S525, the coverage estimation unit 14D stores the system information obtained from the current process (the current fuzz process) as the system information (N=1) obtained during the input execution immediately preceding the current process. Then, the coverage estimation unit 14D notifies the input generation unit 11 that "new coverage is available," that is, "coverage has increased" (step S526), ​​and ends the process. 【0129】 In the case of N=2, the coverage estimation unit 14D stores the system information obtained from the processes one and two steps prior to the current process. Then, following the determination process of the system information obtained in step S522, the coverage estimation unit 14D determines whether the system information obtained from the current process (the current fuzz process) and the system information obtained from the input two steps prior to the current process are the same execution path. Similarly, in the case of N=3 or later, the coverage estimation unit 14D stores the system information up to N steps prior to the current process and uses the system information obtained from the input N steps prior to the current process to determine whether the execution path is the same. 【0130】 In this way, the binary fuzzing device 1D's coverage estimation unit 14D compares the system information obtained from the most recent N inputs to the test target software 800 with the system information obtained during the current fuzz execution, and can determine whether it is new coverage, that is, whether there is an increase in coverage. As a result, the binary fuzzing device 1D can reduce the number of comparison targets compared to the binary fuzzing device 1 according to the first embodiment, and can determine whether there is an increase in coverage at high speed. 【0131】<Sixth Embodiment> Next, a binary fuzzing device 1E according to the sixth embodiment will be described. The binary fuzzing device 1E according to the sixth embodiment is characterized in that, when the system information collection unit collects system information from the execution environment 700 (OS / kernel / CPU) and the response signal collection unit collects response signals from the test target software 800, it processes the collected information to extract feature quantities and outputs them to the coverage estimation unit 14. In this way, the amount of information stored by the binary fuzzing device 1E is reduced. Furthermore, the binary fuzzing device 1E can improve the accuracy of the coverage estimation process and improve the processing speed. 【0132】 Figure 17 is a functional block diagram showing the configuration of a binary fuzzing device 1E according to the sixth embodiment of the present invention. Components having the same functions as the binary fuzzing devices 1 and 1A of the first and second embodiments shown in Figures 1 and 6 are given the same names and reference numerals and their descriptions are omitted. 【0133】 The control unit 10 (fuzzing tool) of the binary fuzzing device 1E comprises an input generation unit 11, a test target execution unit 12, a system information collection unit 13E, and a coverage estimation unit 14. The control unit 10 may also include a response signal collection unit 15E. In the following description, the control unit 10 will be described as including a response signal collection unit 15E. 【0134】 In addition to the functions of the system information collection unit 13 shown in Figure 1, the system information collection unit 13E has the function of processing the collected information (hereinafter referred to as "collected information"; here referred to as "system information") and performing a process (a predetermined feature extraction process) to extract features. The system information collection unit 13E then outputs the feature information extracted from the system information to the coverage estimation unit 14. 【0135】 In addition to the functions of the response signal acquisition unit 15 shown in Figure 6, the response signal acquisition unit 15E has the function of processing the acquired information (here, "response signal") to extract features (a predetermined feature extraction process). The response signal acquisition unit 15E then outputs the feature information extracted for the response signal to the coverage estimation unit 14. 【0136】 The system information collection unit 13E and the response signal collection unit 15E perform a process to extract predetermined features by processing the collected information (predetermined feature extraction process). For example, the following processing steps are performed on the collected information to extract features: (Processing example 1) The collected information is expressed as a "percentage". For example, the number of CPU instruction executions, cache reference counts, and cache miss counts are expressed as values ​​per CPU cycle. (Processing example 2) The collected information is "summed up". For example, the collected system information is expressed as a total value. (Processing example 3) The "average" of the collected information is taken. (Processing example 4) The "maximum / minimum value" is selected from the collected information. (Processing example 5) The "logarithm" of the collected information is taken. 【0137】 The system information collection unit 13E and the response signal collection unit 15E perform a process (a predetermined feature extraction process) to process the collected information and extract features, as described above. The coverage estimation unit 14 then estimates whether the coverage has increased using the extracted features of the system information. As a result, the binary fuzzing device 1E can reduce the amount of information to be stored, improve the accuracy of the coverage estimation process, and increase the processing speed. 【0138】 <Seventh Embodiment> Next, a binary fuzzing device 1F according to the seventh embodiment will be described. The binary fuzzing device 1F according to the seventh embodiment estimates whether coverage has increased by comparing system information obtained when the fuzz is executed on the software under test 800 with some of the system information obtained when the software under test 800 was executed immediately before the execution of the fuzz. Specifically, the binary fuzzing device 1F is characterized in that, as a method for estimating whether coverage has increased, it uses the measurement error rate between the system information obtained when the fuzz is executed on the software under test 800 and the system information in the input (seed) before the mutation processing of the fuzz as the basis for estimation. As a result, the binary fuzzing device 1F can perform coverage-based fuzzing without reducing the execution speed even when the software under test 800 is a binary file and cannot be compiled. 【0139】 Figure 18 is a functional block diagram showing the configuration of a binary fuzzing device 1F according to the seventh embodiment of the present invention. Components having the same functions as the binary fuzzing device 1C of the fourth embodiment shown in Figure 11 are given the same names and reference numerals and their descriptions are omitted. 【0140】 The control unit 10 (fuzzing tool) of the binary fuzzing device 1F comprises an input generation unit 11F, a test target execution unit 12, a system information collection unit 13, and a coverage estimation unit 14F. This differs from the binary fuzzing device 1 shown in Figure 11 in that it includes an input generation unit 11F and a coverage estimation unit 14F. 【0141】 The input generation unit 11F, in addition to the functions of the input generation unit 11C of the fourth embodiment, is characterized by repeating the process of inputting the pre-mutation seed to the software under test 800 a predetermined number of times. By causing the software under test 800 to execute the pre-mutation seed a predetermined number of times, the input generation unit 11F can obtain multiple pieces of system information from when the pre-mutation seed was executed, which can be acquired by the coverage estimation unit 14F. 【0142】 The coverage estimation unit 14F performs measurement error reduction processing, such as calculating an average value, on multiple system information obtained by having the pre-mutation seed run a predetermined number of times on the software 800 under test. The coverage estimation unit 14F then generates this average value as the system information that will serve as the basis for the pre-mutation seed. 【0143】 Next, the coverage estimation unit 14F calculates the measurement error rate between the system information that serves as the standard for the seed before mutation and the system information obtained by inputting the mutated seed, Fuzz. If the calculated measurement error rate is below a predetermined threshold, the coverage estimation unit 14F determines that it is the same execution path and there is no new coverage (no increase in coverage). On the other hand, if the calculated measurement error rate exceeds a predetermined threshold, the coverage estimation unit 14F determines that it is not the same execution path and there is new coverage (increase in coverage). 【0144】[Processing of the Seventh Embodiment] Next, the processing flow executed by the binary fuzzing device 1F according to the seventh embodiment will be described. Figure 19 is a sequence diagram showing the overall processing flow of the binary fuzzing device 1F according to the seventh embodiment of the present invention. Note that the same processing as that of the binary fuzzing device 1C according to the fourth embodiment shown in Figure 12 is given the same step number and its explanation is omitted. 【0145】 As shown in Figure 19, the input generation unit 11F selects one seed (test case) from among the seeds (test cases) stored in the seed queue (step S701). Then, the input generation unit 11F outputs the selected seed (seed before mutation) to the test target execution unit 12 (step S702). 【0146】 From that point onward, the processes from steps S103 to S109 are performed on the "pre-mutation seed" instead of the "fuzz," and the coverage estimation unit 14F acquires and stores the system information obtained from the system information collection unit 13 based on the input of the pre-mutation seed. The binary fuzzing device 1F then repeats the processes from steps S703 to S109 for the selected seed (pre-mutation seed) a predetermined number of times. 【0147】 Next, the coverage estimation unit 14F generates reference system information for the seed before mutation by, for example, calculating the average value of the system information acquired a predetermined number of times (step S703) in order to reduce the measurement error of the system information of the seed before mutation. Note that the generation of this reference system information for the seed before mutation may be performed by the system information collection unit 13, and the result may be output to the coverage estimation unit 14F. Here, in the coverage estimation process of the reference system information, the coverage estimation unit 14F estimates that there is no new coverage (no increase in coverage) because no mutation processing has been performed on the seed, and outputs the estimation result to the input generation unit 11F (step S111). 【0148】Next, the input generation unit 11F performs mutation processing on the seed selected in step S701 to generate fuzz (step S101). After that, it performs the processes from steps S102 to S109, and the coverage estimation unit 14F acquires system information obtained from the system information collection unit 13 using the input fuzz. 【0149】 Then, the coverage estimation unit 14F calculates the measurement error rate between the system information that serves as the standard for the seed before mutation and the system information obtained by inputting the mutated seed, Fuzz, to determine whether or not they are the same execution path (Step S704: Coverage estimation process). 【0150】 The coverage estimation unit 14F then outputs the estimated result of whether or not there is an increase in coverage to the input generation unit 11 (step S705). Specifically, if the calculated measurement error rate is below a predetermined threshold, the coverage estimation unit 14F determines that it is the same execution path and there is no new coverage (no increase in coverage), and outputs this determination result to the input generation unit 11F. If the calculated measurement error rate exceeds a predetermined threshold, the coverage estimation unit 14F determines that it is not the same execution path and there is new coverage (increase in coverage), and outputs this determination result to the input generation unit 11F. After that, the input generation unit 11F stores the seed after mutation processing that has been determined to be new coverage in the seed queue (step S112), returns to step 701, selects the next seed, and continues processing. 【0151】 (Coverage Estimation Process) Next, the coverage estimation process performed by the coverage estimation unit 14F to estimate whether or not coverage has increased will be explained in detail. 【0152】 Figure 20 is a flowchart showing the flow of the coverage estimation process performed by the coverage estimation unit 14F of the binary fuzzing apparatus 1F according to the seventh embodiment, which estimates whether or not coverage has increased. Figure 20A is a flowchart when seed selection occurs, that is, when the seed before modification processing is executed a predetermined number of times. Figure 20B is a flowchart when the seed (fuzz) after mutation processing is executed. 【0153】The coverage estimation process during seed selection will be explained with reference to Figure 20A. First, in step S711, the coverage estimation unit 14F of the binary fuzzing device 1F receives system information acquired by the system information collection unit 13 when the seed before mutation is executed on the software 800 under test. The coverage estimation unit 14F receives system information for a predetermined number of times if the input generation unit 11F has repeated the process of inputting the seed before mutation into the software 800 under test a predetermined number of times. 【0154】 Next, the coverage estimation unit 14F performs measurement error reduction processing, such as calculating the average value, on multiple system information obtained by running the pre-mutation seed a predetermined number of times, and generates system information that will serve as the basis for the pre-mutation seed (step S712). 【0155】 The coverage estimation unit 14F then stores the generated system information as the seed's reference system information (step S713). 【0156】 Next, the coverage estimation unit 14F estimates that there is no new coverage (no increase in coverage) because no mutation processing has been performed on the seed, and outputs this estimation result to the input generation unit 11F (step S714). 【0157】 Next, the coverage estimation process during the execution of the seed (fuzz) after mutation processing will be explained with reference to Figure 20B. First, the coverage estimation unit 14F receives system information collected during the execution of the seed (fuzz) after mutation processing from the system information collection unit 13 (step S721). 【0158】 Next, the coverage estimation unit 14F calculates the measurement error rate using the system information that serves as the reference for the seed before mutation and the system information obtained by inputting the mutated seed, Fuzz (step S722). 【0159】Next, the coverage estimation unit 14F determines whether it is the same execution path by determining whether the calculated measurement error rate exceeds a predetermined threshold (step S723). If the calculated measurement error rate is below the predetermined threshold (step S723 → No), the coverage estimation unit 14F determines that it is the same execution path and there is no new coverage (no increase in coverage), and notifies the input generation unit 11F of the determination result (step S724), and ends the process. On the other hand, if the calculated measurement error rate exceeds the predetermined threshold (step S723 → Yes), the coverage estimation unit 14F determines that it is not the same execution path and there is new coverage (increase in coverage), and notifies the input generation unit 11F of the determination result (step S725), and ends the process. 【0160】 Thus, the binary fuzzing device 1F can use the measurement error rate between the input and system information before mutation processing as an estimation criterion for determining whether coverage has increased. As a result, the binary fuzzing device 1F can perform coverage-based fuzzing without reducing execution speed, even when the software 800 under test is only a binary file and cannot be compiled. 【0161】 <Hardware Configuration> The binary fuzzing devices 1, 1A, 1B, 1C, 1D, 1E, and 1F according to this embodiment are implemented by a computer 900 having a configuration such as that shown in Figure 21. Figure 21 is a hardware configuration diagram showing an example of a computer 900 that implements the functions of the binary fuzzing devices 1, 1A, 1B, 1C, 1D, 1E, and 1F according to this embodiment. The computer 900 has a CPU 901, a ROM (Read Only Memory) 902, a RAM 903, an HDD (Hard Disk Drive) 904, an input / output I / F (Interface) 905, a communication I / F 906, and a media I / F 907. 【0162】The CPU 901 operates based on a program (binary fuzzing program) stored in the ROM 902 or HDD 904, and controls the functional units of the binary fuzzing devices 1, 1A, 1B, 1C, 1D, 1E, and 1F. The ROM 902 stores the boot program executed by the CPU 901 when the computer 900 starts up, as well as programs related to the computer 900's hardware. 【0163】 The CPU 901 controls input devices 910, such as a mouse or keyboard, and output devices 911, such as a display or printer, via the input / output interface 905. The CPU 901 acquires data from the input devices 910 and outputs the generated data to the output devices 911 via the input / output interface 905. In addition to the CPU 901, a GPU (Graphics Processing Unit) or the like may also be used as a processor. 【0164】 The HDD 904 stores programs executed by the CPU 901 and data used by those programs. The communication I / F 906 receives data from other devices via a communication network (e.g., NW (Network) 920) and outputs it to the CPU 901, and also transmits data generated by the CPU 901 to other devices via the communication network. 【0165】 The media interface 907 reads a program (binary fuzzing program) or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 903. The CPU 901 loads the program related to the desired processing from the recording medium 912 onto the RAM 903 via the media interface 907 and executes the loaded program. The recording medium 912 can be an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto Optical Disk), a magnetic recording medium, or a semiconductor memory. 【0166】For example, when computer 900 functions as a binary fuzzing device 1, 1A, 1B, 1C, 1D, 1E, 1F of the present invention, the CPU 901 of computer 900 realizes the function of the binary fuzzing program by executing a program loaded onto RAM 903. The HDD 904 stores the data from RAM 903. CPU 901 reads and executes a program related to the target processing from the recording medium 912. Alternatively, CPU 901 may read a program related to the target processing from another device via a communication network (NW 920). 【0167】 <Effects> The effects of the binary fuzzing device, etc. according to this embodiment will be described below. The binary fuzzing device according to this embodiment is a binary fuzzing device 1 (Figure 1) that performs fuzzing tests on a test target software 800 composed of binary files, and comprises: an input generation unit 11 that generates one or more fuzzes by changing a seed indicating a test case; a test target execution unit 12 that inputs the generated fuzzes into the test target software 800 and causes it to execute; a system information collection unit 13 that collects system information indicating the operating status of the execution environment on which the test target software 800 is run when the test target software 800 is executed; and a coverage estimation unit 14 that estimates whether or not the coverage has increased using the collected system information. 【0168】 In this way, even if the software under test 800 cannot be compiled using only a binary file, the binary fuzzing device 1 can use system information obtained from the execution environment 700 to estimate whether or not coverage has increased, without reducing the execution speed. 【0169】 In the binary fuzzing device 1B (Figure 9), the coverage estimation unit 14B is characterized by estimating whether or not the coverage has increased in multiple stages using system information. 【0170】 By doing so, the binary fuzzing device 1B can reduce the time required for coverage estimation processing and improve execution speed. 【0171】 The binary fuzzing device 1A (Figure 6) further includes a response signal acquisition unit 15 that collects response signals from the software 800 under test during execution of the software 800 under test, and the coverage estimation unit 14A estimates whether or not the coverage has increased using system information and response signals. 【0172】 In this way, the binary fuzzing device 1A can improve the estimation accuracy by adding response signals from the software under test 800 to the system information obtained from the execution environment 700 as an information source for coverage estimation processing. 【0173】 In the binary fuzzing device 1B (Figure 9), the coverage estimation unit 14B is characterized by estimating whether or not the coverage has increased in multiple stages using system information and response signals. 【0174】 By doing so, the binary fuzzing device 1B can reduce the time required for coverage estimation processing and improve execution speed. 【0175】 In the binary fuzzing devices 1C (Figure 11), 1D (Figure 14), and 1F (Figure 18), the input generation units 11C, 11D, and 11F output the pre-mutation seed to the test target execution unit 12 and have the test target software 800 execute it, and the coverage estimation units 14C, 14D, and 14F estimate whether or not the coverage has increased by comparing the system information obtained when the fuzz is executed on the test target software 800 with some of the system information obtained when the test target software 800 was executed immediately before the execution of the fuzz. 【0176】 In this way, the binary fuzzing devices 1C, 1D, and 1F can reduce the number of comparison points in the coverage estimation process by comparing with a portion of the system information obtained from the most recent execution of the software under test 800, rather than comparing with all the execution paths observed up to that point. Therefore, the binary fuzzing devices 1C, 1D, and 1F can estimate whether or not coverage has increased at high speed. 【0177】 In the binary fuzzing device 1E (Figure 17), the system information collection unit 13E performs a predetermined feature extraction process to process the collected system information and extract features, and the coverage estimation unit 14 estimates whether or not the coverage has increased using the extracted features of the system information. 【0178】 By doing so, the binary fuzzing device 1E can reduce the amount of information to be stored, improve the accuracy of the coverage estimation process, and increase the processing speed. 【0179】 It should be noted that the present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of the present invention by those with ordinary skill in the art. For example, in the binary fuzzing devices 1 to 1F according to this embodiment, each functional unit of the control unit 10 (fuzzing tool) is provided in the same device as the device on which the software under test 800 is running, that is, in a single housing. However, it is also possible to realize the functions of the present invention by providing the functions of the input generation unit, the test target execution unit, and the coverage estimation unit in separate devices, and transmitting and receiving information between these devices and the device containing the software under test. 【0180】 1, 1A, 1B, 1C, 1D, 1E, 1F Binary fuzzing device 10 Control unit (fuzzing tool) 11, 11C, 11D, 11F Input generation unit 12 Test target execution unit 13, 13E System information collection unit 14, 14A, 14B, 14C, 14D, 14F Coverage estimation unit 15, 15E Response signal collection unit 700 Execution environment (OS / kernel / CPU) 800 Test target software

Claims

1. A binary fuzzing device for performing fuzzing tests on software under test, which consists of binary files, comprising: an input generation unit that generates one or more fuzzes by mutating a seed representing a test case; a test target execution unit that inputs the generated fuzzes into the software under test and causes it to execute; a system information collection unit that collects system information indicating the operating status of the execution environment on which the software under test is run during the execution of the software under test; and a coverage estimation unit that estimates whether or not coverage has increased using the collected system information.

2. The binary fuzzing apparatus according to claim 1, characterized in that the coverage estimation unit estimates whether or not the coverage has increased in a multi-stage manner using the system information.

3. The binary fuzzing apparatus according to claim 1, further comprising a response signal acquisition unit that collects a response signal from the software under test during execution, wherein the coverage estimation unit estimates whether or not coverage has increased using the system information and the response signal.

4. The binary fuzzing apparatus according to claim 3, characterized in that the coverage estimation unit estimates whether or not the coverage has increased in a multi-stage manner using the system information and the response signal.

5. The binary fuzzing device according to claim 1, characterized in that the input generation unit outputs the seed before mutation to the test target execution unit and has the test target software execute it, and the coverage estimation unit estimates whether or not the coverage has increased by comparing the system information obtained when the test target software executes the fuzz with some of the system information obtained when the test target software was executed immediately before the execution of the fuzz.

6. The binary fuzzing apparatus according to claim 1, characterized in that the system information collection unit performs a predetermined feature extraction process to process the collected system information and extract features, and the coverage estimation unit estimates whether or not the coverage has increased using the extracted features from the system information.

7. A binary fuzzing method for performing fuzzing tests on software under test, which consists of binary files, comprising: a step of generating fuzz by mutating a seed representing a test case; a step of inputting the generated fuzz into the software under test and executing it; a step of collecting system information indicating the operating status of the execution environment on which the software under test is run during execution of the software under test; and a step of estimating whether or not coverage has increased using the collected system information.

8. A program for causing a computer to function as a binary fuzzing device according to any one of claims 1 to 6.

Images