Inspection device and inspection program

Abstract

This inspection device (79) has: a test execution unit (11) that executes fuzz testing by inputting fuzz generated by a service provision device (71) that provides an NW service; a measurement control unit (14) that controls a measurement agent so as to measure an evaluation index related to the impact of the fuzz on the NW service when the fuzz testing is executed; an analysis determination unit (12) that determines the impact of the fuzz testing on the NW service on the basis of the measurement results of various indices from the measurement agent; and a reporting unit (13) that reports the determination results from the analysis determination unit (12).

Description

Inspection device and inspection program

[0001] This invention relates to an inspection device and an inspection program.

[0002] To ensure the smooth provision of network services, network providers inspect system operation by separately performing "load testing" and "fuzzing testing." First, load testing is a method of measuring the impact on network services (impact on performance) under overload conditions or long-term testing using quality measurement tools. The quality measurement tool described in Non-Patent Literature 1 performs load testing under overload conditions by intentionally generating a large number of packets and processing requests. Through this, the quality measurement tool measures the quality of the network service. If an impact on the network service occurs, the quality measurement tool mitigates the impact on the network service by implementing control measures during operation.

[0003] On the other hand, fuzzing testing is a negative test in which a fuzzing tool intentionally creates exceptional situations by inputting fuzz, which is unexpected data, into the system (Non-Patent Documents 2, 3). Fuzzing tools can efficiently discover bugs and vulnerabilities by performing fuzz generation processing and defect detection processing. The procedure for performing fuzzing testing is as follows: (Procedure 1) The fuzzing tool creates and inputs unexpected data or requests as test cases into the software under test and observes the software's behavior in response. (Procedure 2) If unexpected behavior occurs in the software under test, the fuzzing tool notifies the test engineer of the problem information and its test case as a report. Unexpected behavior includes, for example, crashes, memory violations, processing hangs or timeouts, and abnormal consumption of specific resources such as memory usage or power consumption. (Procedure 3) The test engineer reviews and analyzes the report contents regarding the unexpected behavior.

[0004] Non-Patent Literature 2 describes that fuzz generation, defect detection, coverage-based test search optimization, etc. are automated as a fuzzing technique for observing the internal behavior of software. Non-Patent Literature 3 describes that observation based on the usage amount of internal resources such as memory and sockets used by an application is performed as a fuzzing technique for measuring resource usage.

[0005] Takanori Hayashi, "QoE-centric Operation for Optimizing User Quality of Experience", [online], [retrieved January 30, 2025], Internet <URL: https: / / www.ntt-review.jp / archive / ntttechnical.php?contents=ntr201509fa3.pdf&mode=show_pdf>, NTT Technical Review Andrea Fioraldi et al., "AFL++: Combining Incremental Steps of Fuzzing Research", [online], [retrieved January 30, 2025], Internet <URL: https: / / aflplus.plus / papers / aflpp-woot2020.pdf> Einar Broch Johnsen, Manuel Wimmer (Eds.), "Fundamental Approaches to Software Engineering", [online], [retrieved January 30, 2025], Internet <URL: https: / / link.springer.com / chapter / 10.1007 / 978-3-030-99429-7_5>, 25th International Conference, FASE 2022 Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2022 Munich, Germany, April 2-7, 2022 Proceedings

[0006] As described below, quality measurement tools such as Non-Patent Document 1 and fuzzing tools such as Non-Patent Document 2 target different types of defects they wish to detect. First, load testing, such as that described in Non-Patent Document 1, measures the impact on network quality as part of system testing during the development and verification of network software. However, load testing is not a security-focused test and does not perform tests caused by specific packets or processes. Furthermore, it is costly and time-consuming to execute and therefore inefficient.

[0007] On the other hand, in fuzzing tests such as those described in Non-Patent Documents 2 and 3, the fuzzing tool primarily observes the internal behavior of the software under test, but does not measure the impact of fuzzing on network quality. Similarly, in Non-Patent Document 3, the test observation target during fuzzing is internal resources, and external resources such as network quality are not measured.

[0008] Here, the impact of fuzzing tests is not limited to severe defects that immediately affect the quality of network services, such as bugs and vulnerabilities. In other words, a system that receives fuzz input from a fuzzing tool may return a normal response, but defects that cause a long-term deterioration in the quality of the system may occur. In this case, conventional fuzzing tools do not have a mechanism to measure the quality of network services over the long term and therefore cannot detect such defects.

[0009] Therefore, the main objective of this invention is to detect a malfunction in which the quality of network services deteriorates due to the input of fuzz.

[0010] To solve the aforementioned problems, the inspection device of the present invention comprises the following means. The present invention is characterized by having: a test execution unit that performs a fuzzing test by inputting a generated fuzz to a service provider that provides network services; a measurement control unit that controls a measurement agent to measure evaluation indicators related to the impact of the fuzz on the network service when the fuzzing test is performed; an analysis and judgment unit that determines the impact of the fuzzing test on the network service based on the measurement results of various indicators from the measurement agent; and a notification unit that notifies the judgment result of the analysis and judgment unit.

[0011] According to the present invention, it is possible to detect a problem in which the quality of network services deteriorates due to the input of fuzz.

[0012] This is a configuration diagram of the service provision system according to this embodiment. This is a configuration diagram of the service provision system according to this embodiment. This is a configuration diagram of the service provision system according to this embodiment. This is a configuration diagram of the service provision system according to this embodiment. This is a configuration diagram showing the details of the test execution unit according to this embodiment. This is a flowchart showing the operation of the inspection device according to this embodiment. This is a table showing an example of indicator measurement data according to this embodiment. This is a table showing an example of a threshold value compared with the indicator measurement data according to this embodiment. This is a configuration diagram of the service provision system according to this embodiment. This is a table showing indicator measurement data collected from the measurement unit of the user terminal according to this embodiment. This is a table showing indicator measurement data collected from the measurement unit of the user terminal according to this embodiment. This is a table showing the resource efficiency improvement process by the measurement control unit according to this embodiment. This is an explanatory diagram showing the form in which measurement is performed with high overhead before efficiency improvement according to this embodiment. This is an explanatory diagram showing the form in which measurement is performed with lower overhead than in Figure 13 after efficiency improvement according to this embodiment. This is an explanatory diagram showing the measurement packet rewritten in the form of Figure 14 according to this embodiment. This is a flowchart in which the efficiency improvement process for executing a test scenario based on indicator measurement data has been added to the flowchart of Figure 6 according to this embodiment. This is a hardware configuration diagram of each device in the service provision system according to this embodiment. This is a table summarizing the inspection items of the inspection device according to this embodiment.

[0013] One embodiment of the present invention will be described in detail below with reference to the drawings.

[0014] Figure 1 is a diagram showing the configuration of service provision systems 70A to 70C. As described below, service provision systems 70A to 70C provide NW services via the service provision device 71. • Service provision system 70A has a service provision device 71 and a user terminal 72. The service provision device 71 provides services such as a search engine that responds to input (request packets, etc.) from the user terminal 72 handled by the end user. • Service provision system 70B has a service provision device 71 and user terminals 72A and 72B. The service provision device 71 provides end-to-end services between multiple users (two user terminals 72A and 72B in Figure 1). The service provision device 71 mediates transmitted data (e.g., video and audio in video conferencing) from each user terminal 72A and 72B to other user terminals. • Service provision system 70C has a service provision device 71, a user terminal 72, and a server 73. The service provider 71 acts as an intermediary between the user terminal 72 and the server 73 to provide data such as video content from the server 73 to the user terminal 72.

[0015] Figure 2 is a configuration diagram of the service provision system 70D. The service provision system 70D is configured by adding an inspection device 79 for inspecting the service provision device 71 compared to the service provision system 70B. On the other hand, the service provision systems 70A and 70C may also be configured by adding an inspection device 79 for inspecting the service provision device 71 (not shown).

[0016] The inspection device 79 operates as a fuzzing tool to inspect the software 71A under test within the service provision device 71 by inputting a fuzz signal to the software 71A under test. The inspection device 79 measures evaluation metrics (details in Figure 4) that affect the network service from the software 71A under test to which the fuzz signal has been input. If the inspection device 79 determines from the measured evaluation metrics that the network service has been affected, it executes controls on the software 71A under test to improve the impact on the network service.

[0017] In Figure 2, the software under test 71A of the service provider 71 works in conjunction with the coordinating component 71X to provide network services to user terminals 72A and 72B. In other words, network services such as microservices and containers are provided by two (or more) components: the software under test 71A and the coordinating component 71X. In this case, the test device 79 performs fuzzing on the control signals transmitted and received by the software under test 71A. The test device 79 then detects whether there are any unexpected operations or malfunctions in the software under test 71A based on its response to the fuzz and the internal behavior of the component software.

[0018] However, as a result of fuzzing the software under test 71A, the fuzz may cause the software under test 71A to execute an invalid request processing to the interoperating component 71X. In that case, the test device 79 will receive a correct response (200 OK) from the software under test 71A, but it may affect the network service functions provided by the interoperating component 71X. In particular, if the control signals and user data signals are separated, it is expected that the processing of network services related to the user data signals provided by the interoperating component 71X may be delayed, or the connection state between the interoperating component 71X and the user terminals 72 (72A, 72B) may become unstable or disconnected.

[0019] Therefore, when the inspection device 79 was operating as a conventional fuzzing tool, it would have missed detecting defects in the "user data signals between the interoperating component 71X and the user terminal 72," which were outside the scope of inspection. Thus, the inspection device 79 adds components that were not included in the fuzzing test to the scope of indicator measurement. Here, "indicator measurement" means measuring evaluation indicators that affect the NW service. These evaluation indicators may be NW quality evaluation indicators assumed by conventional quality measurement tools, or they may be evaluation indicators of QoE (Quality of Experience), which is the quality that users experience from the NW service.

[0020] Furthermore, the inspection device 79 simultaneously performs NW service index measurements when executing the fuzzing test. This allows the inspection device 79 to accurately detect any degradation in the quality of user data signals caused by performing a fuzzing test on the control signals of the software under test 71A. In addition, the inspection device 79 can efficiently inform the test personnel of the impact on NW services, such as quality degradation, by notifying them of the results of the fuzzing test and the index measurement results (index measurement data) as a report.

[0021] Figure 3 is a configuration diagram of the service provision system 70E. In the service provision system 70E, the software under test 71A of the service provision device 71 provides NW services to user terminals 72A and 72B using the buffer cache 71B and disk 71C within the service provision device 71. The inspection device 79 inputs fuzz to the software under test 71A when the fuzzing test is executed. The input fuzz can cause unnecessary data to be stored in the buffer cache 71B, which can lead to a decrease in the effectiveness of the buffer cache 71B or even cause the buffer cache 71B to be cleared. As a result, I / O processing occurs on the disk 71C that supplies data to the buffer cache 71B, and this I / O processing affects the quality of the NW service.

[0022] Therefore, when the inspection device 79 was operating as a conventional fuzzing tool, it would have missed detecting defects in "data access to the buffer cache 71B and disk 71C," which were not included in the inspection target. To address this, the inspection device 79 adds components not included in the fuzzing test to the scope of indicator measurement. Furthermore, when executing the fuzzing test, the inspection device 79 simultaneously performs indicator measurements related to data access to the buffer cache 71B and disk 71C. This allows the inspection device 79 to accurately detect any degradation in data access quality caused by performing a fuzzing test on the control signals of the software 71A under test. Additionally, the inspection device 79 can efficiently inform the test personnel of the impact on network services, such as quality degradation, by notifying them of the results of the fuzzing test and the indicator measurements in a report.

[0023] As described above, Figures 2 and 3 illustrate examples where it is difficult to detect defects and bugs that cause a degradation of network service quality using existing fuzzing tests. Even in these examples, the inspection device 79 of this embodiment can detect defects and vulnerabilities that could affect network services during the development and verification phase.

[0024] Figure 4 is a configuration diagram of the service provision system 70F. The service provision system 70F provides a detailed explanation of the function of the inspection device 79 in relation to the service provision system 70D in Figure 2 and the service provision system 70E in Figure 3, and also indicates that measurement agents are provided in devices other than the inspection device 79. On the other hand, the service provision system 70F may also be operated as the inspection device 79 of the service provision systems 70A and 70C.

[0025] Measurement agents that measure evaluation metrics are classified into one of the following categories: • "Client type" measures evaluation metrics as a user terminal 72 (client), assuming an impact on other users. This measurement process is, for example, an E2E (End-to-End) measurement between client types. The client type may be placed inside the user terminal 72, or it may be placed outside the user terminal 72 and operate as a client simulating the user terminal 72. • "Server-resident type" measures evaluation metrics by residing outside the test software 71A on the service provision device 71 where the test software 71A runs. Implementation examples for the server-resident type include running in a separate process, virtualization infrastructure, virtual machine, or container. • "Process type" operates as a separate process within the server-resident type. • "Built-in type" measures evaluation metrics by being installed and compiled into the test software 71A. The built-in type can acquire evaluation metrics that cannot be obtained with the server-resident type.

[0026] Each device is equipped with at least one measurement agent, as follows: • Measurement unit 81 in the service provision device 71 (server-resident or built-in type) • Measurement units 82A and 82B in the user terminals 72A and 72B (client type) • Measurement unit 83 in the inspection device 79 (client type simulating user terminal 72)

[0027] The measurement agent measures, for example, at least one of the following as a network quality evaluation metric: packet delay, jitter, packet loss, throughput, connection interruption, and error correction. Furthermore, the measurement agent may also measure metrics standardized by ITU-T SG12 WP1: Terminal Specification and Subjective Quality Evaluation Method, WP2: Objective Quality Evaluation, and WP3: QoS / QoE Specification. A decrease in these network quality evaluation metrics can be presumed to indicate a decline in network quality and an impact on network services. When providing video and audio services, the measurement agent measures, for example, at least one of the following as a QoE evaluation metric: video degradation, block noise count, bitrate, frequency of video playback interruptions, frequency of audio noise occurrence, and frequency of audio delay occurrence. Note that the QoE evaluation metrics vary depending on the type of network service provided.

[0028] In this way, by deploying measurement agents over a wide area within the service provision system 70F, the measurement targets of the fuzzing test, which were previously limited to the behavior of the software under test 71A itself and the internal resources of the service provision device 71, are expanded to include network service user quality. Furthermore, by deploying measurement agents in multiple forms (client type simulating other users, server-resident / inserted type, etc.), diverse indicator measurement data can be collected from a wide range of sources.

[0029] The inspection device 79 comprises a test execution unit 11, an analysis and judgment unit 12, a notification unit 13, and a measurement control unit 14. The test execution unit 11 performs a fuzzing test by inputting the generated fuzz to the service provider device 71 that provides the NW service. The measurement control unit 14 controls the measurement agent to measure evaluation indicators related to the impact of the fuzz on the NW service when the fuzzing test is performed. The analysis and judgment unit 12 determines the impact of the fuzzing test on the NW service based on the measurement results of various indicators from the measurement agent. The notification unit 13 notifies the analysis and judgment unit 12 of its determination result. The details of each part of the inspection device 79 will be described below.

[0030] The test execution unit 11 starts and configures the software under test 71A, and then inputs the generated fuzz into the software under test 71A to perform a fuzzing test. During the fuzzing test, the test execution unit 11 waits for a response from the software under test 71A and for measurement results from the measurement agent. When the fuzzing test is performed, the measurement agent measures various indicators related to the impact on network services and users, and transmits the measurement results to the analysis and judgment unit 12.

[0031] The measurement control unit 14 dynamically controls the quality measurement tools of each measurement agent to measure evaluation indicators during the period in which the test execution unit 11 performs the fuzzing test. This dynamic control includes, for example, determining which evaluation indicators each measurement agent will measure, where to place the measurement agents, and how to manage settings such as the lifecycle management of the measurement agents. For example, when a network service (end-to-end service between clients) is provided between multiple user terminals 72 (72A, 72B) via the service provision device 71, the measurement control unit 14 controls the measurement agents (measurement units 82A, 82B) within the user terminals 72 to measure evaluation indicators related to that network service.

[0032] The analysis and judgment unit 12 determines whether the fuzzing test is successful or not as a fuzzing tool based on whether or not it has discovered bugs or vulnerabilities caused by the fuzz input. Furthermore, the analysis and judgment unit 12 determines whether or not the fuzzing test has an impact on the network service (quality evaluation) based on the indicator measurement data collected from the measurement agent and the threshold values ​​of each indicator. The threshold values ​​are set in advance by the test personnel for the indicators to be measured, or calculated using a machine learning algorithm with normal data as training data. There may be multiple judgment levels; for example, the judgment results can be subdivided based on the degree of impact on the network service, such as impact on network service = high, medium, low. These judgment levels can be changed using coefficient parameters.

[0033] The notification unit 13 receives a notification from the analysis and judgment unit 12, compiles the fuzz that affected the NW service and the underlying indicator measurement data, and notifies the testers, etc., of a report. The report will include one of the following evaluation results, for example, with higher scores indicating a better evaluation. - Evaluation score "5" = The NW service is operating normally and the quality of service is good (fuzzing test = passed, and no impact on the NW service during the fuzzing test). - Evaluation score "4" = The NW service is operating normally, but very rarely unstable situations occur (fuzzing test = passed, and small impact on the NW service during the fuzzing test). - Evaluation score "3" = The NW service is operating normally, but unstable situations may occur (fuzzing test = passed, and medium impact on the NW service during the fuzzing test). - Evaluation score "2" = The NW service is operating normally, but unstable situations occur frequently (fuzzing test = passed, and high impact on the NW service during the fuzzing test). - Rating "1" = Network service is not operational (fuzzing test = failed).

[0034] Figure 5 is a configuration diagram showing the details of the test execution unit 11. The test execution unit 11 includes a seed data storage unit 11A, a seed data selection unit 11B, and a test data generation unit 11C. The seed data storage unit 11A stores the original fuzz data (normal data) used in the fuzzing test. The seed data storage unit 11A may also store the original fuzz data (unexpected data) for test cases that were judged to have had a high probability of affecting the network service in the previous fuzzing test, so that it can be used from the next fuzzing test.

[0035] The seed data selection unit 11B selects the source data for the fuzz to be used in the next test from the seed data storage unit 11A. The seed data selection unit 11B, for example, prioritizes the selection of the "unexpected data" over the "normal data". The seed data selection unit 11B can also determine the selection priority by combining it with other feedback information (e.g., coverage, resource usage). The test data generation unit 11C uses the fuzz selected by the seed data selection unit 11B to perform arbitrary fuzz mutation processing (e.g., bit string inversion, string replacement / insertion, etc.) to generate the fuzz to be used in the next fuzzing test.

[0036] Figure 6 is a flowchart illustrating the operation of the inspection device 79. This flowchart is executed, for example, as a security inspection process before the start of operation of a network service. This allows the inspection device 79 to detect and identify defects and vulnerabilities that affect the service quality of the user terminal 72 and the entire network during the security inspection phase (development and verification phase), which is a preliminary stage to service operation.

[0037] First, the test execution unit 11 of the inspection device 79 starts the fuzzing test according to S101 to S105. The test execution unit 11 starts the software to be tested 71A (S101) and starts the measurement agent (measurement unit) via the measurement control unit 14 (S102). The seed data selection unit 11B selects the seed data (original data for the test case) to be executed next from the seed data storage unit 11A (S103). The test data generation unit 11C performs an arbitrary data modification process on the selected seed data (S104). The test execution unit 11 sends fuzz (test data) to the software to be tested (S105).

[0038] During the fuzzing test, the measurement agent collects metric measurement data for the NW service (S106). Then, the analysis and judgment unit 12 receives the metric measurement data from the measurement agent after the fuzzing test (after fuzz transmission and behavior observation are complete) (S107). Furthermore, the analysis and judgment unit 12 analyzes the metric measurement data and determines whether or not it has an impact on the NW service (S108). If the answer in S108 is Yes, the analysis and judgment unit 12 generates a report indicating that the NW service has an impact and notifies the user of this report as an alert from the notification unit 13 (S111). The analysis and judgment unit 12 then proceeds to process S109. If the answer in S108 is No, the seed data selection unit 11B determines whether or not there is a test case to be executed next (S109). If the answer in S109 is Yes, the process returns to S103, and the process ends if the answer is No. The process shown in Figure 6, as described above, allows for the detection of defects and bugs that occur during security testing (fuzzing tests), as well as the detection of the impact on service quality resulting from fuzzing tests.

[0039] Figure 7 is Table 301, which shows an example of the metric measurement data collected in S106. As shown in Table 301, the measurement agent measures each metric (packet delay, number of packet losses, throughput, etc.) for each measurement period in a fuzzing test where a fuzz (any number) is input.

[0040] Figure 8 shows Table 302, which is an example of a threshold value that is compared with the indicator measurement data determined in S108. This Table 302 is set in advance by the test executor. The analysis and judgment unit 12 determines whether or not there is an impact on the NW service by comparing the indicator measurement data in Table 301 in Figure 7 with the threshold value in Table 302 in Figure 8 (S108). For this reason, Table 302 associates the item to be measured with the threshold value setting according to the analysis result of the indicator measurement data, as a rule for each indicator.

[0041] For example, in the measurement period "21-30" in Table 301, the packet delay value, one of the indicators, is measured at 18ms. Therefore, the analysis and judgment unit 12 determines that for the measurement period "21-30", the threshold setting "upper limit" value of rule "1" in Table 302, which is set at 15ms, has been exceeded, and thus the analysis and judgment unit 12 determines that an impact on the NW service (quality degradation) has occurred. Also, in the measurement period "21-30" → "31-40" in Table 301, the packet loss value, one of the indicators, has changed from 10 packets to 21 packets. Therefore, the analysis and judgment unit 12 determines that for the measurement period "31-40", the threshold change amount of rule "2" in Table 302, which is set at 10 packets, has been exceeded, and thus the analysis and judgment unit 12 determines that an impact on the NW service (quality degradation) has occurred.

[0042] The following describes a fuzzing method for detecting the cause of network service quality degradation with high accuracy, referring to Figures 9 to 11. First, network service quality degradation can occur due to a combination of various factors and causes, not just the effects of fuzzing tests. Therefore, it can be difficult to determine the cause of network service quality degradation based on the results of a judgment using only one type of fuzz and one measurement agent. For example, the analysis and judgment unit 12 determines whether or not the observed effects of network service quality degradation during the fuzzing test are due to the fuzzing test. As a method of determination, the analysis and judgment unit 12 may use a reproduction test using the sent fuzz, but this takes time to reproduce and may not reproduce correctly, which is problematic in terms of detection accuracy and efficiency.

[0043] It should be noted that examples of events that may affect the measurement of NW service quality include the following: - Events of NW quality degradation. For example, when the fuzzing tool and the physical enclosure (such as a server) of the software 71A under test are separated, and the NW between the enclosures is a wireless or mobile network. Or, when a large number of measurement packets or background calls flow in for indicator measurement, causing congestion in the NW or interface. - Events of influence caused by measurement methods (variations in delay measurement caused by measurement-dedicated applications and packets, resource changes of the measurement source terminal) - Events of insufficient CPU or memory resources, events of OS interruptions of the software 71A under test, events of thread contention, events of disk I / O, events of file system processing

[0044] Accordingly, an example of improving detection accuracy by performing correlation analysis on indicator measurement data from multiple locations will be described below. FIG. 9 is a configuration diagram of a service providing system 70G. In this service providing system 70G, the measurement unit 81 in the service providing apparatus 71 of the service providing system 70F in FIG. 4 is replaced with the following two measurement units: - Measurement unit 81A (built-in type) in the software 71A under test - Measurement unit 81B (server-resident type) in the operation infrastructure 71V. It should be noted that the operation infrastructure 71V refers to an OS or a virtualization infrastructure. Then, similar to the service providing system 70F, client-type measurement units 82A, 82B, and 83 are also arranged in the service providing system 70G. These measurement agents at multiple locations each measure indicator measurement data at the start of the fuzzing test.

[0045] Then, the analysis and determination unit 12 collects indicator measurement data from measurement agents arranged at multiple locations in the service providing system 70G. The analysis and determination unit 12 performs correlation analysis on the collected indicator measurement data, thereby determining the influence of NW service quality degradation from a plurality of information sources. That is, the analysis and determination unit 12 identifies the cause of NW service quality degradation through correlation analysis processing between measurement agents at multiple locations, or correlation analysis processing between internal resource information and external resource information of the software 71A under test acquired from each measurement agent.

[0046] First, as an example of correlation analysis processing between a plurality of measurement agents at multiple locations, when a certain measurement agent detects a degradation in NW service quality, the analysis and determination unit 12 performs time-series correlation analysis to analyze whether similar service quality degradation has occurred in other measurement agents and whether any changes have occurred in other index measurement items. This allows the analysis and determination unit 12 to isolate whether the event affects the entire NW service or is a problem caused by a specific measurement agent or the NW environment.

[0047] On the other hand, as an example of correlation analysis processing between internal resource information and external resource information of software under test 71A, when a certain measurement agent detects a degradation in NW service quality, the analysis and determination unit 12 performs time-series correlation analysis with internal resources (CPU, memory, disk I / O, etc.) of the software under test 71A, to isolate whether the degradation in NW service quality is caused by the software under test 71A. If the result of the correlation analysis shows that the cause of the quality degradation is an internal resource (such as disk I / O processing), the analysis and determination unit 12 analyzes index measurement data such as whether there is fuzz that may cause disk I / O processing or whether there is any interrupt processing. This allows the analysis and determination unit 12 to determine whether the degradation in NW service quality is caused by the fuzzing test or by an external factor.

[0048] As described above, the measurement control unit 14 controls a plurality of measurement agents at multiple locations to measure evaluation indices. Then, the analysis and determination unit 12 performs correlation analysis on the evaluation indices measured by the plurality of measurement agents at multiple locations to determine whether the factor of the influence on the NW service is caused by the fuzzing test. In other words, the analysis and determination unit 12 improves detection accuracy by performing correlation analysis on index measurement data from a plurality of locations. This allows the analysis and determination unit 12 to improve detection accuracy even when it is difficult to determine whether a change in a measurement index is caused by fuzzing or an external factor, compared with the crash detection mechanism used in conventional fuzzing (memory violation by address sanitizer, timeout observation).

[0049] The following describes a specific example of correlation analysis processing between multiple measurement agents. Figure 10 is a table 311 showing metric measurement data collected from the measurement unit 82A of user terminal 72A. Figure 11 is a table 312 showing metric measurement data collected from the measurement unit 82B of user terminal 72B. Tables 311 and 312 each record metric measurement data (packet delay, number of packet losses, throughput, etc.) for each measurement period.

[0050] When the measurement unit 82A detects a decline in NW service quality, the analysis and judgment unit 12 performs a correlation analysis of the measurement results from the measurement unit 82B in time series (in units of measurement period). Based on the results of the correlation analysis, the analysis and judgment unit 12 determines whether the decline in NW service quality is due to the fuzzing test, external factors, the suspected location, etc. For example, regarding the number of packet losses during the measurement period "11-20", the analysis and judgment unit 12 analyzes that it has increased in the measurement unit 82A (0 to 30 in table 311), but has hardly increased in the measurement unit 82B (0 to 2 in table 312). Therefore, the analysis and judgment unit 12 determines that since only the measurement unit 82A has changed, the disturbance is related to the client side of the measurement unit 82A or the NW environment of the measurement unit 82A.

[0051] In another example, regarding the packet delay during the measurement period "21-30", the analysis and judgment unit 12 analyzes that it has increased in both the measurement unit 82A (33 → 120 in table 311) and the measurement unit 82B (33 → 100 in table 312). Therefore, the analysis and judgment unit 12 determines that since both measurement units 82A and 82B have changed, the fuzzing test has affected the network service for multiple measurement agents.

[0052] Figure 12 is a table showing the resource efficiency optimization process performed by the measurement control unit 14. The table in Figure 12 associates the location of the measurement agent, the form of the measurement agent, and the measurement items (1, 2, 3, ...) of the measurement agent with each table entry ID. Note that the "measurement items" in Figure 12 are described in the format of "measurement frequency / resource limit". The measurement control unit 14 performs dynamic control of the measurement agents by setting and changing the settings for any item in the table in Figure 12 for each measurement agent.

[0053] For example, the measurement control unit 14 performs the following dynamic controls on the table in Figure 12: • Adding and deleting records with "ID: 1 to n" • Changing placement locations and measurement items to make them more effective • Adding and deleting measurement items • Changing the threshold value of measurement items (e.g., CPU resources 30% or less) • Changing the measurement agent and measurement items set according to the time

[0054] Such dynamic control is implemented when the deployment of multiple measurement agents necessitates dynamic changes to the placement of measurement agents or the metrics measured. For example, dynamic control is required in the following cases: • When you want to dynamically change service metric measurement items based on test cases. • When the resources of the terminal where the measurement agents are deployed are limited, such as an IoT terminal or terminal simulation. • When you want to increase the number of measurement agents from a single to multiple to improve measurement accuracy. • When you want to measure the impact on network services when a mobile terminal performs a handover.

[0055] Furthermore, the measurement control unit 14 may perform dynamic control as illustrated by the following variations: • Control / management of indicator measurement items and frequency: Controls the indicator items and measurement frequency measured by the measurement agent. In particular, the measurement control unit 14 sets which indicator measurement data to acquire for which indicator items. • Control / management of indicator measurement resources: Controls the amount of resources such as memory and CPU used by the measurement agent. For example, when the amount of resources is low, the measurement control unit 14 dynamically controls the measurement content, such as limiting the number of measurement items or reducing the measurement frequency, or when there is ample resources, expanding the number of measurement items. • Control / management of delivery format: Controls the form of the measurement agent (process type, client type, server-resident type, built-in type) and the delivery format (process, VM, container, etc.). • Control / management of deployment location and number of deployments: Controls the deployment location and number of deployments of the measurement agent. • Measurement agent lifecycle management: Executes lifecycle management of the measurement agent.

[0056] As explained above in Figure 12, the measurement control unit 14 dynamically controls the number and placement of measurement agents, causing each measurement agent to perform resource control and data collection. This enables the measurement control unit 14 to achieve efficient resource utilization and efficient exploration.

[0057] The following describes an efficient configuration for a measurement agent using eBPF (extended Berkeley Packet Filter) or general-purpose hardware, with reference to Figures 13 and 14. Figure 13 is an explanatory diagram showing a configuration in which measurements are performed with high overhead before efficiency improvements. Server 73 (or service provider 71) sends a packet 79A for NW services (background call for video distribution, etc.) to user terminal 72A. Then, the measurement unit 83 in server 73 (or service provider 71) measures the indicator measurement data with the measurement unit 82A in user terminal 72A using a measurement-dedicated packet 79B of the user program. Here, each measurement agent consumes resources and overhead is incurred in connection with the measurement. As a result, there may be an impact on the fuzzing test itself, or it may not be possible to efficiently obtain accurate measurement data.

[0058] Figure 14 is an explanatory diagram showing a configuration in which measurements are performed with lower overhead than in Figure 13 after optimization. The measurement control unit 14 sets up and starts the measurement agents (measurement unit 82A, measurement unit 83B) in a low-overhead environment. These measurement agents can perform measurements with low overhead in order to reduce overhead during indicator measurement (e.g., kernel processing when measuring as a user program) and resource consumption by measurement packets.

[0059] An example of a low-overhead environment is eBPF or XDP (eXpress Data Path), a packet processing program that utilizes eBPF. The measurement agent executes the eBPF program at the OS kernel layer, acquiring network and I / O event information and using eBPF packet rewriting processing. This makes it possible to measure latency and other parameters without using latency measurement packets, enabling the acquisition of real-time, low-overhead, efficient, and more accurate metric measurement data.

[0060] As another example of a low-overhead environment, the measurement agent acquires metric measurement data with high accuracy and low overhead by performing measurement processing on general-purpose hardware. As a technology for general-purpose hardware on which the measurement agent operates, for example, the technology for high-precision network monitoring and control on general-purpose hardware, "HANMOC (High-Accuracy Network MOnitoring and Control)", is described at <URL: https: / / www.rd.ntt / research / NIC0008.html>. The measurement unit 83B on the transmitting side of the measurement packet 79C adds a timestamp and packet ID by rewriting a part of the general-purpose packet (e.g., options in the header). The measurement unit 82A on the receiving side of the measurement packet 79C reads the rewritten part of the packet.

[0061] Figure 15 is an explanatory diagram showing the measurement packet 79C rewritten in the form shown in Figure 14. Here, an RTP (Real Time Transport Protocol) packet used for transmitting video data in a real-time video distribution NW service is used as an example of the measurement packet 79C. This measurement packet 79C stores, in order from the beginning, the Ethernet® header, IP header, UDP header, RTP header, and RTSP Payload. The RTP header stores, in order from the beginning, the V field, P field, X field, CC field, M field, PT field, ..., and the Extension header. The measurement unit 83B rewrites the RTP packet to embed measurement information through the following process. - Modification of the extended header flag (X) - Modification of the payload type (PT) - Embedding a timestamp, measurement packet ID, etc. within the extended header This eliminates the need to prepare a new measurement-dedicated packet 79B and allows the use of existing control signals (RTP packets), enabling the measurement agent to efficiently measure network service quality impact indicators (reduction of measurement packets, reduction of packet processing overhead, etc.).

[0062] Figure 16 is a flowchart of Figure 6 with the addition of a process to improve the efficiency of test scenario execution based on indicator measurement data. The test execution unit 11 selects the next fuzz candidate when selecting a fuzz using one of the following methods: - A method based on past coverage information. - A method based on internal resource usage, selecting a fuzz that consumes more resources as the next fuzz candidate.

[0063] On the other hand, the test execution unit 11 utilizes the NW service metric measurement data for fuzz selection to generate fuzzes that are more likely to have an impact on service quality (quality degradation). Therefore, the flowchart in Figure 16 adds the process of S112 after S111 in the flowchart in Figure 6. In S112, the test execution unit 11 adds the fuzzes that caused the degradation of NW service quality (fuzzes that were marked "Yes" in S108) to the seed data storage unit 11A, or changes the weight of the test case selection algorithm by the seed data selection unit 11B (making them more important). In other words, the analysis and judgment unit 12 stores the fuzzes that caused the degradation of NW service quality in the seed data storage unit 11A as candidates for the next fuzz. The seed data selection unit 11B then prioritizes selecting the fuzzes that caused the degradation of NW service quality from the next time onward (feedback).

[0064] Thus, if the analysis and judgment unit 12 determines that a fuzzing test will affect the network service, it stores the fuzz used in that fuzzing test and prioritizes the use of the stored fuzz in future fuzzing tests. As a result, the test execution unit 11 (test data generation unit 11C) selects the fuzz that is most likely to cause a degradation in network service quality and improves test efficiency in conjunction with the test scenario.

[0065] Figure 17 is a hardware configuration diagram of each device in the service provision system 70. Each device in the service provision system 70 (service provision device 71, user terminal 72, server 73, and inspection device 79) is configured as a computer 900 having a CPU 901, RAM 902, ROM 903, HDD 904, communication I / F 905, input / output I / F 906, and media I / F 907, respectively. The communication I / F 905 is connected to an external communication device 915. The input / output I / F 906 is connected to an input / output device 916. The media I / F 907 reads and writes data to the recording medium 917. Furthermore, the CPU 901 controls each processing unit by executing a program (instruction conversion program) loaded into the RAM 902. This program (application, also called an app) can be distributed via a communication line or recorded on a recording medium 917 such as a USB memory and distributed.

[0066] Figure 18 is a table summarizing the inspection items of the inspection device 79. First, the first row (upper section) of the table shows the items inspected by the measurement control unit 14 and the measurement agent (quality measurement tool). The quality measurement tool can detect crashes, memory violations, internal resource consumption (CPU, Mem), and impacts on network services and users (delay, packet loss, QoE, etc.) through load testing with overload conditions (overpacket / overprocessing) as the test target / monitoring target.

[0067] On the other hand, the second row (lower section) of the table shows the items inspected by the fuzzing test performed by the test execution unit 11 (fuzzing tool). The fuzzing tool can detect crashes, memory violations, and internal resource consumption (CPU, Mem) through fuzzing tests, with security checks (specific packets / specific processes) as the test / monitoring targets. Furthermore, the inspection device 79 (test execution unit 11) to which this embodiment is applied has a measurement agent measure the impact on NW services and users (delay, packet loss, QoE, etc.) during the fuzzing test (see "Applicable area added in this embodiment" in Figure 18). As a result, quality degradation caused by fuzz, which was not covered in conventional fuzzing tests, is now also observed.

[0068] [Effects] The inspection device 79 of the present invention is characterized by comprising: a test execution unit 11 that performs a fuzzing test by inputting the generated fuzz to a service provision device 71 that provides NW services; a measurement control unit 14 that controls the measurement agent to measure evaluation indicators related to the impact of the fuzz on the NW service when the fuzzing test is performed; an analysis and judgment unit 12 that determines the impact of the fuzzing test on the NW service based on the measurement results of various indicators from the measurement agent; and a notification unit 13 that notifies the judgment result of the analysis and judgment unit 12.

[0069] As a result, the inspection device 79 can detect defects that degrade the quality of network services due to fuzz input, which could not be detected by fuzz testing alone. Therefore, it becomes possible to perform inspections (high-precision and efficient inspections) that can improve network services more effectively in terms of reliability, availability, service quality, and service continuity / stability than security inspections using fuzz testing alone.

[0070] The inspection device 79 of the present invention is characterized in that, when a network service is provided between multiple user terminals 72 via a service provision device 71, the measurement control unit 14 controls the measurement agent in the user terminal 72 to measure an evaluation index related to that network service.

[0071] As a result, the inspection device 79 can detect the impact of the fuzzing test on the network service over a wide range of areas by including the user terminal 72, which is outside the service provision device 71, as part of its observation target.

[0072] The inspection device 79 of the present invention is characterized in that the measurement control unit 14 controls multiple measurement agents to measure evaluation indicators, and the analysis and judgment unit 12 determines whether or not the factors affecting the network service are due to fuzzing tests by performing a correlation analysis of the evaluation indicators measured by the multiple measurement agents.

[0073] As a result, the inspection device 79 can detect malfunctions in the network service with high accuracy, and security inspections can be made more efficient.

[0074] The inspection device 79 of the present invention is characterized in that, when the analysis and judgment unit 12 determines that the fuzzing test affects the network service, it stores the fuzz used in the fuzzing test and preferentially uses the stored fuzz in future fuzzing tests.

[0075] This allows the inspection device 79 to select a fuzz that is more likely to cause a degradation in network service quality, and to improve test efficiency by linking it with the test scenario.

[0076] 11 Test Execution Unit 11A Seed Data Storage Unit 11B Seed Data Selection Unit 11C Test Data Generation Unit 12 Analysis and Judgment Unit 13 Notification Unit 14 Measurement Control Unit 70 Service Provision System 71 Service Provision Device 71A Software under Test 71B Buffer Cache 71C Disk 72 User Terminal 71V Operating Board 71X Interoperability Components 73 Server 79 Inspection Device 81, 82A, 82B, 83 Measurement Unit (Measurement Agent)

Claims

1. An inspection device comprising: a test execution unit that performs a fuzzing test by inputting a generated fuzz to a service provider that provides network services; a measurement control unit that controls a measurement agent to measure evaluation indicators related to the impact of the fuzz on the network services during the execution of the fuzzing test; an analysis and judgment unit that determines the impact of the fuzzing test on the network services based on the measurement results of various indicators from the measurement agent; and a notification unit that notifies the judgment result of the analysis and judgment unit.

2. The inspection apparatus according to claim 1, characterized in that, when the network service is provided between multiple user terminals via the service provision device, the measurement control unit controls the measurement agent in the user terminal to measure the evaluation index related to the network service.

3. The inspection apparatus according to claim 1, characterized in that the measurement control unit controls the measurement agents at multiple locations to measure the evaluation index, and the analysis and judgment unit determines whether or not the factors affecting the network service are due to the fuzzing test by performing a correlation analysis of the evaluation index measured by the multiple measurement agents.

4. The inspection apparatus according to claim 1, characterized in that, if the analysis and determination unit determines that the fuzzing test affects the network service, it stores the fuzz used in the fuzzing test and preferentially uses the stored fuzz in future fuzzing tests.

5. An inspection program for causing a computer to function as the inspection device described in any one of claims 1 to 4.

Images