Computer-implemented method for carrying out explorative testing of a technical system to be tested
Patent Information
- Application Number
- PCT/EP2026/060575
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-04-17
- Filing Date
- 2026-04-13
- Publication Date
- 2026-10-22
Smart Images
Figure EP2026060575_22102026_PF_FP_ABST
Abstract
Description
[0001] 202505094
[0002] 1
[0003] Description
[0004] Computer-implemented method for the exploratory testing of a technical system under test
[0005] 1. Technical field
[0006] The invention relates to a computer-implemented method for the exploratory testing of a technical system under test. Furthermore, the invention relates to a corresponding test unit and a corresponding computer program product.
[0007] 2. State of the art
[0008] Software systems are often state-dependent, such as software controllers in cyber-physical systems. This means that the software performs different actions depending on the system state. The system state arises during operation through a combination of external events from the environment, such as received sensor data, and internal states of the control logic itself.
[0009] Furthermore, at least in some areas, partial specifications of the technical system and, in particular, the control software exist. These partial specifications offer degrees of freedom for designing specific operating scenarios of the technical system and define the expected system behavior at an abstract level.
[0010] The challenge in testing software systems and validating their behavior lies in providing a sufficient number of test cases that cover the state space of the technical system or subsystem as completely as possible. However, this is rarely achievable in practice. Providing adequately tested software systems remains a major challenge, especially for embedded software in the context of cyber-physical systems.
[0011] Several state-of-the-art approaches have been developed to overcome these challenges. One such approach involves test case generation based on code coverage, such as statement or condition coverage. With this approach, the tests are executed automatically or semi-automatically.
[0012] 2
[0013] This is generated based on the control flow in the software. However, considering the control flow alone does not relate to the system states. Repeated execution of the same code in different states can lead to different responses from the technical system.
[0014] Another state-of-the-art approach is model-based test case generation, also known as Model-Based Testing (MBT). After creating a model from existing specifications that represents the counterpart to the system or subsystem under test, the tests are systematically derived according to appropriate coverage criteria, similar to those used for code coverage. One advantage of this approach is that the models can accurately represent states, for example, when directly using state machines. This approach therefore yields effective tests. However, it requires fully specified models from which the tests can be derived before test execution.
[0015] Exploratory testing is a technique in which the technical system under test is explored ad hoc. Test cases are created during test execution and are not predefined. It is a time-boxed approach based on the test engineer's personal freedom. The technique does not define a coverage measure. Since the tests are not predefined, they can adapt to system changes. However, they are difficult to reproduce if individual test steps are incompletely logged. Exploratory tests are typically performed manually by a test engineer. The disadvantage is that the approach is error-prone, time-consuming, and inefficient. Therefore, the effort is high compared to automated approaches.
[0016] The known approaches each have their strengths and limitations in dealing with the complexity of testing software systems, especially for stateful technical systems for which only partial specifications exist. There remains a need for improved methods that can efficiently explore and test such technical systems while balancing automation, adaptability, and coverage.
[0017] The present invention therefore aims to provide a computer-implemented method for the exploratory testing of a technical system under test, which is efficient and reliable.
[0018] 3
[0019] 3. Summary of the invention
[0020] The above-mentioned problem is solved according to the invention by a computer-implemented method for the exploratory testing of a technical system to be tested, comprising the steps:
[0021] a. Provision of a database by a test unit; wherein
[0022] the data basis comprises a test model, an initial exploration model and at least one metric; wherein
[0023] The test model defines a plurality of system inputs and corresponding system outputs; where
[0024] Each system output of the majority of corresponding system outputs is a response to a system input; where
[0025] The initial exploration model represents at least one system behavior; where the at least one metric defines a coverage measure over the initial exploration model;
[0026] b. Determining at least one first system input from the majority of the system inputs of the test model by the test unit based on the exploration model and the at least one metric using a test strategy; wherein
[0027] the testing strategy relates to at least one exploration target and at least one metric; wherein
[0028] the test unit via which at least one initial system input provides stimulation to the technical system under test;
[0029] c. Execution of at least one initial system input on the technical system under test by the test unit;
[0030] d. Updating the initial exploration model, taking into account at least one initial system input and at least one corresponding initial system output of the technical system under test by the test unit;
[0031] e. Updating at least one metric taking into account the updated exploration model by the test unit;202505094
[0032] 4
[0033] f. Determine at least one second system input from the majority of the system inputs of the test model by the test unit based on the updated exploration model and the updated at least one metric using the test strategy;
[0034] g. Execution of at least one second system input on the technical system under test by the test unit;
[0035] h. Updating the updated exploration model, taking into account at least one second system input and at least one corresponding second system output of the technical system under test by the test unit;
[0036] i. Updating the updated at least one metric, taking into account the updated exploration model, by the test unit;
[0037] j. Termination of the iterative execution by the test unit when at least one exploration goal has been achieved.
[0038] Accordingly, the invention relates to a computer-implemented method for the exploratory testing of a technical system under test (SUT). In other words, the exploratory testing of the technical system under test is performed automatically by the test unit. The technical system under test can be any technical system and is preferably designed as a software system. The software system can comprise one or more software components. The test unit can be any computing unit, such as a data processing device.
[0039] The technical system under test can be provided and stimulated by system inputs. Furthermore, the corresponding reactions of the technical system under test to the system inputs can also be observed. These reactions are the system outputs. The system outputs can also be understood as system responses. The behavior of the technical system under test is preferably unknown at the outset and can be explored during testing and validated against the specified expectations from the test model. The technical system under test can be a simulation of the technical system or an executable system model, provided that the system reactions important for the tests are sufficiently simulated or modeled by the simulation or the executable system model.
[0040] 5
[0041] Furthermore, the technical system under test is preferably resettable. This means that the initial state of the technical system is known and can be accessed again at any time. Depending on the controllability of the technical system under test, other accessible states may be possible as starting states for the test execution.
[0042] In exploratory testing, there are typically no predefined test cases. The test strategy is preferably adaptive. This means that the behavior of the test unit is not predetermined during execution. The test unit selects its system input based on current knowledge from the database, guided by the test strategy. The exploration goal can be defined by a metric.
[0043] In the first step of the process, the input data is provided in the form of the database. The database can be understood as a knowledge base and serves as the basis for the test strategy to determine the input for the subsequent process steps. The database comprises the test model, the initial exploration model, and the metric.
[0044] The test model comprises the system inputs and expected system outputs. The possible system inputs of the technical system under test are defined in the states of the test model. Furthermore, their parameters and / or constraints, under which the system inputs are executed, can also be defined in the test model. Additionally, the assertions regarding the system outputs can be defined, taking into account the selected system input and the respective state in the test model. The test model is preferably defined only partially, more preferably depending on the level of detail and / or available requirements.
[0045] In other words, in some implementations, the test model can be represented using formal modeling languages or domain-specific languages. For example, it can be described using state machines, where states represent different system conditions and transitions represent inputs that move the system between states. The test model need not be complete or fully specified. In some cases, it may contain partial information about the system, reflecting the knowledge available at the time of testing. This partial specification allows for flexibility in exploring system behaviors that may not be fully understood or documented.
[0046] 6
[0047] The test model can also contain information about the relative importance or priority of different system inputs or system behaviors. This information can be used by the test team to focus on areas of the technical system that are considered more critical or prone to failure.
[0048] Consequently, the test model describes a target specification: allowed system inputs in the respective state, expected system outputs after input in the respective state.
[0049] The initial exploration model is built from the system inputs and observed system outputs. This model can be constructed during test runtime or post-mortem using logs and / or execution protocols. The execution protocols allow the use of field data. In other words, the exploration model can be a state model. With a state model, a new state is assumed after each system input. The initial state is known. The exploration model is extended during the exploration process and updated based on new observations.
[0050] The initial exploration model is the exploration model available at the start of the procedure. If the procedure has not been executed previously, it is usually empty. That is, it consists only of the initial state. If the procedure is applied repeatedly, the initial exploration model describes the exploration model from the last execution, e.g., from regression tests. This reuse of a previously created exploration model allows its incremental extension to include new states with their associated system inputs and outputs. This enables the procedure to more closely approximate the exploration goal chosen in the test strategy.
[0051] Consequently, the exploration model describes an as-is specification: observed system inputs and outputs, as well as observed states.
[0052] The metric describes a coverage measure across the exploration model, serving as a basis for decision-making in the test strategy for further exploration and test completion. In other words, the metric can be used to define the exploration goal.
[0053] The test strategy can also be provided as input. The test strategy attempts to achieve at least one exploration goal. The test strategy can be either 202505094
[0054] 7
[0055] Consider both the test model and the exploration model updated in one execution step.
[0056] The database can be received via one or more input interfaces of the test unit. These input interfaces advantageously ensure efficient and smooth data transfer between computing units. Data can be exchanged bidirectionally without data congestion.
[0057] In a first iteration or pass, the first system input is determined from the majority of system inputs in the test model. This determination is performed by the test unit using the test strategy. The test strategy might, for example, specify that each system input must be determined at least once and executed on the technical system under test. After determining the first system input, it is executed on the technical system under test, and the corresponding system output is observed. This execution is performed by the test unit. Following the execution of the first system input, the initial exploration model is updated. During this update, both the first system input and the corresponding observed system output are considered as responses from the technical system under test. This update is also performed by the test unit.The metric will also be updated.
[0058] The first iteration results in the updated exploration model as an intermediate step. This updated exploration model is further refined and adapted with each subsequent iteration. However, the second and therefore every subsequent update takes into account the second (or subsequent) system input and the corresponding system output. Consequently, the first system input or previous inputs are no longer considered.
[0059] In other words, the exploration model grows and becomes more refined. Initially, the exploration model might consist only of the known initial state of the technical system. With each cycle, new states, transitions, and / or behaviors can be added based on the observed responses of the technical system under test to various system inputs.
[0060] The construction of the exploration model can typically involve the creation of nodes representing system states and edges representing transitions between states.
[0061] 8
[0062] which are triggered by specific system inputs. For example, if a system input A is applied to the technical system in state S1 and leads to a transition to state S2, this would be represented in the exploration model as two nodes (S1 and S2) connected by an edge labeled with the system input A.
[0063] The exploration model can be updated in several ways:
[0064] 1. Adding at least one state: If a previously unobserved system state occurs, a new node is added to the exploration model.
[0065] 2. Adding at least one transition: When a new input-output pair is observed for a given state, a new edge is added to the exploration model.
[0066] 3. Refining at least one existing state: As more information is gathered about a particular state, its representation in the exploration model can be updated with additional details or attributes.
[0067] 4. Merging compatible states: In some cases, states initially considered different can be identified as equivalent based on their behavior. These states can be merged in the exploration model to create a more precise representation.
[0068] In some implementations, the test model can be updated or refined as new information about system behavior is discovered. This dynamic aspect of the test model allows the test team to adapt its test strategy based on the evolving understanding of the technical system under test.
[0069] The iterative execution is terminated in the final process step by the test unit as soon as the exploration goal is met. This exploration goal can be defined by a metric. For example, the exploration goal is met as soon as the exploration model no longer changes, or in other words, remains constant or unchanged after a further update. The exploration goal can also specify a certain number of iterations, a specific time period, a size of the exploration model, at least one criterion of the exploration model, and / or a computational effort.
[0070] 9
[0071] Regarding size, the exploration objective is fulfilled, for example, as soon as the predefined size is reached and / or exceeded. The exploration objective is also fulfilled, for example, when the available resources, especially the execution time, have been exhausted.
[0072] The computer-implemented procedure can be carried out within a specific timeframe. Additionally or alternatively, the computer-implemented procedure can be event-driven and / or dependent on specific criteria.
[0073] Unless otherwise specified, all steps of the computer-implemented method can be performed by the test unit, which can also be referred to as the data processing device. In particular, the data processing device, which comprises at least one processing circuit configured or adapted to carry out a computer-implemented method according to the invention, can perform the steps of the computer-implemented method. For this purpose, a computer program can be stored in the data processing device, in particular one containing instructions which, when executed by the data processing device, in particular the at least one processing circuit, cause the data processing device to execute the computer-implemented method.
[0074] The present invention enables the systematic approach of model-based testing to be reconciled with the approach of adaptive, exploratory testing. In contrast to the known approach, Model-Based Testing (MBT), according to the prior art, a complete model is not required for test derivation; instead, the next test step is determined at runtime according to predefined criteria. The evaluation of the system outputs is based on the existing (partial) specification.
[0075] Further advantages of the present invention are:
[0076] • The computer-implemented method makes it possible to test technical systems where the specification is only incomplete and partial.
[0077] • Knowledge about the system's runtime behavior is explicitly recorded in an exploration model that describes the true, observed behavior of the technical system.
[0078] 10
[0079] • Testing and exploration take place together in one step. System responses are tested according to existing system knowledge even during the exploration phase.
[0080] • System behavior is explored efficiently and reliably, unlike traditional exploratory testing, which is usually manual (human-in-the-loop). This significantly reduces effort and the potential for errors, enabling a much wider range of tests.
[0081] • The effort required to create the exploration model can be controlled by defining suitable test strategies for selecting the states to be explored, the actions to be taken, the parameters and / or the test end criterion.
[0082] • An objectively verifiable criterion for the completeness of the exploration is specified in the form of the exploration goal, so that an objective basis for achieving test coverage exists.
[0083] In one embodiment, the test model, the exploration model, and / or at least one metric of the database are stored in a volatile or non-volatile storage unit. The database, or parts thereof, are stored in a volatile or non-volatile storage unit, preferably a database or cloud. Accordingly, the storage unit can be a database, a cloud, or any other volatile or non-volatile storage unit. The storage unit enables reliable and fast data backup. The storage unit can be flexibly selected with regard to its storage capacity, scalability, etc. A further advantage is the efficient access to the data. The database, particularly the resulting exploration model, can advantageously be reused for subsequent test runs, for example, in a regression test phase, and thus incrementally expanded.
[0084] In a further embodiment, the test unit implements a Monitor-Analyze-Plan-Execute over Knowledge (MAPE-K) architecture. The test unit acts as a counterpart to the technical system under test and, via its system inputs, stimulates the system. Furthermore, the test unit monitors the system outputs of the technical system under test. The test unit can utilize the MAPE-K model of Autonomous Computing (Monitor, Analyze, Plan, Execute over a shared Knowledge).
[0085] 11
[0086] Implement a Base architecture. This architecture enables adaptive and systematic exploration of the behavior of the system under test.
[0087] In a further development, updating the exploration model also includes:
[0088] - Consolidating compatible states; where consolidation includes
[0089] - Identifying states that produce identical system outputs in response to identical system inputs, and
[0090] - Merging the identified states.
[0091] Accordingly, compatible states are identified in the exploration model. Two states are compatible if, given two identical system inputs, they produce identical system outputs, and the subsequent states are also compatible. By identifying compatible states, the number of states in the exploration model can be drastically reduced if compatible states are mapped to a single state.
[0092] In a further iteration, the test strategy includes:
[0093] - Execute each system input of the majority of system inputs a minimum number of times;
[0094] - Paired testing;
[0095] - Executing all executable system inputs from a known system state; or - Executing at least one system input according to an application-specific prioritization.
[0096] In a further development, the procedure continues to show
[0097] - Provision of the updated exploration model; and / or
[0098] - Generation of at least one test based on the updated exploration model. Accordingly, the updated exploration model can be used to generate one or more tests using different test generation methods. The test model from the database can be included in the generation process. The generated test is preferably a regression test. Generating the regression tests ensures that the system behavior is maintained in subsequent development steps. Additionally or alternatively, the updated exploration model can be output, saved, and / or verified. Verification makes it possible to determine whether the updated 202505094
[0099] 12
[0100] Exploration model safety guarantees or other properties were violated, or what guarantees and properties exist.
[0101] Furthermore, the invention relates to a data processing device for carrying out the above method.
[0102] The invention further relates to a computer program product comprising a computer program which includes means for carrying out the method described above when the computer program is executed on a program-controlled device.
[0103] A computer program product, such as a computer program tool, can be provided or delivered from a server on a network, for example, as a storage medium such as a memory card, USB stick, CD-ROM, DVD, or as a downloadable file. This can be done, for example, in a wireless communication network by transmitting the corresponding file containing the computer program product or tool. A suitable program-controlled device is, in particular, a control unit such as an industrial control PC, a programmable logic controller (PLC), or a microprocessor for a smart card or similar device.
[0104] 4. Brief description of the drawings
[0105] In the following detailed description, preferred embodiments of the invention are further described with reference to the following figures.
[0106] FIG 1 shows a schematic flowchart of the method according to the invention.
[0107] FIG 2 shows a schematic view of the test unit for exploratory testing according to an embodiment of the invention.
[0108] 5. Description of preferred embodiments
[0109] Preferred embodiments of the present invention are described below with reference to Figure 1. Figure 1 shows the flowchart of the computer-implemented method with process steps S1 to S10.
[0110] 13
[0111] In other words, a test model, which can contain the expectations of the technical system, is explored. The test model can be partially described and / or explored at runtime. During this process, knowledge about the system behavior is gathered and enriched in a separate exploration model. This can also be done at runtime. Explicit criteria can be defined to determine the next test step and to specify when the exploration of the technical system has achieved sufficient coverage of the system behavior to end the test. The knowledge gained in the form of the exploration model can be used for future tests, for example, in recurring regression tests.
[0112] According to one embodiment of the invention, the test unit for exploratory testing of a technical system to be tested can comprise the following subunits:
[0113] a. a storage unit that stores the database; wherein
[0114] the data basis includes the test model, the initial exploration model and at least one metric; wherein
[0115] The test model defines the majority of the system inputs and corresponding system outputs; where
[0116] Every system output is a response to a system input;
[0117] the initial exploration model that represents at least one system behavior;
[0118] which defines at least one metric as the coverage measure over the initial exploration model;
[0119] b. a monitoring subunit which is configured to monitor at least one first system input, at least one second system input, at least one first system output and at least one second system output;
[0120] c. a determination subunit which is used for
[0121] - Determining at least one initial system input from the majority of system inputs of the test model based on the initial exploration model using the test strategy; and
[0122] - Determining at least one second system input from the majority of the system inputs of the test model based on the updated exploration model set up according to the test strategy; 202505094
[0123] 14
[0124] d. an execution subunit which is used for
[0125] - Executing at least one initial system input on the technical system to be tested; and
[0126] - Execution of at least one second system input that is set up on the technical system to be tested; and
[0127] e. an update subunit which is used for
[0128] - Updating the initial exploration model, taking into account at least one initial system input and at least one corresponding initial system output of the technical system under test; and
[0129] - Updating the updated exploration model, taking into account at least one second system input and at least one corresponding second system output of the technical system under test, is configured. Furthermore, the update sub-unit for updating the metric is configured.
[0130] Figure 2 shows a schematic view of the test unit 10 for exploratory testing according to one embodiment of the invention. The test unit 10 can comprise one or more of the following subunits. The subunits can also be understood as components or subcomponents of the test unit 10.
[0131] Monitoring subunit
[0132] According to one embodiment of the invention, the monitoring subunit performs observation or monitoring of the technical system 20 under test. Monitoring can be continuous and / or runtime. Furthermore, the system environment can be monitored via sensors, interfaces of the technical system, and / or additional test interfaces. The observations can serve as a basis for determining the current system state, developing an understanding of it, and / or verifying the correctness of the system behavior according to the specifications of the test model. Additionally or alternatively, monitoring can be performed post-mortem and / or using previously recorded traces or logs.
[0133] 15
[0134] Update sub-unit
[0135] According to one embodiment of the invention, the update subunit analyzes the observed reactions of the technical system 20 under test. Furthermore, the update subunit can also analyze changes in the environment recorded by the monitoring subunit. The analysis can be divided into several subtasks:
[0136] a. Evaluation of the observed system outputs of the technical system under test 20 and / or environmental changes based on the specifications from the test model, which are described as conditions over variables and states that must be met during the test (so-called assets). In other words, the update subunit can detect any deviations from the expectations of the test model during test execution. A deviation can be signaled by a violated assert condition.
[0137] There are two possible outcomes, which can be resolved during subsequent error analysis, for example, through user interaction: The expectation is correct, and the deviation results from a fault in the technical system under test (so-called true positive in the test). Alternatively, the expectation is incorrect, at least in the state in which the deviation occurred, and must be addressed by a correction in the test model (so-called false positive in the test).
[0138] b. Construction of the exploration model, preferably in the form of a state model from the observations, more preferably from the observed system inputs and system outputs.
[0139] c. Updating the metrics that represent a measure of the coverage achieved in the exploration model, such as the coverage achieved of the transitions (edges) and the explored states (nodes in the model when interpreted as a directed graph) and / or possible system inputs in the respective states.
[0140] d. Identification and / or aggregation of compatible states in the exploration model with the aim of creating an exploration model with fewer states.
[0141] 16
[0142] created when originally required to represent the observations (Model Learning).
[0143] Evaluation, construction, and / or updating can take place during test execution. Identification and / or summarization can also occur post-mortem, as this subtask is computationally intensive. The disadvantage of a post-mortem solution is that certain metrics defined by states are not readily available for executing the test strategy. This can lead to decisions being made based on outdated information. A compromise is to provide identification and / or summarization with a delay, provided that the knowledge is not disproportionately outdated over sufficiently short periods, thus allowing for continued meaningful decision-making.
[0144] Determination subunit
[0145] The determination subunit selects the next system input from the test model to the technical system under test. Additionally, the determination subunit can also select at least one required parameter, preferably taking into account the chosen test strategy.
[0146] The determining subunit can continue the test as follows:
[0147] The determining subunit can initially choose from the following three options:
[0148] a. Continue testing from the current state;
[0149] b. Resetting the technical system under test and continuing the test from the initial state; or
[0150] c. Ending the test.
[0151] The decision to continue testing in its current state can be influenced by the chosen test strategy and / or the exploration goal. Examples of exploration goals include:
[0152] • The exploration model is fully explored under the given test strategy. • The exploration model is not fully explored, but no improvement can be achieved under the given test strategy. 202505094
[0153] 17
[0154] • In the current system state, there is no executable system input (deadlock).
[0155] • The resources for exploration have been exhausted, for example, the maximum execution time has been exceeded.
[0156] • The analysis of the assertions regarding the observed system outputs revealed a discrepancy.
[0157] • The allocated resources for exploration (e.g., time limit) have been exhausted.
[0158] The determination subunit can determine the input set as follows:
[0159] If the test continues, the set of possible next system inputs can be determined from the test model in the selected state. For example, conditions (constraints) can be defined in the test model for each individual input, describing when it is executable. If error-oriented tests are also to be performed, system inputs that are not possible in the current state according to the current data can also be deliberately included in the input set.
[0160] The determination subunit can select the next system input as follows:
[0161] Once the input set is determined, the next system input from this set can be selected according to the test strategy. The test strategy is preferably defined at the start of the test. Examples of test strategies are:
[0162] • All system inputs defined in the test model (possibly per compatible state) are executed at least n times.
[0163] • All two-way combinations of system inputs are executed at least once.
[0164] • Starting from a known system state that has already been reached, all executable system inputs are executed once.
[0165] • In selected states, specified undefined system inputs are executed.
[0166] The system inputs are selected randomly according to an application-specific prioritization and / or weighting of possible system inputs. 202505094
[0167] 18
[0168] • All executable system inputs up to a depth z (all possible input traces of length z) are executed.
[0169] In addition to the test strategy, the selection of the next system input can also be influenced by metrics. The selection can be controlled to prioritize previously underexplored states and / or behavioral components, and / or to further explore prioritized parts of the exploration model despite already high coverage. Furthermore, the selection of the system input can be additionally or alternatively randomized. The system input selection can be weighted to distribute the application's testing effort across relevant system components, for example, based on risk. Additionally or alternatively, the selection of the system input can be based on reinforcement learning. With reinforcement learning, the selection can be subsequently evaluated based on the actual coverage achieved (which requires the existence of a corresponding metric) (reward function).The determination subunit can therefore improve the selection of the system input when the same state is reached again.
[0170] The determination subunit can determine the input parameters as follows:
[0171] System inputs often include parameters that must be filled in for execution. The required values (data) can be provided or generated by the test strategy. The test model can define the parameter domain. Different strategies can be used to select specific data from this domain.
[0172] One possibility is to select the parameters uniformly from a defined domain, such as according to an application-specific expected distribution, or to select known test methods such as boundary tests or representatives of equivalence classes according to the representatives for the respective equivalence classes stored in the test model.
[0173] Execution sub-unit
[0174] The execution subunit executes the specified system input, preferably with the specified parameter values. Furthermore, the execution subunit can map the abstractly formulated system input in the test model to a concrete interaction.
[0175] 19
[0176] to take over the technical system to be tested, preferably via existing technical interfaces.
[0177] The executed system output can be logged in the database and / or the storage unit in order to correlate subsequent system responses and stimuli.
[0178] Set up
[0179] The database can be initially created during setup. This database comprises the test model, which defines the execution conditions (constraints) under which system inputs can be made to the technical system under test, and the test conditions (asserts) that evaluate the system outputs. Furthermore, the database includes the initial exploration model, which is initially defined only by its initial state. Alternatively, the initial exploration model can be an exploration model from a previous test run. The database also includes the metrics required for exploration.
[0180] After setup, the test unit, according to one embodiment of the invention, cyclically performs the following steps, preferably according to the MAPE-K model:
[0181] Setup (1) Determination subunit (2) Execution subunit (3) Monitoring subunit (4) Update subunit ->
[0182]
[0183] (1) (4) End.
[0184] Based on the exploration objectives and the current state of exploration using the available metrics, the determination subunit can decide whether to continue or terminate further exploration. If exploration is to be continued, the determination subunit can proceed as described above.
[0185] The execution subunit can execute the next system input determined by the determination subunit, such that the technical system under test is stimulated. Furthermore, the execution subunit can log the stimulation, thus supplementing the database.
[0186] The monitoring subunit can continuously observe the technical system and / or the system environment at the relevant interfaces. Furthermore, the monitoring subunit can store the observations in the database, making them available to the other subunits.
[0187] 20
[0188] The update subunit can evaluate the observations. Furthermore, the update subunit can perform a database update. The update can include updating the exploration model based on the observation. The update can be an extension. Additionally, the update can also include updating the metrics and / or consolidating the exploration model by reducing it to compatible states.
[0189] The computer-implemented process and the cyclical execution of the process steps are terminated as soon as the exploration objective is met. With each cycle, the exploration model becomes more detailed. This cycle continues until the exploration objective is achieved.
[0190] The state-of-the-art MAPE-K model can be used for the self-adaptive control of technical systems in unknown environments. The focus here can be on adapting and improving the control of the technical system to achieve the desired system behavior. This can be implemented during system operation.
[0191] Unlike current best practices, the test unit is preferably deployed as early as the development phase. In the sense of a "software-in-the-loop" system, the test unit takes control of the technical system under test by selecting the next system inputs. Accordingly, the test unit assumes the role of an adversary, pushing the technical system under test even to the limits of its specifications and testing it. Furthermore, learning an exploration model offers the possibility of systematically capturing system behavior and thus making it available as an additional input for the test strategy, alongside the test model.
Claims
202505094 21 Claims 1. Computer-implemented method for exploratory testing of a technical system under test (20), comprising the steps: a. Provision of a database by a test unit (10) (S1); wherein the data basis comprises a test model, an initial exploration model and at least one metric; wherein The test model defines a plurality of system inputs and corresponding system outputs; where Each system output of the majority of corresponding system outputs is a response to a system input; where The initial exploration model represents at least one system behavior; where the at least one metric defines a coverage measure over the initial exploration model; b. Determining at least one first system input from the majority of the system inputs of the test model by the test unit (10) based on the exploration model and the at least one metric using a test strategy (S2); wherein the testing strategy relates to at least one exploration target and at least one metric; wherein the test unit (10) via which at least one first system input takes over a stimulation of the technical system (20) to be tested; c. Execution of at least one initial system input on the technical system (20) to be tested by the test unit (10) (S3); d. Updating the initial exploration model taking into account at least one initial system input and at least one corresponding initial system output of the technical system to be tested (20) by the test unit (10) (S4); e. Updating at least one metric taking into account the updated exploration model by the test unit (10) (S5); f. Determining at least one second system input from the majority of the system inputs of the test model by the test unit (10) based on the updated 202505094 22 Exploration model and the updated at least one metric based on the test strategy (S6); g. Execution of at least one second system input on the technical system to be tested (20) by the test unit (10) (S7); h. Updating the updated exploration model taking into account at least one second system input and at least one corresponding second system output of the technical system under test (20) by the test unit (10) (S8); i. Updating the updated at least one metric taking into account the updated exploration model by the test unit (10) (S9); j. Termination of the iterative execution by the test unit (10) when at least one exploration goal has been achieved (S10).
2. Computer-implemented method according to claim 1, wherein the test model, the initial exploration model and / or the at least one metric of the database is stored in a volatile or non-volatile storage unit.
3. Computer-implemented method according to claim 1 or claim 2, wherein the test unit (10) implements a Monitor-Analyze-Plan-Execute over Knowledge (MAPE-K) architecture.
4. Computer-implemented method according to any of the preceding claims, wherein updating the exploration model further comprises: - Consolidating compatible states; where consolidation includes - Identifying states that produce identical system outputs in response to identical system inputs, and - Merging the identified states.
5. Computer-implemented method according to any of the preceding claims, wherein the test strategy comprises: 202505094 23 - Execute each system input of the majority of system inputs a minimum number of times; - Paired testing; - Executing all executable system inputs from a known system state; or - Executing at least one system input according to an application-specific prioritization.
6. Computer-implemented method according to any one of the preceding claims, further comprising - Provision of the updated exploration model; and / or - Generation of at least one test based on the updated exploration model.
7. Test unit (10) for carrying out the computer-implemented method according to one of the preceding claims.
8. Computer program product comprising a computer program comprising means for carrying out the method according to any one of claims 1 to 6, when the computer program is executed on a program-controlled device.