Method and system for determining an operational domain outside an operational design domain in a system under test

Abstract

The invention relates to a computer-implemented method and system (1) for determining an operational domain (10) outside an operational design domain (ODD) in a system under test (SUT), in particular an automated driving function of a vehicle, comprising comparing a subset (18) of an aggregated current operational domain (aCOD) with a defined operational design domain (ODD) of the system under test (SUT), and determining an operational domain (10) which is outside the operational design domain (ODD) of the system under test (SUT) and in which all the test results (16) satisfy predefined test criteria.

Description

[0001] 24-046-EM DSP072S 1

[0002] Description

[0003] title

[0004] Method and system for determining an operating range of a system under test that lies outside an Operational Design Domain

[0005] The invention relates to a computer-implemented method for determining an operating range outside an Operational Design Domain of a system to be tested, in particular an automated driving function of a vehicle.

[0006] The invention further relates to a system for determining an operating range outside an Operational Design Domain of a system to be tested, in particular an automated driving function of a vehicle.

[0007] The invention further relates to a computer program product comprising a computer program and a computer-readable data carrier comprising program code of a computer program.

[0008] State of the art

[0009] The development of automated driving functions presents numerous challenges, particularly in defining and validating the conditions under which a system should operate reliably. One aspect is the Operational Design Domain (ODD), which describes specific parameters such as road types, weather conditions, traffic density, and speed. It serves 24-046-EM DSP072S 2

[0010] as a basis to assess the performance of a system under test (SUT) and to ensure that it meets the specified requirements.

[0011] In practice, the ODD is often extended iteratively. The process begins with a narrowly defined ODD covering basic functions and grows with the increasing complexity of the operating conditions. This extension requires precise testing methods, as the number and diversity of test scenarios increase with the complexity of the ODD. Automated and data-driven approaches are essential to facilitate the analysis of test data and increase the efficiency of the development processes.

[0012] Currently used testing methods, such as scenario-based testing and data replay testing, examine the SUT under realistic conditions. Scenario-based testing utilizes standardized or existing test scenarios that typically cover broader environmental areas than the defined ODD.

[0013] Data replay testing uses recorded real-world driving data that exhibits similar limitations. Both methods often result in the SUT traversing areas outside the ODD. While the tests provide valuable information about the SUT's behavior, the potential of these areas often remains untapped. Analyzing them requires manual processes that are time-consuming and prone to errors. 24-046-EM DSP072S 3

[0014] Another problem lies in the lack of differentiation between operational areas within and outside the ODD. During testing, the focus is usually only on whether the requirements of the defined ODD are met. Areas where the SUT operates successfully, even though they are not part of the ODD, are neither systematically recorded nor utilized.

[0015] These findings could help to expand the ODD in a data-driven way or to optimize the system. This is particularly valuable in iterative development processes, as continuous adaptation of the ODD forms the basis for further development.

[0016] Standards such as ISO 34503 and ASAM OpenODD provide a solid foundation for describing the ODD, but focus on the formal definition and not on the use of data outside the ODD. While existing scenario selection methods optimize ODD coverage, a systematic analysis of areas outside the defined ODD, which nevertheless offer insights into the SUT's performance, is lacking.

[0017] A weakness of the current state of the art is the limited use of test results from areas outside the ODD. These could show that the SUT functions under unforeseen conditions, which is useful in iterative development processes. Furthermore, there is a lack of automation in the analysis of these results. Manual analyses to determine operating ranges are therefore necessary.

[0018] Identifying and deriving possible extensions of the ODD are complex and error-prone.

[0019] Current best practices focus on the precise definition and testing of the ODD, but leave significant potential untapped. Automated approaches could not only increase the efficiency of testing but also improve the quality of the results. Data-driven analysis of operating environments outside the ODD would enable developers to identify new applications and further develop the SUT in a targeted manner. This could make a crucial contribution to the advancement of automated driving functions.

[0020] The object of the invention is therefore to provide a method and a system that automatically identifies operating areas of a system under test that lie outside the defined Operational Design Domain, but meet predefined test criteria.

[0021] Disclosure of the invention

[0022] The problem is solved by a computer-implemented method for determining an operating range outside an Operational Design Domain of a system to be tested, in particular an automated driving function of a vehicle with the features of claim 1.

[0023] The task is further enhanced by a system for determining an outside of an Operational Design Domain 24-046-EM DSP072S

[0024] lying operating area of ​​a system to be tested, in particular an automated driving function of a vehicle with the features of claim 13 solved.

[0025] The problem is further solved by a computer program product comprising a computer program according to claim 14 and by a computer-readable data carrier comprising program code of a computer program according to claim 15.

[0026] The invention relates to a computer-implemented method for determining an operating range outside an Operational Design Domain of a system to be tested, in particular an automated driving function of a vehicle.

[0027] The procedure comprises a continuous determination of environmental parameters of the system under test during the execution of a virtual test, whereby a current operating range of the system under test is determined, as well as an aggregation of data of the current operating range over the entire runtime of the virtual test, and a determination of test results for each journal of the virtual test, wherein the test results are linked with the data of the aggregated current operating range.

[0028] Furthermore, the procedure includes deriving a subset of the aggregated current operating area, which exclusively comprises areas, in24-046-EM DSP072S 6

[0029] in which all test results meet predefined test criteria, a comparison of the subset of the aggregated current operating domain with the defined Operational Design Domain of the system under test, and a determination of an operating domain outside the Operational Design Domain of the system under test in which all test results meet the predefined test criteria.

[0030] The invention further relates to a system for determining an operating range outside an Operational Design Domain of a system to be tested, in particular an automated driving function of a vehicle.

[0031] The system includes a first computing unit configured to continuously determine environmental parameters of the system under test during the execution of a virtual test, whereby a current operating range of the system under test can be determined.

[0032] Furthermore, the system includes a second computing unit configured to aggregate current operating range data over the entire runtime of the virtual test, and a third computing unit configured to determine test results for each journal of the virtual test, whereby the test results can be linked to the aggregated current operating range data. .24-046-EM DSP072S 7

[0033] The system also includes a fourth computing unit which is configured to derive a subset of the aggregated current operating range that includes only areas in which all test results meet predefined test criteria.

[0034] Furthermore, the system includes a comparison device configured to compare the subset of the aggregated current operating domain with the defined Operational Design Domain of the system under test, and a determination device configured to determine an operating domain outside the Operational Design Domain of the system under test in which all test results meet the specified test criteria.

[0035] The Operational Design Domain (ODD) describes the specific conditions and environmental parameters under which a system under test is expected to function correctly. This definition includes, among other things, road types, weather conditions, lighting conditions, traffic density, and speed ranges. The ODD serves as a reference framework for evaluating the performance of the system under test (SUT) and specifies the scenarios under which the system must be validated. It is essential to ensure that the SUT meets the defined requirements and can be operated safely.

[0036] Environmental parameters of the system under test include all relevant external factors that affect its functionality and 24-046-EM DSP072S 8

[0037] Factors influencing the system's performance include road types (e.g., urban, rural, or highway roads), weather conditions (e.g., rain, snow, sunshine), traffic density, lighting conditions (e.g., daylight, darkness), and the system's speed. These parameters are continuously recorded during virtual tests to enable precise analysis and evaluation of the SUT in different scenarios.

[0038] The system under test (SUT) refers to the system undergoing comprehensive validation. This is typically an automated driving function of a vehicle, tested under defined conditions. The SUT encompasses the hardware and software components required to implement the tested function. The goal of the tests is to evaluate the performance, reliability, and safety of the SUT and to identify potential weaknesses.

[0039] The Current Operational Domain (COD) refers to the current operating range in which the system under test (SUT) operates during a virtual test. This is determined by continuously acquiring environmental parameters such as road type, weather, and traffic density at each point during the test. The COD thus describes the instantaneous conditions under which the system is being tested and serves as the basis for the subsequent aggregation and analysis of the test results. 24-046-EM DSP072S 9

[0040] The Aggregated Current Operational Domain (aCOD) represents the sum of all CODs (Common Operating Domains) captured during a virtual test. It encompasses all operating domains that the system under test (SUT) traverses throughout the entire test run. The aCOD enables a comprehensive analysis of environmental parameters and test results by aggregating and structuring the data over the entire test duration. It thus forms the basis for deriving subsets and evaluating system performance.

[0041] One aspect of the present invention is to improve the accuracy and efficiency of the analysis and evaluation of test systems, particularly automated driving functions of vehicles. The method enables the precise and comprehensive identification of operating ranges outside the defined Operational Design Domain in which the tested system meets the specified test criteria. This is achieved by continuously determining the environmental parameters during the virtual test and aggregating them over the entire test duration. The data thus obtained allow for a reliable correlation of the test results with the aggregated operating ranges, thereby enabling systematic evaluation.

[0042] One effect of this method is the automated derivation of areas that the system successfully covers beyond its original specifications. The targeted delimitation of these areas in comparison to the defined ODD offers valuable insights. 24-046-EM DSP072S 10

[0043] Information for the further development and optimization of the system.

[0044] This data allows for the identification of operating areas where the system performs unexpectedly well, facilitating the planning of future tests and the adjustment of the ODD.

[0045] The combination of test criteria with aggregated operating ranges also enables efficient evaluation of large datasets generated during simulations. This significantly reduces the effort required for manual analysis and minimizes potential sources of error. Automation increases processing speed, which is particularly advantageous in iterative development processes, as results are available more quickly and development cycles can be shortened.

[0046] The described system supports these functions through specialized computing units that ensure the continuous acquisition, aggregation, and analysis of environmental parameters. Each computing unit performs a specific task, from determining the current operating range to deriving and delimiting subsets of the aggregated operating range.

[0047] The comparison tool allows for a direct comparison of the aggregated data with the defined ODD. This provides a clear visualization of the differences and enables identification. 24-046-EM DSP072S 11

[0048] additional operating areas that are not fully covered by the defined test scenarios.

[0049] This structured approach ensures consistent quality of test results. The ability to systematically identify operational areas outside the ODD opens up new perspectives for optimizing the tested systems. Developers can identify early on which areas require additional testing or which parameters should be added to the ODD. This contributes to improved safety and reliability of the tested systems.

[0050] Reducing manual intervention in the analysis processes increases the efficiency of the testing methodology. At the same time, automated processing ensures the traceability and reproducibility of the results. This characteristic is particularly important for the validation and certification of automated driving functions, as it provides an objective and data-driven basis for evaluation.

[0051] Additionally, the system enables improved resource utilization during the development process.

[0052] Detailed analysis of operational areas allows for more targeted planning of test scenarios and optimization of existing scenarios. This reduces the need for unnecessary tests and saves both time and money. Furthermore, the ability to analyze comprehensive data from virtual tests enables developers to identify potential weaknesses in the System24-046-EM DSP072S 12.

[0053] to identify problems early and take targeted measures to improve them.

[0054] The combination of continuously determining environmental parameters, aggregating the data, and systematically analyzing it makes it possible to raise the performance of automated driving functions to a new level. The method and the system contribute to making the development and validation of such systems more effective by providing valuable insights into the actual operation and performance limits of the system. This creates a solid foundation for future innovations and contributes to the further development of automated driving technologies.

[0055] According to a preferred embodiment of the invention, it is provided that the Operational Design Domain of the system to be tested is extended by the specific operating area outside the Operational Design Domain of the system to be tested, in which all test results meet the specified test criteria, or at least a sub-area thereof, in particular by generating an updated data set of the Operational Design Domain.

[0056] Extending the Operational Design Domain to include operating areas where test results meet the criteria enables data-driven adaptation and expands the system's application possibilities. This improves efficiency, as fewer iterative adjustments are required, and allows for a 24-046-EM DSP072S 13

[0057] better representation of the actual performance of the system in real-world use.

[0058] According to a preferred embodiment of the invention, it is provided that a subset of the aggregated current operating range is derived, which includes exclusively areas in which all test results do not meet the specified test criteria.

[0059] Identifying subsets where test criteria were not met helps to detect weaknesses in the system early on. This creates a basis for targeted optimizations and ensures that development resources can be focused on problematic areas.

[0060] According to a preferred embodiment of the invention, it is provided that, based on the derived areas in which all test results do not meet the specified test criteria, an adjustment, in particular of an algorithm, of the system to be tested is carried out.

[0061] Targeted algorithm adjustments based on areas where the system fails to meet test criteria lead to improved performance and reliability. This minimizes the risk of malfunctions during later use and accelerates the development process through more precise iterations.

[0062] According to a preferred embodiment of the invention, it is provided that it is determined from which test scenarios 24-046-EM DSP072S 14

[0063] The operational domain, which lies outside the Operational Design Domain of the system under test and in which all test results meet the specified test criteria, is composed of...

[0064] Determining the test scenarios that led to operating ranges outside the defined ODD with successful test results enables a targeted analysis of these scenarios. This supports the identification of parameters that are crucial for extending the ODD.

[0065] According to a preferred embodiment of the invention, the specific test scenarios are modified in such a way that they test adjacent boundary areas outside the aggregated current operating range in which all test results meet the specified test criteria.

[0066] Adapting test scenarios to cover adjacent edge areas outside the ODD promotes more comprehensive system validation. This allows for the identification of new use cases and the uncovering of potential weaknesses at the edges of operational capability.

[0067] According to a preferred embodiment of the invention, the data aggregated over the entire duration of the virtual test, covering the current operating range, are combined into a single data set. 24-046-EM DSP072S 15

[0068] Consolidating all aggregated operational data into a single dataset improves clarity and facilitates analysis. This saves time during evaluation and enables more efficient processing of test results in subsequent development steps.

[0069] According to a preferred embodiment of the invention, it is provided that the environmental parameters of the system to be tested, determined during the virtual test, are checked in a frame-based manner for each simulation time step to determine whether the test results meet the requirements of the Operational Design Domain.

[0070] Detailed verification of environmental parameters for each simulation time step ensures a precise correlation of test results to specific conditions. This enables a more granular analysis and increases the reliability of the test results.

[0071] According to a preferred embodiment of the invention, the continuous determination of the environmental parameters of the system to be tested includes parameters relating to at least one of the following categories: road type, weather conditions, traffic density, lighting conditions, and speed of the system.

[0072] The continuous recording of parameters such as road type, weather, traffic density, lighting conditions, and speed ensures a realistic representation of environmental conditions. This improves the accuracy of the simulations and the relevance of the test results for real-world applications. 24-046-EM DSP072S 16

[0073] According to a preferred embodiment of the invention, it is provided that the determination of the operating range lying outside the Operational Design Domain of the system to be tested is carried out by determining parameter ranges of the environmental parameters.

[0074] Identifying specific parameter ranges that exceed the limits of the ODD provides valuable insights into the system's performance under different conditions. This supports the planning of future tests and the optimization of the system.

[0075] According to a preferred embodiment of the invention, the results of the comparison of the subset of the aggregated current operating range with the defined Operational Design Domain of the system to be tested are provided by a data set that describes the operating ranges with the associated environmental parameters and test criteria.

[0076] Providing the results as a structured dataset enables clear documentation and traceability of the test results. Developers can use this data to make informed decisions regarding the further development and adaptation of the system.

[0077] According to a preferred embodiment of the invention, the method is applied to a plurality of test methods, including data replay testing and scenario-based testing. 24-046-EM DSP072S 17

[0078] The compatibility of the method with different testing methods, such as data replay and scenario-based testing, increases its flexibility and breadth of applicability. This allows different approaches to be combined to gain a comprehensive understanding of system performance.

[0079] The described configurations and training programs can be combined in any way desired.

[0080] Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention described previously or subsequently with regard to the exemplary embodiments that are not explicitly mentioned.

[0081] Brief description of the drawings

[0082] For a better understanding of the present invention and its advantages, reference is now made to the following description in conjunction with the associated drawings.

[0083] The invention will now be explained in more detail with reference to exemplary embodiments shown in the schematic illustrations of the drawings.

[0084] They show:

[0085] Fig. 1 shows a comparison of an Operational Design Domain, an aggregated current operating domain, and successful outcomes outside the Operational Design Domain. 24-046-EM DSP072S 18

[0086] Operating areas of a system to be tested, in particular an automated driving function of a vehicle according to a preferred

[0087] From the implementation form of the present invention;

[0088] Fig. 2 shows a schematic flowchart of an embodiment of a computer-implemented method for determining an operating range outside an Operational Design Domain of a system to be tested, in particular an automated driving function of a vehicle, according to the preferred embodiment of the present invention; and

[0089] Fig. 3 shows an exemplary illustration of a system for determining an operating range outside an Operational Design Domain of a system to be tested, in particular an automated driving function of a vehicle according to the preferred embodiment of the present invention.

[0090] Detailed description of the embodiments

[0091] In the figures of the drawings, identical reference symbols denote identical or functionally equivalent elements, parts or components, unless otherwise stated.

[0092] Fig. l shows the functional structure of the computer-implemented method for determining 24-046-EM DSP072S 19

[0093] Operating ranges that lie outside the defined Operational Design Domain (ODD) of a system under test (SUT). The diagram shows the current operating range (COD), which is captured by continuously determining environmental parameters 12 during a virtual test 14. These environmental parameters 12 include categories such as road type, weather conditions, traffic density, lighting conditions, and speed.

[0094] During the test, the COD data is aggregated over the entire runtime, resulting in the aggregated current operating area aCOD. This aggregation allows a systematic linking of the test results 16 with the corresponding aCOD data.

[0095] Figure 1 further illustrates how a subset 18 of the aCOD is derived, comprising exclusively operating ranges in which all test results 16 meet the specified test criteria. This subset 18 is then compared with the defined ODD.

[0096] The comparison reveals two areas: The first area, 19, lies within the ODD and corresponds to the system's predefined specifications. The second area, 10, lies outside the ODD and encompasses operating conditions in which the SUT nevertheless meets the test criteria. These findings enable a data-driven extension of the ODD, achieved through the creation of an updated dataset. 24-046-EM DSP072S 20

[0097] Additionally, Figure 1 highlights how areas 21 are identified where the test results 16 do not meet the test criteria. This information can be used to specifically adapt the SUT, particularly its algorithms. Furthermore, the figure illustrates how those test scenarios 20 that contributed to the successful operating areas outside the ODD are identified, and how these test scenarios 20 can be modified to specifically test adjacent boundary areas.

[0098] Finally, the diagram illustrates that all data aggregated during the tests are combined into a single dataset to facilitate analysis and traceability of the results. This comprehensive structure ensures a precise, frame-based review of the environmental parameters 12 at each simulation time step and provides the basis for a sound evaluation and optimization of the SUT.

[0099] Fig. 2 shows a computer-implemented method for determining an operating range 10 outside an Operational Design Domain (ODD) of a system under test (SUT), in particular an automated driving function of a vehicle.

[0100] The method comprises a continuous determination S1 of environmental parameters 12 of the system under test (SUT) during the execution of a virtual test 14, whereby a current operating range COD of the system under test (SUT) is determined. 24-046-EM DSP072S 21

[0101] Furthermore, the procedure includes aggregating S2 data of the current operating area over an entire runtime of the virtual test 14 and determining S3 test results 16 for each journal of the virtual test 14, whereby the test results 16 are linked with the data of the aggregated current operating area aCOD.

[0102] Furthermore, the procedure includes deriving S4 a subset 18 of the aggregated current operating area aCOD, which includes only areas in which all test results meet 16 predefined test criteria.

[0103] The procedure further includes comparing S5 of the subset 18 of the aggregated current operating domain aCOD with the defined Operational Design Domain ODD of the system under test SUT, and determining S6 an operating domain 10 outside the Operational Design Domain ODD of the system under test SUT in which all test results 16 meet the specified test criteria.

[0104] The Operational Design Domain (ODD) of the system under test (SUT) is extended by the specific operating area 10, which lies outside the Operational Design Domain (ODD) of the system under test (SUT) and in which all test results 16 meet the specified test criteria, or at least a sub-area thereof, in particular by generating an updated data set of the Operational Design Domain (ODD). .24-046-EM DSP072S 22

[0105] Furthermore, a subset 18 of the aggregated current operating area aCOD is derived, which includes only areas 21 in which all test results 16 do not meet the specified test criteria.

[0106] Based on the derived areas 21, in which all test results 16 do not meet the specified test criteria, an adjustment, in particular of an algorithm, of the system under test (SUT) is then carried out.

[0107] Furthermore, it is determined which test scenarios 20 comprise the operating area 10 lying outside the Operational Design Domain ODD of the system under test SUT, in which all test results 16 meet the specified test criteria.

[0108] The specified test scenarios 20 are further modified in such a way that they test adjacent boundary areas outside the aggregated current operating area aCOD in which all test results 16 meet the specified test criteria.

[0109] The data aggregated over the entire duration of virtual test 14 from the current COD operating area are combined into a single data set.

[0110] The environmental parameters 12 of the system under test (SUT) determined during the virtual test 14 are checked frame-based for each simulation time step to ensure that the test results 16 meet the requirements of the Operational Design Domain (ODD). 24-046-EM DSP072S 23

[0111] The continuous determination of the environmental parameters 12 of the system under test (SUT) includes parameters that relate to at least one of the following categories:

[0112] Road type, weather conditions, traffic density, lighting conditions, and system speed.

[0113] Determining the operating range 10 outside the Operational Design Domain (ODD) of the system under test (SUT) is carried out by determining parameter ranges of the environment parameters 12.

[0114] The results of the comparison of the subset 18 of the aggregated current operating domain aCOD with the defined Operational Design Domain ODD of the system under test SUT are then provided by a data set that describes the operating domains with the associated environment parameters 12 and test criteria.

[0115] The procedure is furthermore applied to a number of test methods, including data replay testing and scenario-based testing.

[0116] Fig. 3 shows an exemplary illustration of a system for determining an operating range 10 of a system SUT to be tested, in particular an automated driving function of a vehicle according to the preferred embodiment of the present invention, which lies outside an Operational Design Domain ODD.

[0117] System 1 comprises a first computing unit 22, which is configured to operate a continuous 24-046-EM DSP072S 24

[0118] Determination of environmental parameters 12 of the system under test (SUT) during the execution of a virtual test 14, wherein a current operating range (COD) of the system under test (SUT) can be determined.

[0119] Furthermore, the system 1 includes a second computing unit 24, which is configured to aggregate data of the current operating range over an entire runtime of the virtual test 14, and a third computing unit 26, which is configured to determine test results 16 for each journal of the virtual test 14, wherein the test results 16 can be linked with the data of the aggregated current operating range aCOD.

[0120] The system further includes a fourth computing unit 28, which is configured to derive a subset 18 of the aggregated current operating area aCOD, which includes only areas in which all test results 16 meet predefined test criteria.

[0121] The system further comprises a comparison device 30, which is configured to compare the subset 18 of the aggregated current operating domain aCOD with the defined Operational Design Domain ODD of the system under test (SUT), and a determination device 32, which is configured to detect a 24-046-EM DSP072S 25 outside the Operational Design Domain ODD of the system under test (SUT).

[0122] to determine operating area 10 in which all test results 16 meet the specified test criteria.

[0123] Although specific implementation forms have been illustrated and described herein, it is understandable to those skilled in the art that a multitude of alternative and / or equivalent implementations exist. It should be noted that the exemplary implementation form or forms are merely examples and are not intended to limit the scope, applicability, or configuration in any way.

[0124] Rather, the above summary and detailed description provides the person skilled in the art with convenient guidance for implementing at least one exemplary embodiment, whereby it is understood that various changes in the scope of functions and the arrangement of the elements can be made without deviating from the scope of the attached claims and their legal equivalents.

[0125] In general, this application is intended to cover changes, adaptations or variations of the forms of implementation presented herein.

[0126] For example, the sequence of process steps can be changed. Furthermore, the processes according to the invention can be carried out at least sectionally sequentially or in parallel. 24-046-EM DSP072S 26

[0127] Reference character list

[0128] 1 system

[0129] 10 Operating area outside the ODD

[0130] 12 environmental parameters

[0131] 14 virtual tests

[0132] 16 test results

[0133] 18 Aggregated subset of the operating area 19 Areas within the ODD

[0134] 20 test scenarios

[0135] 21 derived areas

[0136] 22 first calculating device

[0137] 24 second calculating device

[0138] 26 third calculating device

[0139] 28 fourth calculating unit

[0140] 30 Comparative Institution

[0141] 32 Destination facility

[0142] aCOD aggregated current operating range COD current operating range

[0143] ODD Operational Design Domain

[0144] SI - S 6 process steps

Claims

24-046-EM DSP072S Claims 1. Computer-implemented method for determining an operating range ( 10 ) of a system under test (SUT ), in particular an automated driving function of a vehicle, lying outside an Operational Design Domain (ODD ), comprising the steps: Continuous determination (Sl ) of environmental parameters ( 12 ) of the system under test (SUT) during an execution of a virtual test ( 14 ) , wherein a current operating range (COD) of the system under test (SUT) is determined; Aggregation (S2 ) of data of the current operating range (COD) over an entire runtime of the virtual test ( 14 ) ; Determine (S3) test results ( 16) for each journal of the virtual test ( 14 ), linking the test results ( 16) with the aggregated current operating area (aCOD) data; Deriving (S4 ) a subset ( 18 ) of the aggregated current operating area (aCOD) that includes only areas in which all test results ( 16) meet specified test criteria; Comparing (S5) the subset (18) of the aggregated current operating domain (aCOD) with the defined Operational Design Domain (ODD) of the system under test (SUT); and Determine (S6) an outside the Operational Design Domain (ODD) of the system under test (SUT)24-046-EM DSP072S lying operating area ( 10) in which all test results ( 16) meet the specified test criteria .

2. Computer-implemented method according to claim 1, wherein the Operational Design Domain (ODD) of the system under test (SUT) is extended by the specific operating domain (10) located outside the Operational Design Domain (ODD) of the system under test (SUT) in which all test results (16) meet the specified test criteria, or at least a sub-domain thereof, in particular by generating an updated data set of the Operational Design Domain (ODD).

3. Computer-implemented method according to claim 1 or 2, wherein a derivation of a subset ( 18 ) of the aggregated current operating range (aCOD) is performed, comprising exclusively areas (21 ) in which all test results ( 16) do not meet the specified test criteria.

4. Computer-implemented method according to claim 3, wherein, based on the derived areas (21) in which all test results (16) do not meet the specified test criteria, an adjustment, in particular of an algorithm, of the system under test (SUT) is carried out.

5. Computer-implemented method according to one of the preceding claims, wherein it is determined from which test scenarios (20) the outside the Operational Design Domain (ODD) of the device to be tested 24-046-EM DSP072S 29 Systems (SUT) operating area in which all test results ( 16) meet the specified test criteria.

6. Computer-implemented method according to claim 5, wherein the specified test scenarios (20) are modified such that they test adjacent boundary areas outside the aggregated current operating range (aCOD) in which all test results (16) meet the specified test criteria.

7. Computer-implemented method according to one of the preceding claims, wherein the data of the current operating range (COD) aggregated over the entire runtime of the virtual test ( 14 ) are combined into a single data set.

8. Computer-implemented method according to one of the preceding claims, wherein the environment parameters (12) of the system under test (SUT) determined during the virtual test (14) are checked frame-based for each simulation time step to see if the test results (16) meet the requirements of the Operational Design Domain (ODD).

9. Computer-implemented method according to any of the preceding claims, wherein the continuous determination of the environmental parameters (12) of the system under test (SUT) comprises parameters relating to at least one of the following categories: road type, weather conditions, traffic density, light conditions, and speed of the system. 24-046-EM DSP072S 10. Computer-implemented method according to one of the preceding claims, wherein the determination of the operating domain ( 10 ) lying outside the Operational Design Domain (ODD) of the system under test (SUT) is carried out by determining parameter ranges of the environment parameters ( 12 ).

11. Computer-implemented method according to one of the preceding claims, wherein the results of the comparison of the subset ( 18 ) of the aggregated current operating domain (aCOD) with the defined Operational Design Domain (ODD) of the system under test (SUT) are provided by a data set that describes the operating domains with the associated environment parameters ( 12 ) and test criteria .

12. Computer-implemented method according to one of the preceding claims, wherein the method is applied to a plurality of test methods comprising data replay testing and scenario-based testing.

13. System ( 1 ) for determining an operating range ( 10 ) outside an Operational Design Domain (ODD ) of a system under test (SUT ), in particular an automated driving function of a vehicle, comprising: a first computation unit (22) which is configured to continuously determine environmental parameters (12) of the system under test (SUT) during an execution of a-046-EM DSP072S to perform virtual tests ( 14 ) wherein a current operating range (COD) of the system under test (SUT) can be determined; a second computing unit (24) which is configured to aggregate data from the current operating range over the entire runtime of the virtual test (14); a third computing device (26) which is configured to determine test results ( 16) for each journal of the virtual test ( 14 ), wherein the test results ( 16) can be linked to the data of the aggregated current operating area (aCOD); a fourth computation unit (28) which is configured to derive a subset (18) of the aggregated current operating area (aCOD) which includes only areas in which all test results (16) meet specified test criteria; a comparison device (30) which is configured to compare the subset (18) of the aggregated current operating domain (aCOD) with the defined operational design domain (ODD) of the system under test (SUT); and a determination device (32) which is configured to determine an operating domain (10) outside the Operational Design Domain (ODD) of the system under test (SUT) in which all test results (16) meet the specified test criteria. 24-046-EM DSP072S 14. Computer program product comprising a computer program comprising software means for carrying out a method according to any one of claims 1 to 12, wherein the computer program is executed on a computer.

15. Computer-readable data carrier containing program code of a computer program for executing at least parts of a method according to any one of claims 1 to 12 when the computer program is executed on a computer.

Images