Method for verifying multiple simulation models for collision avoidance function, and system therefor

Abstract

The present document relates to a method for verifying multiple simulation models for a collision avoidance (CA) function, and a system therefor. The proposed simulation model verification method includes: performing software in the loop simulation (SILS) in which simulation of a CA functional module is performed; performing integrated SILS in which an integrated test between the CA functional module verified through the SILS and another module is performed; and performing hardware in the loop simulation (HILS) in which an integrated module verified through the integrated SILS is verified in an actual ship operating environment.

Description

Method for verifying multiple simulation models for collision avoidance function and system for the same

[0001] The following description relates to the Collision Avoidance (CA) function of a ship, specifically to a method for efficiently verifying multiple simulation models for the CA function and a system for such verification.

[0002] Conventional navigation aids for ships only simply recognize objects near the ship and have the function of displaying the recognized objects on a screen, thus providing only an alarm about obstacles to the operator of the powered ship.

[0003] However, since the driver focuses on the road ahead while driving, they do not continuously monitor the displayed information; furthermore, because the driver must personally perform actions such as flight planning and collision avoidance based on the provided information, it has not effectively provided the driver with functions for driving convenience and safe navigation.

[0004] Accordingly, there is a demand for an integrated ship navigation assistance system that provides collision avoidance (hereinafter referred to as “CA”) functions, and the applicant provides this as the HiNAS (Hyundai intelligent Navigation Assistant System).

[0005] These integrated ship navigation aids provide various CA functions, and research is needed on methods to efficiently verify multiple simulation models for each CA function.

[0006] In order to solve the problem described above, one aspect of the present invention defines various CA functions and defines a method for efficiently verifying models for each CA function in stages such as SILS (Software In the Loop Simulation) and HILS (Hardware In the Loop Simulation).

[0007] Specifically, considering the characteristics of the model for the CA function, we intend to define an Integrated SILS step between SILS and HILS to define a verification method and system that takes into account the characteristics of the CA function, which requires the application of many scenarios.

[0008] In addition, according to the embodiment, regarding SILS, the evaluation criteria are diversified by considering not only the possibility of collision in the CA function but also whether path generation is performed without problems, and multiple types of SILS according to each evaluation method are proposed.

[0009] The problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below.

[0010] In one aspect of the present invention for solving the problem described above, a method for verifying a plurality of simulation models for a Collision Avoidance (CA) function is proposed, comprising: performing a Software In the Loop Simulation (SILS) to simulate a CA function module; performing an Integrated SILS to perform an integration test between the CA function module verified through the SILS and other modules; and performing Hardware In the Loop Simulation (HILS) to verify the integrated module verified through the Integrated SILS in an actual ship operation environment.

[0011] Performing the above SILS may involve performing one or more of the following: a first type SILS performed based on a baseline scenario and a scenario reported for a bug situation, a second type SILS performed by generating a random scenario, or a third type SILS performed based on a resampled scenario.

[0012] Specifically, the first type SILS may include calculating a performance metric during a simulation based on the standard scenario; performing the second type SILS if the verification success rate based on the performance metric exceeds the verification success rate criterion of the SILS; and performing the first type SILS as a scenario reported for the bug situation if the verification success rate based on the performance metric does not exceed the verification success rate criterion of the SILS.

[0013] In addition, the second type SILS may include calculating a performance metric through a simulation based on the random scenario; determining whether a performance index satisfies a verification metric based on the performance metric; and, if the performance index satisfies a predetermined criterion, performing the integrated SILS.

[0014] In this case, the above performance indicator is defined based on whether a collision occurs during a scenario-based simulation, and the above performance index may additionally consider whether the CA function succeeded in path generation.

[0015] In addition, the above-mentioned second type SILS may additionally include performing a first type ODD (Operational Design Domain) reconstruction in consideration of a path-ungenerated scenario.

[0016] In addition, the third type SILS may apply a first sampling rate to a success scenario and apply second and third sampling rates to a failure scenario and a path-ungenerated scenario, respectively, to form the resampled scenario, wherein the second and third sampling rates may be set to be greater than the first sampling rate.

[0017] Meanwhile, the above-mentioned other module may include a TCS (Track Control System), and through the integrated SILS, an integrated test of the CA function module verified through the SILS and the TCS can be performed.

[0018] The verification success rate criterion for the above integrated SILS may be 100%, and if the verification of the above integrated SILS fails, it may include improving the CA function module; redefining the second type ODD; and resampling the scenario for the above integrated SILS.

[0019] In addition, the integrated SILS may additionally verify whether the CA function module verified through the SILS produces a result having a similarity of at least a predetermined standard in the integrated SILS.

[0020] Meanwhile, the actual ship operating environment may include FMEA (Failure Mode and Effects Analysis) and a third type ODD module, and verification of the integrated module, the FMEA module, and the third type ODD module can be performed through the HILS.

[0021] Additionally, the verification success rate criterion for the above HILS may be 100%, and if the verification of the above HILS fails, it may include improving the Ui / Ux or fallback conditions; and resampling the scenario for the above HILS.

[0022]

[0023] Meanwhile, in another aspect of the present invention for solving the problem described above, a simulation model verification system is proposed, comprising: at least one processor; at least one memory that can be operably connected to the at least one processor and stores instructions that cause the at least one processor to perform operations when executed; and a display that displays the verification status of the plurality of simulation models according to the operations, wherein the instructions include performing a Software In the Loop Simulation (SILS) that performs a simulation of a CA function module; performing an Integrated SILS that performs an integration test between the CA function module verified through the SILS and other modules; and performing a Hardware In the Loop Simulation (HILS) that verifies the integrated module verified through the Integrated SILS in an actual ship operation environment.

[0024] At this time, the above commands may include a scenario generator, a scenario simulator, and a controller to perform the SILS.

[0025] The processor may be configured to perform the SILS as one or more of a first type SILS based on a standard scenario and a scenario reported for a bug situation, a second type SILS based on a random scenario generated by the scenario generator, or a third type SILS based on a resampled scenario.

[0026] Additionally, the processor may be configured to perform the second type SILS by calculating a performance metric through the simulation of the scenario simulator based on the random scenario generated by the scenario generator; the controller determines whether the performance index satisfies a predetermined criterion based on the performance metric; and, if the performance index satisfies the predetermined criterion, perform the integrated SILS.

[0027] In addition, the processor may be configured to additionally include performing a first type ODD (Operational Design Domain) reconstruction of the second type SILS in consideration of a path-ungenerated scenario.

[0028] Meanwhile, the above-mentioned other module includes a TCS (Track Control System), and through the above-mentioned integrated SILS, an integrated test of the CA function module verified through the above-mentioned SILS and the above-mentioned TCS can be performed.

[0029] In addition, the actual ship operating environment may include FMEA (Failure Mode and Effects Analysis) and a third type ODD module, and verification of the integrated module, the FMEA module, and the third type ODD module can be performed through the HILS.

[0030] According to the embodiments of the present invention as described above, various CA functions can be efficiently verified by clearly defining models for each CA function, such as SILS, integrated SILS, and HILS, based on the definition of each CA function.

[0031] In addition, by considering the characteristics of the model for the CA function, an integrated SILS can be additionally defined between SILS and HILS to reduce the time required for simulation.

[0032] In addition, according to the embodiment, in the case of SILS, evaluation criteria are diversified by considering not only the possibility of collision in the CA function but also whether path generation is performed without problems, and multiple types of SILS according to each evaluation method are proposed so that clear verification can be performed according to the verification purpose from the user's perspective.

[0033] The effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below.

[0034] FIG. 1 is a diagram illustrating a method for defining the state of a CA function and providing a notification accordingly, in accordance with an embodiment of the present invention.

[0035] FIG. 2 is a diagram illustrating CA functions according to an embodiment of the present invention.

[0036] FIG. 3 is a diagram illustrating a method for verifying multiple simulation models for a CA function according to an embodiment of the present invention.

[0037] FIG. 4 is a diagram illustrating the structure of SILS according to one embodiment of the present invention, and FIG. 5 is a diagram illustrating a method for performing SILS according to one embodiment of the present invention.

[0038] FIG. 6 is a drawing for explaining a first type SILS according to an embodiment of the present invention.

[0039] FIG. 7 is a drawing for explaining a second type SILS according to an embodiment of the present invention.

[0040] FIGS. 8 and 9 are drawings for illustrating a third type SILS according to an embodiment of the present invention.

[0041] FIG. 10 is a diagram illustrating a method for performing integrated SILS according to an embodiment of the present invention.

[0042] FIGS. 11 and FIGS. 12 show simulation examples of integrated SILS according to one embodiment of the present invention.

[0043] FIG. 13 is a drawing for illustrating a first type integrated SILS according to one embodiment of the present invention.

[0044] FIG. 14 is a drawing for illustrating a second type integrated SILS according to one embodiment of the present invention.

[0045] FIG. 15 is a diagram illustrating a method for performing HILS according to an embodiment of the present invention.

[0046] FIGS. 16 and 17 are drawings for specifically illustrating standard HILS tests and CA requirement tests according to embodiments of the present invention.

[0047] FIG. 18 is a diagram illustrating a multiple simulation model verification system for a CA function according to one embodiment of the present invention.

[0048] Hereinafter, embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0049] Throughout the specification, when a part is described as "including" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0050]

[0051] In the software field, verification and validation may be defined as distinct processes. 'Verification' can generally be viewed as presenting objective evidence that the design results at a specific stage of the software development life cycle fully meet the clear requirements input for that stage; it is typically seen as a process that involves repeating requirements specifications, technical verification, and testing. In contrast, 'validation' can be viewed as presenting objective evidence that the software specifications and deliverables are suitable for user requirements and intended use, and that the software development consistently satisfies those requirements; this process may include verification from the perspective of actual users.

[0052] Although verification and validation as described above may be used separately depending on the case, for the sake of convenience in the following explanation, 'verification' will be referred to as a concept that encompasses both the aforementioned 'verification' and 'validation'.

[0053]

[0054] As described above, in one aspect of the present invention, various CA functions are defined, and a method for efficiently verifying a model for each CA function step by step is defined.

[0055] FIG. 1 is a diagram illustrating a method for defining the state of a CA function and providing a notification accordingly, in accordance with an embodiment of the present invention.

[0056] In one embodiment of the present invention, it is proposed to define the status of the above-described notification to include an active state (110), a ready state (120), a low performance state (130), and a failure state (140). The active state (110) may indicate a state in which the corresponding function performs an operation; the ready state (120) may indicate a state in which the corresponding function does not perform an operation but is ready to perform an operation; the low performance state (130) may indicate a state in which there is tolerable performance degradation; and the failure state (140) may indicate a case in which operation is impossible due to an untolerable event / performance degradation.

[0057] In addition, in one embodiment of the present invention, the activation (110) / prep (120) / low performance (130) / failure (140) states defined in this way are set by considering the ODD (150) / OE (160) defined by IEC as shown in FIG. 1, thereby providing ODD / OE indication information.

[0058] ODD (150) may refer to a combination of conditions in which the function can operate normally in all tolerable events. Meanwhile, OE (160) may refer to a combination of conditions that define the operational performance and limitations of the function. As illustrated in FIG. 1, a predictable event / failure may occur within the ODD (150) range and change to the OE (160) range, and may return to the ODD (150) range through recovery.

[0059]

[0060] Specifically, in one embodiment of the present invention illustrated in FIG. 1, the active state (110) and the ready state (120) are defined within the ODD (150) of the corresponding CA function, the low performance state (130) is defined outside the ODD range (150) of the corresponding CA function but within the OE (160) range of the corresponding CA function, and the failure state (140) is defined outside the OE range (160) of the corresponding CA function.

[0061] In this embodiment, a notification is provided based on any one of the activation state (110), readiness state (120), low performance state (130), and failure state (140), and for convenience, this may be referred to as a 'CA notification' below.

[0062] Meanwhile, in another embodiment of the present invention, the four states described above may be defined as ready / not ready / low performance (130) / failure (140) states.

[0063] 'Not Ready' refers to the disabled state of the relevant function, which may mean a state where there is a reason preventing the function from being activated. 'Ready' refers to the disabled state of the relevant function, which may mean a state where preparations are made to activate the function.

[0064]

[0065] FIG. 2 is a diagram illustrating CA functions according to an embodiment of the present invention.

[0066] As illustrated in FIG. 2, one embodiment of the present invention may include CAA (Collision Avoidance Assistance; 210), CAC (Collision Avoidance Control; 220), and TA (Track Control; 230) as CA functions.

[0067] CAA (210) can be defined as a function that provides a safe route to the user in situations where there is a high risk of collision. The safe route generated by CAA (210) can be temporarily provided within the XTL of the planned route.

[0068] The CAC (220) can internally control the vessel based on the safe path provided by the CAA (210) using the TC (230). In some cases, the CAC (220) may be described as a combination of the CAA (210) and the TC (230).

[0069] The state indicated by reference numeral S210 in FIG. 2 represents a state controlled by the aforementioned CAC (220). In this state S210, the navigation control system (HiNAS) by the applicant has control authority over the operation of the vessel and can perform autonomous driving along an agreed-upon safe route. This (S210) can correspond to the state in which a safe route is provided and user approval is obtained in state S220 as shown in FIG. 2, and can be changed back to state S220 when arriving at the destination of the safe route.

[0070] Table 1 below shows an example of a low-performance state of a CAC according to one embodiment of the present invention.

[0071] LOW PERFORMANCE COND.SYSTEM PARAMS.USER PARAMS.NOTES1. The safe path which CAC is following should be up to date. If CAA proposes a new safe path due to situation change, the user should accept the new safe path to avoid this condition---2. The trus or theoretical wind speed should be within 27 knots.Max wind speed.The parameter value is determined according to the vessel.3. The true or theoretical current speed should be within 3 knots.Max current speed.The parameter value is determined according to the vessel.4. The ship should be moving ahead faster or equal to telegraph position HALF AHEAD.---

[0072]

[0073] In FIG. 2, reference numeral S220 can be seen as a state where the CAA (210) and TC (230) are active. In this state (S220), the navigation control system (HiNAS) by the applicant has control authority over the operation of the vessel and follows a pre-prepared route, but can be seen as a state that displays a safe route provided by the CAA (210). In this state, if there is user approval, it can be changed to state S210 as described above. Additionally, if the CAA (210) is deactivated or fails, it can be changed to state S230. Additionally, if the TC (230) is deactivated or fails, it can be changed to state S240.

[0074] In FIG. 2, reference numeral S230 can be seen as a state where the TC (230) is activated. In this state (S230), the navigation control system (HiNAS) by the applicant has control authority over the operation of the vessel and follows a pre-prepared path, but can be seen as a state where the safe path provided by the CAA (210) is absent. In this state (S230), if the CAA (210) OE start condition is achieved and the CAA (210) is activated, the state may be changed to S220. Additionally, if the TC (230) is deactivated or fails, the state may be changed to S250.

[0075] Table 2 below shows an example of a CAA start condition, Table 3 shows an example of a CAA failure condition, and Table 4 shows an example of a CAA low-performance state.

[0076] STARTING COND.SYSTEM PARAMS.USER PARAMS.NOTES1. All internal modules of HiNAS shall not malfunction.---2. Primary sensor providing position of the own ship shall be available and functioning.---3. Primary sensor providing COG of the own ship shall be available and functioning---4. Primary sensor providing SOG of the own ship shall be available and functioning---5. At least one radar supplying target information shall be available and functioning.---6. AIS supplying target information shall be available and functioning properly.---

[0077] FAILURE COND.SYSTEM PARAMS.USER PARAMS.NOTES1. All internal modules of HiNAS shall not malfunction.---2. Primary sensor providing position of the own ship shall be available and functioning.---3. Primary sensor providing COG of the own ship shall be available and functioning---4. Primary sensor providing SOG of the own ship shall be available and functioning---5. At least one radar supplying target information shall be available and functioning.---

[0078] LOW PERFORMANCE COND.SYSTEM PARAM.USER PARAMS.NOTES1. AIS supplying target information should be available and functioning properly.---2. There should not be any sudden appearance of unexpected targets at close range-Distance limit (>3.5 NM)CAA can give unexpected behavior with sudden targets.3. There should not be any sudden appearance of unexpected targets with both low DCPA and low TCPA within certain distance range.DCPA limitTCPA limit-CAA can give unexpected behavior with sudden targets.4. The own ship should not be in turning state.ROT limit-CAA calculates the safe path based on the vessel's projected position if it maintains its current SOG and COG. Continuous change in COG can result unstable safe path generation5. The own ship should be in stable speed.Speed variation limit-CAA calculates the safe path based on the vessel's projected position if it maintains its current SOG and COG. Continuous change in SOG can result unstable safe path generation6.CAA shoul propose a new safe path on new collision risk sitation.--CAA path generation failure can imply the users may need to go through near-miss condition or exceed XTL or preplanned route for collision avoidance.7. CAA should propose a new safe path in case the vessel is no able to follow or not following the safe path, such as the vessel exceeds PDL, or course difference exceeds certain valuePDL limitCourse / BWW difference limit-CAA path generation failure can imply the users may need to go through near-miss codition or exceed XTL of preplanned route for collision avoidance.8. Dangerous targets detected along the safe path should not exceed certain number.Max number of targets along the safe path-CAA can give unexpected behavior with too many dangerous targets.

[0079]

[0080] In FIG. 2, reference numeral S240 can be seen as a state where the CAA (210) is activated. In this state (S240), the user has control over the operation of the vessel and can display the safe route provided by the CAA (210). In state (S240), if the TC (230) start condition is met and the TC (230) is activated, the state may change to S220. Additionally, if the CAA (210) is deactivated or fails in this state (S240), the state may change to S250.

[0081] As illustrated in FIG. 2, if the TC (230) is disabled in state S210 and the user confirms this, the state can be changed directly to state S240. Additionally, if a CAC (220) failure induced by a TC (230) failure occurs, the state can be changed directly to state S250. Additionally, if a CAC (220) failure induced by a CAA (210) failure occurs, and the CAA (210) is disabled and the user confirms this, the state can be changed directly to state S230.

[0082]

[0083] The ODD / OE instruction information described above in relation to FIGS. 1 and 2 can provide the user with information about the current status and mode of the CA function, and, for example, can provide specific information regarding disabled functions, performance degradation, or the OE / ODD of said function.

[0084] While the Alert filed and processed by the applicant focuses on abnormal or failure events, the ODD / OE instruction information, unlike the Alert, aims to identify the cause of the state and mode of the corresponding CA function.

[0085] For example, if the above-described CAA (210) among the CA functions cannot be activated due to a violation of the OE condition and is in a "not ready" state, the violation condition may be provided through a 'CA notification' according to the present embodiment, but in the same case, an 'alarm' may not occur.

[0086] As another example, if the aforementioned CAC (220) among the CA functions fails, a corresponding 'alarm' may be generated to the user, whereas the 'CA notification' according to the present embodiment may not explain the cause of the failure. An explanation of the failed event may be provided through the alarm, and the CA notification may provide information about the "not ready" state after the failure fallback.

[0087]

[0088] FIG. 3 is a diagram illustrating a method for verifying multiple simulation models for a CA function according to an embodiment of the present invention.

[0089] A collision avoidance system according to one embodiment of the present invention can be viewed as an integrated system including a CA module, TCS (Track Control System), alarm, ODD (Operational Design Domain), UI / UX, display, etc., as described above in relation to FIG. 2.

[0090] While such collision avoidance systems require verification across numerous scenarios, verification through real-time simulation may be time-limited, and confirmation of performance improvements may be necessary during product development. Furthermore, as multiple modules interact in a complex manner, verification of each module as well as integrated verification may be required.

[0091] Classification societies may undergo the TA (Type Approval) procedure, which may mean that the verification and validation serving as a mold for the product to the certification body is properly designed and has the reliability of a certified product, and the embodiments of the present invention described below may be viewed as a method proposed for such TA.

[0092] That is, the embodiments of the present invention described below aim to resolve time limitations by introducing a step-by-step verification method for TA and defining a step-by-step verification method.

[0093] In particular, the 'CA module,' that is, the 'collision avoidance algorithm' area, which requires a lot of simulation, is proposed to be solved through SILS (Software In the Loop Simulation; S310) as shown in Fig. 3.

[0094] According to a specific embodiment, it is proposed to devise and utilize a 'scenario generator' for simulating various situations for SILS (S310), and this will be described in detail below.

[0095] In addition, according to specific embodiments, we propose a method to design a validation index and a performance metric to distinguish and verify the validation / performance improvement of a collision avoidance system.

[0096]

[0097] In addition, as illustrated in FIG. 3, one embodiment of the present invention proposes performing an integrated SILS (Integrated SILS; S320) that performs an integration test between a CA function module verified through SILS (S310) and other modules.

[0098] Integrated SILS (S320) can be viewed as a step that verifies the interface and integrated performance through integrated testing between the verified CA module and other modules, and confirms to the classification society that the CA module used in SILS (S310) is a product-level code. Unlike the Hardware In the Loop Simulation (S330) described later, which requires operator interaction, this integrated SILS (S320) has the advantage of being able to apply faster simulations than HILS (S330) without operator interaction, and since it requires many simulations, it can conduct simulations that simulate the actual environment.

[0099] In addition, the verification process can be efficiently designed by considering the vulnerable parts of the results of the integrated SILS (S320) in the subsequent HILS (S330).

[0100]

[0101] In addition, as illustrated in FIG. 3, this embodiment may include performing HILS (S330) to verify the integrated module verified through the integrated SILS (S320) in an actual ship operation environment. That is, the HILS (S330) may be configured to conduct real-time testing through a HILS (S330) set up with equipment similar to an actual HiNAS control operation environment.

[0102] In this HILS (S330), performance verification of UI / UX (User experience / user interface), display, alert, ODD (Operational Design Domain), and CA modules can be performed.

[0103] Figure 3 illustrates the method by which the final agreed test (S340) is performed on a solid line.

[0104] Below, each step described in Fig. 3 will be explained in detail.

[0105]

[0106] SILS

[0107] FIG. 4 is a diagram illustrating the structure of SILS according to one embodiment of the present invention, and FIG. 5 is a diagram illustrating a method for performing SILS according to one embodiment of the present invention.

[0108] SILS according to one embodiment of the present invention aims to verify the CA module of a ship navigation aid such as HiNAS, and thereby to improve the CA algorithm and determine the draft of the ODD.

[0109] SILS focuses solely on the verification of the CA module, and in this sense, it can be distinguished from the integrated SILS and HILS described below. Additionally, the ODD determined through this SILS can be defined only from the perspective of the CA module, and can be defined as 'Type 1 ODD' to distinguish it from the ODD defined in the integrated SILS and HILS.

[0110] In the case of such SILS, the SILS test can be considered successful if the success rate criteria are satisfied within the range of the first type ODD defined as described above. In this case, the simulation results may differ from the product level due to the absence of TCS and UI, etc.

[0111] In this SILS, the product-level code of the CA module may be applied, and statistical analysis may be employed for verification. If the SILS test is passed, the process may proceed to the integrated SILS steps described above in relation to Fig. 3.

[0112]

[0113] Specifically, referring to FIG. 4, SILS is shown performing settings for SILS simulation in the simulation setting unit (410). FIG. 4 illustrates an environment in which CA logic is configured in the C++ language and a CA module (440) is configured as a MEX file for testing, but it is not limited to this.

[0114] Additionally, as illustrated in FIG. 4, SILS is structured to repeat each scenario N times (S410) through a scenario generator (420), a scenario simulator (430), and a dynamic controller (450), and output a final result (S460) when the termination condition is satisfied (S420).

[0115] To explain this in detail through FIG. 5, first, in SILS, a standard (baseline) SILS test can be performed (S510), and, for example, a CA algorithm (module) can be configured to perform a unit test (S520). This unit test is performed by a developer, and FIG. 4 and FIG. 5 illustrate the case where a CA module (440) that has passed this unit test is placed in a simulation environment.

[0116] After that, SILS can generate arbitrary scenarios through the scenario generator (420) and perform tests (S530), and in one example, this generation can be repeated N times. In one embodiment of the present invention, it is assumed that an arbitrary goal is established through this scenario generator (420), but the scenarios generated by the scenario generator (420) may vary depending on the type of SILS described later.

[0117] After that, SILS can repeat the application of the scenario N times to the CA module (440) by the scenario simulator (430), and through this, the dynamic controller (450) can obtain a performance metric.

[0118] After applying these N scenarios (after satisfying the scenario termination condition (S420)), in one embodiment of the present invention, it may be determined whether the verification success rate of SILS is satisfied (S540). At this time, if the verification success rate of SILS is satisfied, the process may proceed to the integrated SILS procedure (A) described below.

[0119] However, if the verification success rate of SILS is not satisfied, a resampled scenario can be tested (S550), but this can be performed selectively as needed, as indicated by the dotted line in Fig. 5.

[0120] In addition, if the verification success rate of SILS is not satisfied, in one embodiment of the present invention, it may be determined whether a change in CA logic is required (S560), and if it is determined that a change is required, an updated standard scenario may be generated by reflecting this (S570). If a change in CA logic is not required, the process proceeds to step S530 to perform an arbitrary scenario test, and the ODD may be redefined as necessary.

[0121] In addition, if necessary (B) in the integrated SILS and HILS procedures described below, it may be returned to the SILS execution step, and this will be described later with reference to FIG. 10 and FIG. 16.

[0122] Meanwhile, the result values ​​of each verification step including SILS can be classified as follows.

[0123] (1) Success: CA that collided or narrowly avoided a collision. A safe path was successfully created.

[0124] (2) Failure: CA where a collision occurred

[0125] In this case, SILS is configured to improve the CA algorithm, integrated SILS is configured to improve the CA / TCS algorithm, and HILS can be configured to improve the Alert and notification systems.

[0126] (3) NA (Path not created): When a safe path is not created

[0127] In such cases, SILS can perform ODD improvement (Target ship maneuver complexity / Min. target distance) and reconfigure ODD (TCS related - Ship specific / Dynamics) through integrated SILS.

[0128]

[0129] Accordingly, performance indicators can be defined as follows.

[0130] [Mathematical Formula 1]

[0131] Success Rate = (Success + NA) / (Success + Fail + NA)

[0132] In other words, the success rate can be viewed as representing the coverage of a predetermined algorithm.

[0133] In contrast, the performance index can be defined as follows.

[0134] [Mathematical Formula 2]

[0135] Performance Index = Success / (Success + Fail + NA)

[0136] In other words, the performance index can be defined as excluding path non-generated (NA) from the CA module.

[0137] Under the aforementioned rule, it is advisable to repeat the simulation until the number of NAs (paths not generated) converges. A non-changing number of NAs indicates a stable ODD.

[0138]

[0139] Meanwhile, in one embodiment of the present invention, performing SILS may involve performing one or more of (1) a first type SILS performed based on a standard scenario, (2) a second type SILS performed by generating a random scenario, or (3) a third type SILS performed based on a resampled scenario, and each of these will be described in detail below.

[0140] FIG. 6 is a drawing for explaining a first type SILS according to an embodiment of the present invention.

[0141] As illustrated in FIG. 6, the first type SILS can utilize standard scenarios (S610), and, for example, can generate and utilize scenarios reported for standard scenarios and bug situations, but is not limited thereto. First, if based on standard scenarios, performance metrics can be calculated and analyzed (S630) through a simulation (S620) based on standard scenarios. This analysis can be performed through quality tests, functional tests, etc. based on expert opinions, and the success rate can be calculated based on this.

[0142] If the above quality / function test is successful and the verification success rate based on the above performance indicators exceeds the verification success rate standard of SILS (100% in Fig. 6), the first type SILS can be terminated and the second type SILS described below, i.e., SILS using a scenario generator, can be performed.

[0143] However, if the above quality / function test is not successful or the verification success rate based on performance indicators does not exceed the verification success rate criteria of SILS, a measure (S650) to debug or report the corresponding scenario as a bug situation may be performed, and if such a scenario corresponds to a scenario reported as a bug situation, the first type SILS may be repeated.

[0144]

[0145] FIG. 7 is a drawing for explaining a second type SILS according to an embodiment of the present invention.

[0146] The above second type SILS assumes a type of SILS that generates a random scenario through a scenario generator (S710) and performs a simulation (S720). Figure 7 illustrates the case of generating a scenario of N = 20,000 with an SPG (spectral projected gradient) triggered, but it is not limited to this.

[0147] Performance metrics can be calculated and analyzed (S730) through simulations based on such random scenarios. In the second type SILS shown in FIG. 7, the success rate is calculated to determine whether the success rate reaches 100%, and additionally, whether the performance index satisfies the verification metric through quality testing based on expert opinions as in FIG. 6 (S740). If the performance index satisfies a predetermined standard, it is proposed to perform integrated SILS.

[0148] That is, as described above in relation to [Equation 1], the performance indicator is defined based on whether a collision occurs during a scenario-based simulation, and as described above in relation to [Equation 2], the performance index may additionally consider whether the CA function succeeded in generating a path. Therefore, the second type SILS assumes a SILS that additionally considers cases of NA (path not generated) to determine whether the criteria are satisfied at a level that core algorithm developers / experts can agree upon.

[0149] Additionally, as illustrated in FIG. 7, the second type SILS can perform improvements (S750), including updating the standard scenario, when the above-described criteria are not satisfied, and specifically, determine whether a change is required in the CA logic (S760).

[0150] If a change in the CA logic is required, the standard scenario can be updated (S770), and a standard scenario test can be performed based on this (S780). If no change in the CA logic is required, the analysis according to step S730 can be performed again.

[0151] In addition, according to the embodiment, a first type ODD reconstruction may be performed during the improvement (S750) operation described above, and the first type ODD reconstruction may include performing the first type ODD reconstruction by considering the NA (path not generated scenario) as described above.

[0152]

[0153] FIGS. 8 and 9 are drawings for illustrating a third type SILS according to an embodiment of the present invention.

[0154] As illustrated in FIGS. 8 and 9, the third type SILS assumes SILS performed based on a scenario (S810) resampled according to the target version.

[0155] That is, the third type SILS can perform simulations based on the current version (S820a) and simulations based on the target version (S820b) in parallel, and can analyze improvements by comparing with PI (Program Increment) (S830).

[0156] In one embodiment of the present invention, a first sampling rate is applied to a success scenario, and a second and third sampling rates are applied to a failure scenario and a path-non-generated scenario, respectively, to form a resampled scenario. In this embodiment, it is proposed that the second and third sampling rates be set to be greater than the first sampling rate. By applying higher sampling rates to the failure scenario and the path-non-generated scenario in this way, performance improvement can be induced more efficiently through the third type SILS.

[0157] A simulation is performed based on the resampled scenario (S820a, S820b), and whether there is performance improvement can be verified based on the performance indicator analysis (S830).

[0158] In Fig. 8, it is determined whether the quality test was passed based on the analysis results, whether the success rate reached 100%, and whether the current PI is greater than the target PI (S840). If these criteria are not met, improvement work (S880) can be performed.

[0159] Alternatively, if the criteria of step S840 are met, it may be additionally determined whether a change to the CA logic is required (S850), and if a change to the CA logic is required, a standard scenario test (S860) may be performed, and it may be determined whether to perform a standard test reflecting this change to the CA logic (S870).

[0160]

[0161] In the example illustrated in FIG. 9, the first sampling rate is determined by Cochran's formula and the finite population correction method, and the second and third sampling rates are set to 200%, but this is exemplary and is not limited thereto.

[0162] Through this third type SILS, the intended performance improvement can be verified, and necessary additional improvements (S880) can be performed.

[0163]

[0164] Integrated SILS

[0165] As mentioned above, the integrated SILS aims to perform integrated testing by including the TCS (Track Control System) as another module in addition to the CA module.

[0166] FIG. 10 is a diagram illustrating a method for performing integrated SILS according to an embodiment of the present invention.

[0167] Integrated SILS can be performed when the SILS test described above in relation to Figure 5 has passed (A).

[0168] To perform integrated SILS, a standard integrated SILS test (S1010) can be performed first. Specifically, performance indicators can be obtained by simulating scenarios for integrated SILS, and this can be performed using a simulation with a product including TCS and a mock-up.

[0169] As mentioned above, the CA function requires many simulations, and since user interaction is essential in the case of HILS, by adding only such a TCS, integrated SILS can be performed efficiently without user interaction, thereby reducing the time required for simulation.

[0170]

[0171] In addition, the integrated SILS according to the embodiment illustrated in FIG. 10 proposes to additionally perform a subset-based test (S1030) subsequently if the test is successful, depending on whether the test is successful. This subset-based test (S1030) can serve to complement the standard integrated SILS test described above.

[0172] If the subset-based test is successful, the process can proceed to the HILS procedure (C) described below, which will be described later in relation to Fig. 15.

[0173] If the subset-based test fails, it can be determined whether a change in CA logic is required (S1050), and if a change in CA logic is required, the process can be returned to the SILS procedure (B).

[0174] If no change to the CA logic is required, an additional improvement (S1060) procedure may be performed, such as in the case of failure of the standard integrated SILS, and the standard scenario may be updated accordingly.

[0175] In the above description, if a change to the CA logic is required, in one embodiment of the present invention, the step of improving the CA function module and redefining the second type ODD may be additionally performed. The second type ODD can be distinguished in that, unlike the ODD in the SILS stage which is defined only from the perspective of the CA module, it is defined to include the TCS which is the target of the integrated SILS.

[0176] Meanwhile, the standard scenario for integrated SILS can be updated through the improvement procedure (S1060), and the simulation for integrated SILS (S1010) can be repeated.

[0177]

[0178] FIGS. 11 and FIGS. 12 show simulation examples of integrated SILS according to one embodiment of the present invention.

[0179] Specifically, FIG. 11 illustrates a comparison of navigation routes when performing 20 simulations by autopilot, ferry, container (CNTR), and navigator / pilot on board (TNRK).

[0180] Meanwhile, Figure 12 shows data from a case where a simulation of a ship being operated by a Bridge Maneuvering System (BMS) was repeated 10 times.

[0181] In one embodiment of the present invention, as a type of SILS verification, it is proposed to configure the system so that the same results can be obtained in more than 90% of scenarios through integrated SILS, which can be viewed as a standard for proving to a certification body that the CA module that underwent SILS verification and the CA module applied to the product are identical.

[0182] For this purpose, one embodiment of the present invention proposes that integrated SILS also be performed by classifying it into three types, just like the SILS described above.

[0183]

[0184] FIG. 13 is a drawing for illustrating a first type integrated SILS according to one embodiment of the present invention.

[0185] The first type of integrated SILS illustrated in Fig. 13 can perform a scenario simulation (S1320) based on a standard scenario (S1310).

[0186] In one embodiment of the present invention, tests can be performed on standard scenarios and scenarios reported for defects through a first type of integrated SILS, and accordingly, can be defined as standard and defect verification tests. The standard test can verify the basic functions of the CA module by testing standard scenarios, and the defect verification test can verify whether changes have been applied as intended by using scenarios reported for defects. Through this, it can be verified whether the CA module used in SILS produces the same results through integrated SILS at the product development level.

[0187] To this end, a simulation can be automatically performed based on the above-described standard scenario and the scenario reported for defects (S1320). Specifically, by performing quality tests and functional tests and calculating the success rate, performance indicators can be obtained and an analysis can be performed (S1330).

[0188] If these quality / function tests are successful but the success rate according to the simulation does not meet the 100% standard (S1340), additional improvements (S1350) may be performed. Additionally, if a change in CA logic is required (S1360), the integrated SILS test may be terminated and the SILS level test may be returned (S1370). Additionally, if a change in CA logic is not required, the standard scenario-based integrated SILS may be repeated (S1310).

[0189]

[0190] FIG. 14 is a drawing for illustrating a second type integrated SILS according to one embodiment of the present invention.

[0191] Type 2 integrated SILS can be viewed as a scenario-based random sampling test. Specifically, Type 2 integrated SILS can be viewed as a test to verify the integration between the CA module and the TCS module, as well as the improvement of the CA algorithm and the redefinition of ODD in product-level software described above, and to confirm whether the CA module used in SILS yields similar results in integrated SILS.

[0192]

[0193] To this end, in the second type of integrated SILS, a subset scenario can be obtained (S1410) from a subset of test scenarios obtained from SILS and a simulation can be performed (S1420). Preferably, the subset of scenarios obtained from SILS can be selected from among the scenarios for which a safe path has been generated.

[0194] After that, a simulation is performed for the second type of integrated SILS (S1420), and an analysis (S1430) can be performed based on the performance indicators.

[0195] If the success rate satisfies the standard (100%) and passes the quality test (S1440), it can additionally determine whether the CA module used in SILS produces similar results in the integrated SILS (S1450). Although FIG. 14 illustrates the similarity judgment standard as 100%, it is not limited to this. If the similarity judgment result is not satisfied, an improvement procedure (S1460) can be performed, for example, by taking measures such as verifying the coupling between CA and TCS. After such improvement (S1460), the integrated SILS based on the subset scenario can be repeated (S1410).

[0196] If the success rate does not meet the standard (100%) or the quality test is not passed (S1440), an improvement procedure (S1470) including logic improvement / ODD reconfiguration may be performed, and if a CA logic change is required (S1480), the integrated SILS test may be terminated and the SILS level test may be returned (S1490). If a CA logic change is not required (S1480), the aforementioned subset scenario-based test (S1410) may be repeated.

[0197]

[0198] HILS

[0199] HILS can be viewed as an integrated verification test to ensure that the CA module operates correctly on the entire ship operation assistance system, including not only the aforementioned TCS but also the hardware configuration.

[0200] In one embodiment of the present invention, it is proposed that the actual ship operating environment additionally considered includes FMEA (Failure Mode and Effects Analysis) and a third type ODD module. The third type ODD is distinguished and referred to as such in that it is an ODD concept that includes not only the CA module and TCS but also the FMEA module, as described above.

[0201] FIG. 15 is a diagram illustrating a method for performing HILS according to an embodiment of the present invention.

[0202] As illustrated in FIG. 15, HILS execution can be performed when the integrated SILS is successful as described in FIG. 10 (C).

[0203] Specifically, HILS can perform a simulation based on standard HILS (S1510) and determine success / failure accordingly (S1520).

[0204] If the standard simulation is successful, a test according to the CA requirements can be performed (S1530) to determine whether it succeeds or fails (S1540). A difference may be that when deriving performance indicators based on these CA requirements, different performance indicators may be derived depending on the user's choice, such as path selection.

[0205] If the test according to these CA requirements fails, it may be further determined whether a change in CA logic is required (S1550). If a change in CA logic is required, it may return to the SILS / Integrated SILS stage (B). If a change in CA logic is not required, HILS may be repeated by performing improvements such as updating the standard scenario (S1560).

[0206] In the case of HILS, just like SILS and integrated SILS, tests can also be performed by classifying them into specific types of HILS depending on the purpose of the test.

[0207]

[0208] FIGS. 16 and 17 are drawings for specifically illustrating standard HILS tests and CA requirement tests according to embodiments of the present invention.

[0209] First, FIG. 16 illustrates the procedure of a standard HILS test, which may include a procedure for analyzing the results (S1630) derived by simulating the scenario (S1620) based on a standard scenario (S1610).

[0210] The success of this standard HILS can be determined based on whether the success rate has reached a predetermined standard (100%) and whether it has passed the quality test (S1640), and if the standard HILS fails, the simulation can be repeated by performing improvement work (S1650).

[0211] Next, FIG. 17 illustrates a specific procedure for testing CA requirements. To do this, an appropriate scenario for CA requirements and methods can be prepared first (S1710). Based on this scenario, CA requirements and test methods can be operated (S1720), and test items for these methods can be identified (S1730).

[0212] If testing fails for all items of the CA requirements and test methods, the CA requirements and test methods can be repeated through additional improvement (S1750) work.

[0213]

[0214] FIG. 18 is a diagram illustrating a multiple simulation model verification system for a CA function according to one embodiment of the present invention.

[0215] As illustrated in FIG. 18, the verification system may include at least one processor (1710), at least one memory (1720) which can be operably connected to the at least one processor (1710) and stores instructions that cause the at least one processor (1710) to perform operations when executed.

[0216] These commands may be commands for software processing of a method to perform verification as described above in relation to Fig. 3.

[0217] In addition, a verification system according to one embodiment of the present invention may include a display (1730) that displays the verification status of a plurality of simulation models according to these operations.

[0218] The display (1730) may be configured to display information to the user regarding the verification process described above in relation to FIGS. 4 to 17.

[0219]

[0220] The detailed description of the preferred embodiments of the present invention disclosed above is provided to enable those skilled in the art to implement and practice the present invention. Although the present invention has been described with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the scope of the invention. For example, those skilled in the art may utilize each configuration described in the above embodiments in a manner that combines with one another.

[0221] Accordingly, the present invention is not intended to be limited to the embodiments shown herein, but to be given the broadest scope consistent with the principles and novel features disclosed herein.

[0222] The method for verifying multiple simulation models for a collision avoidance function and the system for such verification according to the embodiments of the present invention described above can be utilized for verifying a CA module for navigation assistance on a ship, and can also be utilized as a standard for performing certification by a certification body.

Claims

In a method for verifying multiple simulation models for Collision Avoidance (CA) functionality, Perform SILS (Software In the Loop Simulation) to simulate the CA function module; Performing Integrated SILS to perform integration tests between the CA function module verified through the above SILS and other modules; and Includes performing HILS (Hardware In the Loop Simulation) to verify the integrated module verified through the above-mentioned integrated SILS in an actual ship operating environment, Simulation model verification method. In Article 1, Performing the above SILS is, Type 1 SILS performed based on reported scenarios for standard scenarios and bug situations, Type 2 SILS that performs by generating random scenarios, or Type 3 SILS performed based on resampled scenarios performing one or more of, Simulation model verification method. In Article 2, The above-mentioned first type SILS is, Calculate performance metrics during simulation based on the above standard scenario; If the verification success rate based on the above performance indicators exceeds the verification success rate criterion of the above SILS, the above Type 2 SILS is performed; and If the verification success rate based on the above performance indicators does not exceed the verification success rate criteria of the above SILS, the above standard scenario is used as a reported scenario for the above bug situation, including performing the above Type 1 SILS. Simulation model verification method. In Article 2, The above Type 2 SILS is, Calculate performance metrics through simulations based on the above random scenarios; Based on the above performance indicators, it is determined whether the performance index satisfies the verification indicators; and If the above performance index satisfies a predetermined criterion, the method includes performing the above integrated SILS. Simulation model verification method. In Article 4, The above performance indicators are defined based on whether a crash occurs during scenario-based simulation, and The above performance index additionally considers whether the above CA function succeeded in path generation, Simulation model verification method. In Article 4, The above Type 2 SILS is, Additionally including performing Type 1 ODD (Operational Design Domain) reconstruction considering a path non-generation scenario, Simulation model verification method. In Article 2, The above-mentioned third type SILS are, Apply the first sampling rate to the success scenario, and By applying the second and third sampling rates to the failure scenario and the path non-generation scenario, respectively, Construct the above-mentioned resampled scenario, The second and third sampling rates are set to be larger than the first sampling rate, Simulation model verification method. In Article 1, The above other module includes a TCS (Track Control System), and Through the above integrated SILS, performing an integrated test of the CA function module verified through the above SILS and the above TCS, Simulation model verification method. In Article 1, The verification success rate standard for the above integrated SILS is 100%, and If the verification of the above integrated SILS fails, Improve the above CA function module; Redefining Type 2 ODD; and including resampling scenarios for the above integrated SILS, Simulation model verification method. In Article 1, The above integrated SILS is, additionally verifying whether the CA function module verified through the above SILS produces a result having a similarity of a predetermined standard or higher in the above integrated SILS, Simulation model verification method. In Article 1, The above actual ship operating environment includes FMEA (Failure Mode and Effects Analysis) and a third-type ODD module, and Performing verification of the integrated module, the FMEA module, and the third type ODD module through the above HILS, Simulation model verification method. In Article 11, The verification success rate standard for the above HILS is 100%, and If the verification of the above HILS fails, Improve Ui / Ux or fallback conditions; and including resampling the scenario for the above HILS, Simulation model verification method. In a system for verifying multiple simulation models for Collision Avoidance (CA) functions, At least one processor; and At least one memory that can be operably connected to the at least one processor and stores instructions that cause the at least one processor to perform operations when executed; and A display that displays the verification status of the plurality of simulation models according to the above operations, The above commands are, Perform SILS (Software In the Loop Simulation) to simulate the CA function module; Performing Integrated SILS to perform integration tests between the CA function module verified through the above SILS and other modules; and Includes performing HILS (Hardware In the Loop Simulation) to verify the integrated module verified through the above-mentioned integrated SILS in an actual ship operating environment, Simulation model verification system. In Article 13, The above commands are for performing the above SILS, including a scenario generator, a scenario simulator, and a controller Simulation model verification system. In Article 14, The above processor, the above SILS, Type 1 SILS performed based on reported scenarios for standard scenarios and bug situations, A second type SILS based on a random scenario generated by the above scenario generator, or Type 3 SILS performed based on resampled scenarios Configured to perform as one or more of, Simulation model verification system. In Article 15, The above processor, the above Type 2 SILS, Calculate a performance metric through the simulation of the scenario simulator based on the random scenario generated by the scenario generator; The above controller determines whether the performance index satisfies a predetermined standard based on the above performance indicator; and When the above performance index satisfies a predetermined criterion, the system is configured to perform the integrated SILS, including performing the above. Simulation model verification system. In Article 15, The above processor, the above Type 2 SILS, Configured to additionally include performing Type 1 ODD (Operational Design Domain) reconstruction considering a path non-generation scenario, Simulation model verification system. In Article 13, The above other module includes a TCS (Track Control System), and Through the above integrated SILS, performing an integrated test of the CA function module verified through the above SILS and the above TCS, Simulation model verification system. In Article 13, The above actual ship operating environment includes FMEA (Failure Mode and Effects Analysis) and a third-type ODD module, and Performing verification of the integrated module, the FMEA module, and the third type ODD module through the above HILS, Simulation model verification system.

Images