Real-time performance and accuracy evaluation of lidar processing algorithms using HIL approach

Abstract

The present application describes a method and system for real-time performance and accuracy evaluation of LiDAR processing algorithms using Hardware-in-the-Loop (HIL) approach. It describes a method and / or system for real-time performance and accuracy evaluation of Light Detection and Ranging processing algorithms using Hardware-in-the-Loop approach comprising a Simulation server configured to replicate 3D driving scenario point clouds of a vehicle emerged in a real-world environment, said point clouds being further sent for processing on a separate hardware computing system, a Unit-Under-Testing, to implement semantic segmentation or an object detection algorithm to detect the presence of objects in the point clouds, which will be further evaluated with regard to performance by the Simulation server.

Description

Real-time performance and accuracy evaluation of LiDAR processing algorithms using HIL approach

[0001] The present application describes a method and system for real-time performance and accuracy evaluation of LiDAR processing algorithms using Hardware-in-the-Loop (HIL) approach.

[0002] The increase in sophistication and capabilities of artificial intelligence algorithms in the past few decades has led to multiple accolades and widespread adoption by academia and industry alike. At the same time, the automobile’s sensor suite has also been expanded and improved massively over the last two decades. These factors have triggered innovation in the automotive industry, which has shown a renewed interest in the development of highly- and fully autonomous vehicles.

[0003] While the benefits of Artificial Intelligence (AI) are often touted, the growing costs of dataset collection and labelling, training and inference are omitted. In the context of autonomous vehicles, the challenge of preparing large and high-quality datasets is even greater due to strict laws and regulations concerning the deployment of experimental vehicles on public roads. Therefore, it is necessary to explore new avenues to reduce the non-recurring engineering costs associated with artificial intelligence.

[0004] The increases in abundance and availability of computing hardware seen in the last decades has democratized the access and development of AI tools and models.

[0005] In a similar fashion, it has also led to the emergence of sophisticated and realistic simulators to aid in the development and testing of Advanced Driver Assistance Systems (ADAS) solutions.

[0006] The present invention describes a method for real-time performance and accuracy evaluation of Light Detection and Ranging processing algorithms using Hardware-in-the-Loop approach comprising a Simulation server configured to replicate 3D driving scenario point clouds of a vehicle emerged in a real-world environment, said point clouds being further sent for processing on a separate hardware computing system, a Unit-Under-Testing, to implement semantic segmentation or an object detection algorithm to detect the presence of objects in the point clouds, which will be further evaluated with regard to performance by the Simulation server.

[0007] In a proposed embodiment of present invention, the real-world environment comprises objects like vehicles, pedestrians, roads, buildings, trees or other features commonly existing in the surroundings of a vehicle emerged in said real-world environment.

[0008] Yet in another proposed embodiment of present invention, the replicated 3D driving scenario point clouds are originated in Data produced from a real-world sensor comprised in a Car Learning to Act simulator, said data being also priorly analyzed in terms of ground truth; and / or being tagged, with either object class information or instance information, to allow the training and validation of semantic segmentation AI algorithms comprised in the Unit-Under-Testing (UUT).

[0009] Yet in another proposed embodiment of present invention, the performance evaluation comprising an accuracy evaluation and / or an inference performance evaluation, the accuracy evaluation comprising access to both ground truth data and predictions for comparison, the inference performance evaluation comprising determining the amount of time the algorithm takes to generate the intended outputs.

[0010] The present invention also describes the system which is adapted to perform the method for real-time performance and accuracy evaluation of Light Detection and Ranging processing algorithms using Hardware-in-the-Loop approach, characterized by the simulation server comprising a Car Learning to Act simulator, a Robot Operating System middle-layer, a CARLA-ROS bridge and Operating System; and / or the Unit-Under-Testing comprising a Programable Logic module, a central processing unit, a Robot Operating System and an Operating System.

[0011] In a proposed embodiment of present invention, the Car Learning to Act simulator comprises LiDAR sensors configured to output data as Point Cloud Data, Distance Measurements, Intensity Data, Intensity Data and 2D and 3D Gridded from a surrounding world, surrounding pedestrians, surrounding weather and surrounding vehicles; and / or by a CARLA-ROS bridge enabling the virtual machine CARLA with an physical network to facilitate communication with the Robot Operating System middle-layer which comprises a collect ground truth, an accuracy evaluation, a send point cloud, a collect result and a performance evaluation.

[0012] In another embodiment of present invention, the Robot Operating System middle-layer is configured to send RAW unlabelled point clouds to the Unit-Under-Testing through a suitable communication system; and / or to collect result the processed point clouds by the Unit-Under-Testing, comparing; and / or performing a comparison between ground-truth data and data retrieved from UUT; and / or wherein the collected ground-truth enables the simultaneous evaluation of real-time performance and accuracy of the of AI inference algorithm in detection, classification, object tracking and semantic segmentation.

[0013] In another embodiment of present invention, the Unit-Under-Testing is configured to receive data produced by the simulation server which comprises the reception and pre-processing of virtual point clouds coming from the Robot Operating System middle-layer; and / or loading the AI inference algorithm in the Data Processing Unit with data from the virtual point clouds to execute a set of processing routines and generate a set of outputs; and / or collecting inference results from the programmable logic side, post-processing and sending the set of outputs to the simulation server for evaluation.

[0014] The invention still describes a data processing system, comprising the physical means necessary for the execution of the method and the system previously described. Also describes a computer program, comprising programming code or instructions suitable for carrying out the mentioned method and system, in which said computer program is stored, and is executed in a data processing system, remote or in-site, for example a server, performing the respective steps above described. Finally, it makes reference to a computer readable physical data storage device, in which the programming code or instructions of the computer program are stored.

[0015] The present application describes a method and system dor real-time performance and accuracy evaluation of LiDAR processing algorithms using Hardware-in-the-Loop (HIL) approach.

[0016] This invention is related to the development and validation of machine learning algorithms targeting Light Detection and Ranging (LiDAR) data, leveraging modern Advanced Driver Assistance Systems (ADAS) simulators to enable real-time verification without dependence on real-world data, vehicles and sensors.

[0017] The herein disclosed invention enables extensive and varied testing procedures without enabling risk to other drivers or pedestrians in case of system malfunction, targeting the evaluation of the end-to-end latency of a processing system attached to a virtual or real Light Detection and Ranging (LiDAR) sensor.

[0018] The development of LiDAR data processing algorithms is heavily compromised by the relative scarcity of large, richly annotated datasets produced with real LiDAR sensors under realistic conditions. Collecting and preparing these datasets is expensive and requires a lot of man-hours of manually annotating each sample.

[0019] Current research is heavily dependent on open-access generic ADAS datasets, which are based on older LiDAR sensor devices and annotation techniques. This limits the growth and innovation possible in the field, making it lag behind the technological advancements in LiDAR sensor technology by several years.

[0020] Modern testing procedures target only performance evaluation (i.e. inference speed) or model accuracy (correctness of predictions compared to dataset ground truth) at a time. Thus, double the efforts are required to validate the two key dimensions of a LiDAR processing algorithm.

[0021] Furthermore, these methods do not ensure that the algorithm can provide both high performance and accuracy simultaneously when deployed.

[0022] Real-time performance is typically executed with aid of a fully equipped vehicle being driven around on the open road. This approach is extremely costly, time-intensive, inflexible and scales poorly if the need to augment testing arises.

[0023] All the above-mentioned drawbacks are intended to be overcome with the present invention disclosure and solution.

[0024] To overcome the dependence on real-world data, with regard to vehicles and sensors, virtual sensor data produced in real-time by an ADAS simulator may be used instead. An ADAS simulator is a sophisticated software and / or hardware platform designed to test, develop, and validate ADAS technologies in a controlled and safe virtual environment, replicating real-world driving scenarios to evaluate the performance and effectiveness of the produced features without the need for physical road tests. The ADAS simulator will be configured adequately, such that its output data replicates the intended format and data rate characteristics of the real-world data one wishes to emulate.

[0025] The ADAS simulator is then a computer implemented application hosted on a simulation server which will simulate a virtual 3D world, with several objects within. These objects will include vehicles, pedestrians, roads, buildings, tree, etc. It is also possible to configure the environment using software APIs and / or a stand-alone editing application - i.e., to change from daytime to nighttime, spawn additional vehicles, add trees and buildings, etc.

[0026] The virtual sensor is another object, or actor, that exists within the ADAS simulator. Several of these sensors, of different types, may be present in the simulation, and perform, within the simulator, the same function that a real-world sensor performs when deployed and emerged in the real world. They translate physical attributes into digital information. For example, a virtual camera which can obtain a 2D image from inside the ADAS simulator and making it available externally, or, in the case of the proposed application, a virtual LiDAR that produces point clouds which capture the physical attributes of the surrounding environment in a point cloud. Naturally, each sensor and the ADAS simulator as a whole should be configurable enough to enable the desired driving scenarios to be explored virtually.

[0027] Data produced from a real-world sensor is not immediately suitable for training or validating AI algorithms, as it lacks any ground truth information, which is typically provided later (not in real-time) and manually, using additional sources of information (visual information obtained from vehicle-mounted cameras, most commonly) to allow for the manual assignment of 'tags' for each point in the point cloud. Each point will then be tagged, with either object class information (i.e., tree, car, person, cat, dog) and / or instance information (i.e., car1, car2, person1, car3).

[0028] This type of semantic tags allows for the training and validation of semantic segmentation AI algorithms. The training task will leverage the tagged dataset, while the validation task can use raw sensor data (without tags) and will produce adequate tags, possibly in real-time, based on the success of the previous training task.

[0029] Other types of ground truth may support different AI tasks. For example, instead of manually tagging each point with a class / instance tag, it is possible to define object bounding boxes.

[0030] In either case, when using real-world data, the process of obtaining ground truth information is often manual, labour intensive, time intensive, costly, and a major burden to AI model development and deployment, problems that the present invention makes it possible to overcome in a simple and effective way.

[0031] With the current invention a set of advantages can be mentioned, in particular:

[0032] - Does not depend on real-world datasets.

[0033] - Can be adapted to test multiple sensor configurations and experiment with different resolution and field-of-view characteristics to emulate state-of-the-art and future sensors.

[0034] - Can leverage computer hardware to enable more extensive testing procedures and conduct them in parallel and in an automated fashion (i.e., the limitations of a human driver for conducting tests do not apply here).

[0035] - Lowers the cost and resource barrier for real-time testing of LiDAR data processing algorithms.

[0036] - Can be adapted to test the quality of an autonomous driving agent relying on a LiDAR processing algorithm, without posing any risk to other vehicles or pedestrians.

[0037] In the specific use-case being tackled, within the ADAS simulator, one or more virtual vehicles will be equipped with virtual LiDAR sensors, whose characteristics will closely mimic a real-world component. In other words, the virtual LiDAR sensor will produce data that resembles the data produced by its real-world counterpart. This data produced by the virtual LiDAR could be stored in the same format as the real-world dataset. By doing so, a neural network / AI model and associated training and validation APIs could target either the virtual or the real-world dataset, without any modifications.

[0038] The key advantage in leveraging simulation tools to generate a dataset is the possibility to annotate (generate ground truth) in real time. The object class associated with each point in the LiDAR point cloud can be directly accessed within the simulator - the same holds for the object instance information, and the object bounding box. These are the most common types of ground truth information required to produce ADAS datasets which make use of LiDAR data.

[0039] As previously stated, the proposed arrangement offers cost- and time-savings when compared to traditional method of obtaining ground truth for the AI training task.

[0040] In addition, the proposed arrangement is also advantageous for the validation task associated with AI / ML validation. The validation task succeeds the training task and aims to evaluate the accuracy and inference performance of the final trained model.

[0041] To evaluate accuracy, it is necessary to have access to both ground truth data (collected previously) and predictions, generated by the trained model. By comparing the two, it is possible to quantify (using different metrics) how accurate the trained model is.

[0042] To evaluate inference performance, it is necessary to measure how long the algorithm takes to generate the intended outputs. Typically, as LiDAR sensors produce data at a specific and constant frequency, it is expected that the AI model is capable of consuming the inbound data and producing inference outputs at the same or higher cadence. These characteristics are also dependent on the underlying software and hardware architecture used to implement the AI model (i.e., the inference will be faster if a more capable / faster / more modern CPU or GPU component is used, for example).

[0043] While the accuracy evaluation task requires access to both ground truth and sensor data, the inference performance task requires only the sensor data.

[0044] In the proposed arrangement, both sensor and ground truth data are generated in real time, i.e., immediate or near immediate or without noticeable delay. Therefore, it is possible to perform the accuracy evaluation and the inference performance evaluation concurrently, in real-time, as the data is generated.

[0045] For better understanding of the present application, figures representing preferred embodiments are herein attached which, however, are not intended to limit the technique disclosed herein.

[0046] - Block diagram of proposed invention

[0047] With reference to the figures, some embodiments are now described in more detail, which are however not intended to limit the scope of the present application.

[0048] This invention discloses a real inference scenario where data is generated on a capable LiDAR and ADAS virtual environment simulator (1) and where said data is further processed on a separate computing system, a Unit-Under-Testing (UUT) (2), which is separate from the device used to generate the virtual real time data, and which comprises a full Robot Operating System (ROS) stack to act as a bridge between the two devices. Ina possible embodiment of the current invention illustrates the suggested system diagram.

[0049] The goal of the developed system is to enable Real-time performance and accuracy evaluation of LiDAR processing algorithms using a Hardware-in-the-Loop (HIL) approach.

[0050] The Hardware-in-the-Loop (HIL) approach is based on the need to validate a specific hardware component, i.e., the UUT (2), which is to be further integrated in a complex system composed of multiple hardware devices.

[0051] In the proposed HIL approach, a validation engineer has access to the UUT (2), but not to the rest of the system which directly interfaces with it - either because the rest of the system was not developed yet, or because such a testing setup is unfeasible or too costly to produce. The engineer chooses to simulate the rest of the system using software, and interface it with the UUT (2) using the actual ports that would have been used in the final solution. Thus, the UUT (2) is enveloped by a virtualized system (1) that realistically replicates the actual final deployment scenario, allowing for functional and nonfunctional validation of the UUT.

[0052] The complex environment which directly interfaces with the UUT (2) is being simulated using a computer implemented software tool, identified as "capable LIDAR / ADAS virtual environment simulator".

[0053] The role of the simulation server (1) is to provide an interface to the UUT (2) that replicates a real world deployment scenario - in this specific case, the simulator (1) generates virtual LiDAR sensor data in the exact format and cadence expected to arrive at the UUT (2), generated within a virtualized world that resembles an urban or rural driving scenario in accordance with the target application of the UUT (2). This virtualized world has other objects and agents which affect the LiDAR data, i.e., they are visible to the sensor and appear in the LiDAR point cloud. The UUT (2) is expected to implement some sort of semantic segmentation or object detection algorithm, such that it can accurately detect the presence of these other agents in the virtual driving environment.

[0054] As one can tell by the described scenario, replicating this in the real-world would be far too costly and time-consuming as it would require the UUT (2) to be installed in a complete vehicle with an adequate sensor setup, a driver, access to public roads for testing, nearby pedestrians or other agents of interest, etc….

[0055] The simulation server (1) role is then to replace all the real and physical hardware setup with an equivalent virtualized environment and interface with the UUT (2). This simulation server (1) can be implemented, in a non-limiting manner, resorting to the use of a conventional desktop computer or a laptop, all the way up to a supercomputer, depending on the degree of realism intended and the resulting computational burden imposed.

[0056] The UUT (2) is noted as "separate computing system". Within the HIL approach, the UUT (2) is typically a hardware component, although it may also include some necessary firmware or software to perform a specific task or application. The actual hardware might not be particularly relevant in some of the use cases since the major relevance is related with the role that the computing system performs in the proposed arrangement. It should provide a common interface for data exchange with other system components, or the virtualized world analysis results produced by the UUT (2), in an automotive context. In addition, the UUT (2) it is expected to receive raw LiDAR data in a specific format and cadence and should be capable of processing this data with an AI / ML algorithm that can either detect objects in the LiDAR point cloud, assign object / instance tags to each point, or perform other functions of interest of the data. The output of this task must be communicated to external components.

[0057] In a final deployment, a central autonomous driving computer might be configured to make decisions, actuating accelerator, brakes and steering controls, etc., which will require the UUT data outputs in its decision-making process. In the proposed HIL framework, it is merely necessary to ensure that the UUT (2) is producing data at the expected quality (inference accuracy) and cadence (inference performance).

[0058] In a possible embodiment of the current invention, it is disclosed a Simulation Server (1). The simulation server (1) comprises a Car Learning to Act simulator (11), a Robot Operating System middle-layer (12), a CARLA-ROS bridge (13) and Operating System (14). Car Learning to Act simulator (11) may comprise LiDAR sensors (111) configured to output data as Point Cloud Data, Distance Measurements, Intensity Data, Intensity Data and 2D and 3D Gridded. This LiDAR output data might be related with surrounding world (1111), surrounding pedestrians (1112), surrounding weather (1113) and surrounding vehicles (1114). The CARLA-ROS bridge (13) enables the virtual machine CARLA (11) to the physical network, facilitating communication between the machine and the external network, enabling network traffic to flow in and out of the virtual environment. This connection is performed with the Robot Operating System middle-layer (12), which comprises a collect ground truth (121), an accuracy evaluation (122), a send point cloud (123), a collect result (125) and a performance evaluation (124). The OS (14) is responsible for controlling all the hardware and software related tasks and modules of the simulation server (1) previously mentioned.

[0059] In a possible embodiment of the current invention, it is disclosed a Unit-Under-Testing(2). The Unit-Under-Testing (2) comprises a Programable Logic module (21), a central processing unit (22), a Robot Operating System (23) and an Operating System (24) whose main performing tasks will be described in the following paragraphs.

[0060] The disclosed and developed CARLA Simulator (11), comprised in the simulation server (1), provides the necessary virtual LiDAR data (111), which, together with a collected ground-truth (121), enable the simultaneous evaluation of real-time performance and accuracy. This ground-truth (121) relates to the type of AI inference algorithm (2111) under evaluation, and natively, this setup can serve object detection and classification, object tracking and semantic segmentation workloads. This is enabled by the fact that these AI inference algorithms (2111) depend on object characteristics which are readily available within the simulation engine (1) and can be easily exported.

[0061] For interaction between the simulated environment (1) and the Unit-Under-Testing (2), a ROS middle-layer (12) is used. This ROS middle-layer (12) provides a programmable interface, allowing external applications to interact with the virtual sensors and actors inside the CARLA environment (11). ROS middle-layer (12) is a software component, hosted on the simulation server (1). It is a key element to enable the proposed HIL setup.

[0062] In what regards to the 'virtual actors', they represent a specific class of virtual object which is capable of performing actions within the simulation and interact with other actors. For instance, a tree inside the virtual environment is a static object. It performs no actions or notable interactions. However, a pedestrian (1112) is a virtual actor, and it can move throughout the simulated environment, and its movement might be impaired by the presence of other existing actors like for example other virtual pedestrians (1112), or red traffic light, or a vehicle in its path. Another example of virtual actors in the simulated environment includes vehicles (1114), traffic lights and in a sense it can be understood that virtual sensors are also virtual actors. This is because their position and behaviour is also affected by the position of other actors.

[0063] By allowing external actors to influence the simulated environment, the ROS layer (12) is a key-component in enabling the desired closed-feedback loop for evaluating the AI algorithm (2111) under testing. The AI algorithm (2111) is a software component 'hosted' on the UUT (2) hardware device. The UUT (2), in one of the preferred embodiments of the current invention can be performed by a GPU / CPU / FPGA or other adequate type of hardware component, which includes a software stack that enables the deployment of the AI / ML algorithm, responsible for processing incoming LiDAR data and making outputs available externally for validation or for use by an hypothetical autonomous driving agent responsible for high-level decision making.

[0064] The most objective / direct way of evaluating the UUT (2) that is by comparing, in real-time, the expected outputs vs the actual outputs of the UUT (2). Are they accurate? Are they being produced at the expected rate? As previously detailed in the "problem / solution" section, the proposed solution as a whole provides cost and time benefits.

[0065] The invention is a system composed by two core components, the simulator server (1) and the UUT (2). The adoption of HIL methodology in the context of ADAS development, more specifically, the validation of AI / ML inference on LiDAR data, is a groundbreaking and a very valuable approach. HIL approach is often seen in low-level closed-loop, safety-critical control systems which depend on a multitude of external components, such that procuring a real-world validation platform is unfeasible. There are clear parallels to that scenario and the one of ADAS development. Both algorithms, SW and HW components are of great complexity, interdependence, and the opportunities for real-world testing are limited by legal / moral reasons, besides cost and time. Even with infinite time and money, it is not possible to cover every single imaginable scenario on public roads (i.e., how to study system behaviour in case of accident? In different weather conditions? Etc).

[0066] Based on the above, the proposed design relies on a multi-threaded software application, which performs the following tasks concurrently:

[0067] 1) collection of ground-truth LiDAR data (111), in the form of time-stamped object bounding boxes and labelled point clouds based on internal simulation engine parameters;

[0068] 2) transmission (123) of RAW unlabelled point clouds (150) to the Unit-Under-Testing (2), which could for example be an FPGA SoC device, via a suitable communication system like for example Ethernet, CAN, UART or an alternative industry-standard interface;

[0069] 3) retrieval (224) of classification results and latency measurements (160) from the UUT (2);

[0070] 4) comparison (122) between ground-truth data (121) and data retrieved (125) from UUT (2).

[0071] The output of this final comparison stage (122) is a quantified accuracy and / or performance measurement, which can be presented to the user or stored in an external memory device.

[0072] The proposed arrangement is an adaptation of the typical HIL testing and validation methodology applied in the context of ADAS development and validation. The developed system and arrangement can provide real-time accuracy and performance metrics of the UUT (2).

[0073] These metrics are valuable to the user responsible for designing and validating the UUT (2). Thus, somehow the accuracy / performance data must be made available to the user in some form, either through graphical display, or by storing the data in some application-specific file format in a local and / or remote and / or external memory device, for instance, to allow the user to analyse the data on a different system than the one used to host the proposed arrangement, i.e., the simulation server (1).

[0074] The simulation server (1), besides enabling the continuous operation of the simulated environment, is also responsible of enabling the interface (for example, by adopting a middle-layer like ROS) between the simulated environment and the UUT (2). In addition, it is also responsible for producing the desired accuracy and / or performance metrics in real-time.

[0075] This way, it is clear that the main tasks of the simulation server are:

[0076] 1 - produce a simulated environment, complete with vehicles, sensors, pedestrians, etc…

[0077] 2 - maintain an interface between the aforementioned environment and the UUT (2).

[0078] 3 - produce accuracy / performance metrics.

[0079] It is important that the simulation server (1) covers these tasks, so that the UUT (2) is not burdened with them. This way, the UUT (2) performs in the proposed arrangement as it would in a final deployment on an end-product.

[0080] In a proposed embodiment, it should support an embedded software stack, including a ROS middle-layer for interaction with its counterpart on the simulation server.

[0081] The UUT (2), which in abstract is a hardware-software device, is internally responsible for executing the following tasks concurrently:

[0082] 1) receive data produced by the simulation server (1), which in detail comprises the reception (221) and pre-processing (222) of virtual point clouds (150) coming from an external interface (12);

[0083] 2) loading the AI model’s (2111) accelerator (211) with data, executing several processing routines and launching the inference procedure generating 'output products';

[0084] 3) collecting inference results from the programmable logic side, post-processing (223) and sending (224) the 'output products' back to the simulation server (1) for evaluation (122).

[0085] The UUT (2) comprises several built-in components, one of which is a dedicated resource that increases the inference procedure accuracy or speed, this is the Data Processing Unit (211).

[0086] Acting as a co-processor, the DPU (211) speeds up part of the workload of the main central processing unit (22), which is responsible for the remaining overall operations.

[0087] In that sense, it is this main processing unit (22) of the specific UUT (2) being described that is responsible for the step 1), 2) and 3) mentioned in the preceding paragraph.

[0088] More specifically, in task 2), is described in a high-level approach how the DPU is engaged and leveraged to implement the AI inference task (2111), by receiving LiDAR data [be it raw, or pre-processed as in 1)] and produces inference outputs which are communicated externally, as described in 3).

[0089] In this context, the UUT (2) leverages a DPU (211) because AI inference (2111) is a computationally demanding task, which cannot be accomplished with the desired accuracy / performance metrics on other system components (CPU). Thus, the DPU (221) is employed to accelerate the task. In addition, the total latency of the system (i.e., the time interval starting from the moment raw point cloud data enters the device until the corresponding AI inference results are sent to the simulation server) should be measured, preferably from the simulation server side (1) using software resource, so it can also be evaluated.

[0090] The herein proposed method and system for real-time performance and accuracy evaluation of LiDAR processing algorithms using Hardware-in-the-Loop (HIL) approach, in one of the proposed embodiments of the invention, may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer or a programmable hardware component being operable with accordingly adapted software. In other words, the herein described method is then executed on one or more programmable hardware components. Such hardware components may comprise a general-purpose processor, a Digital Signal Processor (DSP), a micro-controller, etc.

[0091] 1 – simulation server;

[0092] 11 – Car Learning to Act – CARLA;

[0093] 111 – LiDAR data;

[0094] 1111 – surrounding world data;

[0095] 1112 – surrounding pedestrians’ data;

[0096] 1113 – surrounding weather data;

[0097] 1114 – surrounding vehicles data;

[0098] 12 - Robot Operating System – ROS;

[0099] 121 – collect ground-truth;

[0100] 122 – Accuracy evaluation;

[0101] 123 – sent point cloud;

[0102] 124 – performance evaluation;

[0103] 125 – collect result;

[0104] 150 – point cloud;

[0105] 160 – processed point cloud;

[0106] 13 – CARLA-ROS bridge;

[0107] 14 – Operating System;

[0108] 2 – Unit-Under-Testing (UUT);

[0109] 21 – Programable logic;

[0110] 211 - Data Processing Unit (DPU);

[0111] 2111 – Artificial Intelligence (AI) Inference;

[0112] 22 – Central processing unit;

[0113] 221 – receive point cloud;

[0114] 222 – pre-processing;

[0115] 223 – post-processing;

[0116] 224 – send result;

[0117] 23 - Robot Operating System – ROS;

[0118] 24 - Operating System;

Claims

Method for real-time performance and accuracy evaluation of Light Detection and Ranging processing algorithms using Hardware-in-the-Loop approach comprising a Simulation server (1) configured to replicate 3D driving scenario point clouds of a vehicle emerged in a real-world environment, said point clouds being further sent for processing on a separate hardware computing system, a Unit-Under-Testing (2), to implement semantic segmentation or an object detection algorithm to detect the presence of objects in the point clouds, which will be further evaluated with regard to performance by the Simulation server (1).Method according to the previous claim, characterized by the real-world environment comprising objects like vehicles, pedestrians, roads, buildings, trees or other features commonly existing in the surroundings of a vehicle emerged in said real-world environment.Method according to the previous claim, characterized by the replicated 3D driving scenario point clouds being originated in Data produced from a real-world sensor comprised in a Car Learning to Act simulator (11), said data being also priorly analyzed in terms of ground truth; and / or being tagged, with either object class information or instance information, to allow the training and validation of semantic segmentation AI algorithms comprised in the Unit-Under-Testing (UUT) (2).Method according to the previous claim, characterized by the performance evaluation comprising an accuracy evaluation and / or an inference performance evaluation, the accuracy evaluation comprising access to both ground truth data and predictions for comparison, the inference performance evaluation comprising determining the amount of time the algorithm takes to generate the intended outputs.System adapted to perform the method for real-time performance and accuracy evaluation of Light Detection and Ranging processing algorithms using Hardware-in-the-Loop approach, characterized by the simulation server (1) comprising a Car Learning to Act simulator (11), a Robot Operating System middle-layer (12), a CARLA-ROS bridge (13) and Operating System (14); and / or the Unit-Under-Testing (2) comprising a Programable Logic module (21), a central processing unit (22), a Robot Operating System (23) and an Operating System (24).System according to any of the previous claims, characterized by the Car Learning to Act simulator (11) comprise LiDAR sensors (111) configured to output data as Point Cloud Data, Distance Measurements, Intensity Data, Intensity Data and 2D and 3D Gridded from a surrounding world (1111), surrounding pedestrians (1112), surrounding weather (1113) and surrounding vehicles (1114); and / or by a CARLA-ROS bridge (13) enabling the virtual machine CARLA (11) with an physical network to facilitate communication with the Robot Operating System middle-layer (12) which comprises a collect ground truth (121), an accuracy evaluation (122), a send (123) point cloud, a collect result (125) and a performance evaluation (124).System according to any of the previous claims, characterized by the Robot Operating System middle-layer (12) being configured to send (123) RAW unlabelled point clouds (150) to the Unit-Under-Testing (2) through a suitable communication system; and / or to collect result (125) the processed point clouds (160) by the Unit-Under-Testing (2), comparing; and / or performing a comparison (122) between ground-truth data (121) and data retrieved (125) from UUT (2); and / or wherein the collected ground-truth (121) enables the simultaneous evaluation of real-time performance and accuracy of the of AI inference algorithm (2111) in detection, classification, object tracking and semantic segmentation.System according to any of the previous claims, characterized by the Unit-Under-Testing (2) being configured to receive data produced by the simulation server (1) which comprises the reception (221) and pre-processing (222) of virtual point clouds (150) coming from the Robot Operating System middle-layer (12); and / or loading the AI inference algorithm (2111) in the Data Processing Unit (211) with data from the virtual point clouds (150) to execute a set of processing routines and generate a set of outputs; and / or collecting inference results from the programmable logic side, post-processing (223) and sending (224) the set of outputs to the simulation server (1) for evaluation (122).Data processing system, characterized by comprising the physical means necessary for the execution of the method and the system described in any one of the preceding claims.Computer program, characterized by comprising programming code or instructions suitable for carrying out the method and the system indicated in preceding claims, in which said computer program is stored, and is executed in a data processing system, remote or in-site, for example a server, performing the respective steps described in the claims.Computer readable physical data storage device, in which the programming code or instructions of the computer program are stored.

Images