Vehicle control performance evaluation method and apparatus, storage medium, and electronic device

Abstract

A vehicle control performance evaluation method and apparatus, a storage medium, and an electronic device. The method comprises: acquiring real-vehicle classification data of a target vehicle model (S110), wherein the target vehicle model is an approved vehicle model, and evaluation dimensions at least include performance metrics; and for each performance metric under each category, calculating a first quantile value and / or a second quantile value of the real-vehicle classification data of the target vehicle model under the corresponding performance metric, using the first quantile value as an excellent performance value under the corresponding performance metric, and using the second quantile value as an anomaly warning value under the corresponding performance metric (S120). Therefore, the excellent performance value and the anomaly warning value can be used as performance evaluation criteria to perform functions such as new vehicle model approval determination and detection of an anomalous value, so that the evaluation of vehicle control performance is more objective and more accurate; in addition, no manual intervention is required throughout the entire process, thereby saving labor and improving evaluation efficiency.

Description

Methods, devices, storage media, and electronic equipment for evaluating vehicle control performance Technical Field

[0001] This application relates to the field of vehicle technology, and more specifically, to a method, apparatus, storage medium, and electronic device for evaluating vehicle control performance. Background Technology

[0002] As a key gatekeeper of vehicle performance in intelligent driving systems, the control module's accurate description of overall vehicle performance is essential. Currently, there is a lack of evaluation tools to quantify changes in control performance during control algorithm development. In most cases, evaluation relies on the subjective feelings of developers or road testers. This approach struggles to cover all scenarios, and individual subjective experiences vary significantly. Therefore, developing a more objective and accurate method for evaluating vehicle control performance is a pressing issue. Summary of the Invention

[0003] This application provides a method, apparatus, storage medium, and electronic device for evaluating vehicle control performance, which can provide a quantitative evaluation standard for vehicle control performance, making the evaluation of vehicle control performance more objective and accurate, and eliminating the need for human intervention, thus saving manpower.

[0004] The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a method for evaluating vehicle control performance, the method comprising:

[0006] Obtain real-vehicle classification data for the target vehicle model, wherein the target vehicle model is a model that has been approved for release, and the real-vehicle classification data includes data after classifying the collected raw real-vehicle data according to multiple evaluation dimensions, wherein the evaluation dimensions include at least performance indicators;

[0007] For each performance index under each category, calculate the first quantile value and / or the second quantile value of the real vehicle classification data of the target vehicle under the corresponding performance index, and take the first quantile value as the excellent performance value under the corresponding performance index, and take the second quantile value as the abnormal warning value under the corresponding performance index, wherein the first quantile is less than the second quantile.

[0008] As can be seen from the above scheme, the embodiments of this application can classify the original real vehicle data of approved models, obtain real vehicle classification data under various categories, and calculate the first quantile value and / or the second quantile value of each real vehicle classification data under the corresponding performance index for each performance index under each category. The first quantile value is used as the excellent performance value under the corresponding performance index, and the second quantile value is used as the abnormal warning value under the corresponding performance index. Thus, the excellent performance value and the abnormal warning value can be used as performance evaluation standards to perform functions such as new model approval judgment and abnormal value detection, making the evaluation of vehicle control performance more objective and accurate. Moreover, the whole process does not require manual intervention, saving manpower and improving evaluation efficiency.

[0009] In one possible implementation, the method further includes:

[0010] For each performance index under each category, a first clearance range for each performance index is determined based on the excellent performance value of any target vehicle model, and / or a second clearance range for each performance index is determined based on the average of the excellent performance values ​​of multiple target vehicles.

[0011] Obtain the actual vehicle classification data of the models to be approved for release;

[0012] Calculate the first quantile value of the actual vehicle classification data of the vehicle model to be approved for approval under the corresponding performance index;

[0013] Among all performance indicators under all categories, if all first quantile values ​​of all the models to be approved for exit are within their respective first exit ranges and / or within their respective second exit ranges, the models to be approved for exit are determined to be approved for exit; otherwise, the models to be approved for exit are determined to be unapproved for exit.

[0014] As can be seen from the above scheme, the embodiments of this application determine the first exclusion range based on the excellent performance value of any target vehicle model, and / or the second exclusion range based on the average excellent performance value of multiple target vehicles model, and judge the first quantile value of the vehicle model to be excluded. This can accurately and quickly assess whether the vehicle model to be excluded is eligible for exclusion. Furthermore, using only the second exclusion range, or combining the first and second exclusion ranges for exclusion judgment, avoids the bias caused by comparing a single vehicle model, and further improves the accuracy of exclusion judgment compared to using only the first exclusion range.

[0015] In one possible implementation, for each performance index under each category, a first abnormal value range for each performance index is determined based on the abnormal warning value of any target vehicle model, or a second abnormal value range for each performance index is determined based on the average of the abnormal warning values ​​of multiple target vehicles.

[0016] If the vehicle model to be approved for exit is determined to be a vehicle model that cannot be approved for exit, the method further includes:

[0017] Calculate the second quantile value of the actual vehicle classification data of the vehicle model to be approved for approval under the corresponding performance index;

[0018] Among all performance indicators under all categories, when the second quantile value of the vehicle model to be approved does not belong to the corresponding target outlier range, the second quantile value that does not belong to the corresponding target outlier range is determined as an outlier, and / or, it is determined that there is an outlier in the real vehicle classification data corresponding to the second quantile value that does not belong to the corresponding target outlier range, so as to optimize the vehicle model to be approved based on the outlier, wherein the target outlier range includes the first outlier range or the second outlier range.

[0019] As can be seen from the above scheme, in the case where it is determined that the model to be approved cannot be approved, the second quantile value of the model to be approved can be judged by using the first abnormal value range determined by the abnormal warning value of any target model, or the second abnormal value range determined by the average of the abnormal warning values ​​of multiple target models. This can quickly locate the abnormalities in the performance aspects of the model to be approved, thereby accelerating the optimization efficiency of the model to be approved.

[0020] In one possible implementation, for each performance index under each category, a first abnormal value range for each performance index is determined based on the abnormal warning value of any target vehicle model, or a second abnormal value range for each performance index is determined based on the average of the abnormal warning values ​​of multiple target vehicles.

[0021] The method further includes:

[0022] During the driving process of a target vehicle of any target model, acquire the real vehicle classification data of the target vehicle;

[0023] Calculate the second quantile value of the real vehicle classification data of the target vehicle under the corresponding performance index;

[0024] Among all performance indicators under all categories, when the second quantile value of the target vehicle does not belong to the corresponding target outlier range, the second quantile value that does not belong to the corresponding target outlier range is determined to be an outlier, and / or, it is determined that there is an outlier in the real vehicle classification data corresponding to the second quantile value that does not belong to the corresponding target outlier range, wherein the target outlier range includes the first outlier range or the second outlier range.

[0025] As can be seen from the above scheme, the embodiments of this application determine the first abnormal value range by the abnormal warning value of the target vehicle to which the target vehicle belongs, or the second abnormal value range by the average of the abnormal warning values ​​of multiple target vehicle models, and judge the second quantile value of the target vehicle, so as to quickly determine in which performance aspects the target vehicle has abnormalities.

[0026] In one possible implementation, the method further includes:

[0027] Calculate performance metrics according to performance statistics requirements and display the statistical results of performance metrics in a preset display format.

[0028] As can be seen from the above solution, this application embodiment displays the statistical results of performance indicators that meet the performance statistics requirements in a preset display format, allowing users to intuitively view the required performance indicator information.

[0029] Secondly, embodiments of this application provide a vehicle control performance evaluation device, the device comprising:

[0030] The acquisition unit is used to acquire real vehicle classification data of the target vehicle model, wherein the target vehicle model is a model that has been approved for release, and the real vehicle classification data includes data after classifying the collected original real vehicle data according to multiple evaluation dimensions, wherein the evaluation dimensions include at least performance indicators;

[0031] The calculation unit is used to calculate the first quantile value and / or the second quantile value of the real vehicle classification data of the target vehicle under the corresponding performance index for each performance index under each category.

[0032] The determining unit is used to take the first quantile value as the excellent performance value under the corresponding performance indicator and the second quantile value as the abnormal warning value under the corresponding performance indicator, wherein the first quantile is less than the second quantile.

[0033] In one possible implementation, the determining unit is further configured to, for each performance index under each category, determine a first clearance range for each performance index based on the excellent performance value of any target vehicle model, and / or, determine a second clearance range for each performance index based on the average of the excellent performance values ​​of multiple target vehicles.

[0034] The acquisition unit is also used to acquire the actual vehicle classification data of the vehicle model to be approved for release;

[0035] The calculation unit is also used to calculate the first quantile value of the actual vehicle classification data of the model to be approved under the corresponding performance index;

[0036] The device further includes:

[0037] The exit determination unit is used to determine the vehicle to be exited as an exitable vehicle when all the first quantile values ​​of all the vehicles to be exited are within their respective first exit ranges and / or within their respective second exit ranges, among all performance indicators under all categories; otherwise, it determines the vehicle to be exited as an unexitable vehicle.

[0038] In one possible implementation, the determining unit is further configured to determine a first abnormal value range for each performance indicator under each category, based on the abnormal warning value of any target vehicle model, or to determine a second abnormal value range for each performance indicator based on the average of the abnormal warning values ​​of multiple target vehicles.

[0039] The calculation unit is also used to calculate the second quantile value of the actual vehicle classification data of the vehicle to be approved under the corresponding performance index when the vehicle to be approved is determined to be a non-approvable vehicle.

[0040] The device further includes:

[0041] The first anomaly determination unit is used to determine the second quantile value that does not belong to the corresponding target anomaly range as an anomaly value when there is a second quantile value of the vehicle model to be approved for approval that does not belong to the corresponding target anomaly range among all performance indicators under all categories, and / or to determine that there is an anomaly value in the real vehicle classification data corresponding to the second quantile value that does not belong to the corresponding target anomaly range, so as to optimize the vehicle model to be approved for approval based on the anomaly value, wherein the target anomaly range includes the first anomaly range or the second anomaly range.

[0042] In one possible implementation, the determining unit is further configured to determine a first abnormal value range for each performance indicator under each category, based on the abnormal warning value of any target vehicle model, or to determine a second abnormal value range for each performance indicator based on the average of the abnormal warning values ​​of multiple target vehicles.

[0043] The acquisition unit is also used to acquire the real vehicle classification data of the target vehicle during the driving process of any target vehicle model.

[0044] The calculation unit is also used to calculate the second quantile value of the real vehicle classification data of the target vehicle under the corresponding performance index;

[0045] The device further includes:

[0046] The second anomaly determination unit is used to determine the second quantile value that does not belong to the corresponding target anomaly value range as an anomaly value when there is a second quantile value of the target vehicle that does not belong to the corresponding target anomaly value range among all performance indicators under all categories, and / or to determine that there is an anomaly value in the real vehicle classification data corresponding to the second quantile value that does not belong to the corresponding target anomaly value range, wherein the target anomaly value range includes the first anomaly value range or the second anomaly value range.

[0047] In one possible implementation, the device further includes:

[0048] The statistical unit is used to compile performance indicators according to performance statistics requirements;

[0049] The display unit is used to display the statistical results of performance indicators in a preset display format.

[0050] As can be seen from the above scheme, the embodiments of this application can classify the original real vehicle data of approved models, obtain real vehicle classification data under various categories, and calculate the first quantile value and / or the second quantile value of each real vehicle classification data under the corresponding performance index for each performance index under each category. The first quantile value is used as the excellent performance value under the corresponding performance index, and the second quantile value is used as the abnormal warning value under the corresponding performance index. Thus, the excellent performance value and the abnormal warning value can be used as performance evaluation standards to perform functions such as new model approval judgment and abnormal value detection, making the evaluation of vehicle control performance more objective and accurate. Moreover, the whole process does not require manual intervention, saving manpower and improving evaluation efficiency.

[0051] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any possible implementation of the first aspect.

[0052] Fourthly, embodiments of this application provide an electronic device, which includes:

[0053] One or more processors;

[0054] The processor is coupled to a storage device for storing one or more programs;

[0055] When one or more programs are executed by one or more processors, the electronic device performs the method as described in any possible implementation of the first aspect.

[0056] Fifthly, embodiments of this application provide a computer program product containing instructions that, when executed on a computer or processor, cause the computer or processor to perform the method described in any possible implementation of the first aspect. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0058] Figure 1 is a flowchart illustrating a method for evaluating vehicle control performance according to an embodiment of this application;

[0059] Figure 2 is a block diagram of a vehicle control performance evaluation device provided in an embodiment of this application;

[0060] Figure 3 is a schematic diagram of the structure of an electronic device or computer device provided in an embodiment of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0062] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0063] This application provides a method for evaluating vehicle control performance, as shown in Figure 1. This method can be applied to electronic devices or computer devices, specifically to terminals or servers. The method includes:

[0064] S110: Obtain the actual vehicle classification data of the target model.

[0065] The target vehicle model refers to a vehicle that has already been approved for release. The real-vehicle classification data includes data categorized from the collected raw real-vehicle data according to multiple evaluation dimensions, with performance indicators being at least one of these dimensions. Classification of the raw real-vehicle data can be achieved manually or automatically according to preset classification rules. The vehicle models mentioned in this application's embodiments can be classified according to the algorithm version of the autonomous driving system's control module, or they can be comprehensively classified in combination with other dimensions.

[0066] In practical applications, the collected raw vehicle data can be classified in a coarse-grained manner or in a fine-grained manner.

[0067] When using coarse-grained classification, the evaluation dimensions can include not only performance metrics but also driving functions and coarse-grained scenarios. That is, after classification, each real-vehicle classification data point represents the actual value of a performance metric for a specific driving function within a coarse-grained scenario. Coarse-grained scenarios can include typical scenarios such as automatic driving, straight driving, turning, lane changing, and braking. Driving functions can include CP (Cruise Pilot), HNP (Highway Navigation Pilot), and UNP (Urban Navigation Pilot). Performance metrics include lateral performance metrics and longitudinal performance metrics. Lateral performance metrics include lateral accuracy metrics and lateral haptic metrics. Lateral accuracy metrics include lateral position error and lateral heading error, while lateral haptic metrics include lateral acceleration and lateral jerk. Longitudinal performance metrics include longitudinal accuracy metrics and longitudinal haptic metrics. Longitudinal accuracy metrics include longitudinal position error and speed error, while longitudinal haptic metrics include longitudinal jerk. Lateral jerk refers to the derivative of lateral acceleration with respect to time during vehicle movement, i.e., the rate of change of centripetal acceleration. Longitudinal jerk refers to the rate of change of acceleration during longitudinal motion of the vehicle, and is usually used to measure the smoothness of vehicle acceleration or deceleration.

[0068] When using fine-grained classification, the evaluation dimensions can include not only performance metrics but also fine-grained scenarios. Fine-grained scenarios can include horizontal control scenarios, vertical control scenarios, and general scenarios.

[0069] Lateral control scenarios include high-speed lane changes, high-speed lane keeping, low-speed lane changes, low-speed lane keeping, low-speed intersection turns, extremely low-speed straight roads, extremely low-speed right-angle curves, extremely low-speed U-turns, and extremely low-speed obstacle avoidance. Among these, high-speed lane changes include curves with high curvature and straight roads; high-speed lane keeping includes curves with high curvature, S-shaped curves, and straight roads; low-speed lane changes include curves with high curvature and straight roads; low-speed lane keeping includes C-shaped curves with high curvature, S-shaped curves, and straight roads; and low-speed intersection turns include curves with high curvature and curves with low curvature.

[0070] Longitudinal control scenarios include three main types: downhill, flat road, and uphill. Downhill, flat road, and uphill can each be further divided into braking, starting, extremely low speed, low speed, and high speed scenarios. Among these, extremely low speed, low speed, and high speed can all include cruise control and braking scenarios.

[0071] Common scenarios can include scenarios such as takeover and automation.

[0072] When using fine-grained classification, performance indicators can include lateral position error, lateral heading error, lateral acceleration, lateral jerk, longitudinal velocity error, longitudinal position error, longitudinal jerk, etc.

[0073] It should be noted that, because with fine-grained scenario segmentation, the performance standard of the control itself should be consistent regardless of whether the scenario occurs in CP or NP functional mode. Therefore, in the case of fine-grained classification, the evaluation dimensions may no longer include driving functions.

[0074] S120: For each performance index under each category, calculate the first quantile value and / or the second quantile value of the actual vehicle classification data of the target model under the corresponding performance index, and take the first quantile value as the excellent performance value under the corresponding performance index, and take the second quantile value as the abnormal warning value under the corresponding performance index.

[0075] The first quantile is less than the second quantile. The first quantile can be the 90th quantile, any quantile between [80, 90), or any other quantile. The second quantile can be the 99th quantile, any quantile between [95, 99), or any other quantile. The specific values ​​of the two quantiles can be set according to actual needs.

[0076] The following examples illustrate the excellent performance values ​​and anomaly warning values ​​mentioned above:

[0077] As shown in Table 1, when the scenario includes automatic and straight-ahead driving, the functions include CP, HNP, and UNP, and the lateral performance indicators include lateral position error, lateral heading error, lateral acceleration, and lateral jerk, and the longitudinal performance indicators include longitudinal position error, velocity error, and longitudinal jerk, the 90th and 99th percentiles of each performance indicator under each category can be obtained, which are used as excellent performance values ​​and abnormal warning values, respectively. Here, "22.0|50.5" indicates that the excellent performance value is 22.0 and the abnormal warning value is 50.5.

[0078] Table 1

[0079] The vehicle control performance evaluation method provided in this application can classify the original real vehicle data of approved models to obtain real vehicle classification data under various categories. For each performance index under each category, it calculates the first quantile value and / or the second quantile value of each real vehicle classification data under the corresponding performance index. The first quantile value is used as the excellent performance value under the corresponding performance index, and the second quantile value is used as the abnormal warning value under the corresponding performance index. Thus, the excellent performance value and abnormal warning value can be used as performance evaluation standards to perform functions such as new model approval judgment and abnormal value detection, making the evaluation of vehicle control performance more objective and accurate. Moreover, the whole process does not require manual intervention, saving manpower and improving evaluation efficiency.

[0080] After obtaining the excellent performance values ​​and anomaly warning values ​​for each performance indicator of each target vehicle model, these values ​​can be used as standards for scenarios such as new model approval and anomaly detection during the driving process of historical models (those already approved). The following describes three application scenarios in detail:

[0081] Application Scenario 1: New Model Launch

[0082] This application allows for the determination of a first exclusion range for each performance indicator under each category, based on the excellent performance value of any target vehicle model, and / or, based on the average of the excellent performance values ​​of multiple target vehicles, to determine a second exclusion range for each performance indicator. This enables the use of the first and / or second exclusion ranges to make an exclusion judgment when a new vehicle model is pending exclusion.

[0083] The first cutoff range can be a range that includes excellent performance values. For example, it can be centered on the excellent performance value and fluctuate within a certain range (such as 20%) to obtain the first cutoff range. The second cutoff range can also be a range of the average of excellent performance values. For example, it can be centered on the excellent performance value and fluctuate within a certain range (such as 20%) to obtain the second cutoff range, or it can be directly centered on the excellent performance value as the threshold value, with values ​​less than or equal to the excellent performance value as the second cutoff range.

[0084] When a new vehicle model is pending approval, the actual vehicle classification data of the model to be approved can be obtained first, and the first quantile value of the actual vehicle classification data of the model to be approved under the corresponding performance index can be calculated. Then, among all performance indices under all categories, if all the first quantile values ​​of all models to be approved are within their respective first approval ranges, and / or are within their respective second approval ranges, the model to be approved is determined to be an approved model; otherwise, the model to be approved is determined to be an unapproved model.

[0085] When using only the first exclusion range to determine the exclusion of a new vehicle model, for each performance indicator under all categories, it is determined whether the first quantile value of the vehicle to be excluded is within the corresponding first exclusion range. If all of them are within their respective first exclusion ranges, the vehicle to be excluded is determined to be an expirable vehicle. If at least one vehicle to be excluded has a first quantile value that is not within its corresponding first exclusion range, the vehicle to be excluded is determined to be an unexpirable vehicle.

[0086] When using only the second exit range to determine the exit of a new vehicle model, for each performance indicator under all categories, it is determined whether the first quantile value of the vehicle to be exited is within the corresponding second exit range. If all of them are within their respective second exit ranges, the vehicle to be exited is determined to be an exitable vehicle. If at least one vehicle to be exited has a first quantile value that is not within the corresponding second exit range, the vehicle to be exited is determined to be an unexitable vehicle.

[0087] When determining the approval of a new vehicle model by combining the first and second approval ranges, for each performance indicator under all categories, it is first determined whether the first quantile value of the vehicle to be approved is within the corresponding first approval range. If all values ​​are within their respective first approval ranges, it is further determined whether the first quantile value of the vehicle to be approved is within the corresponding second approval range. Only when all values ​​are within their respective second approval ranges will the vehicle to be approved be determined as an approved vehicle model; otherwise, the vehicle to be approved will be determined as an unapproved vehicle model.

[0088] This application embodiment uses a first exclusion range determined by the excellent performance value of any target vehicle model, and / or a second exclusion range determined by the average excellent performance value of multiple target vehicles models, to determine the first quantile value of the vehicle model to be excluded. This allows for an accurate and rapid assessment of whether the vehicle model is eligible for exclusion. Furthermore, using only the second exclusion range, or combining the first and second exclusion ranges for exclusion determination, avoids bias caused by comparisons with a single vehicle model compared to using only the first exclusion range, further improving the accuracy of the exclusion determination.

[0089] Scenario 2: Outlier Detection

[0090] In this application embodiment, for each performance indicator under each category, a first abnormal value range for each performance indicator can be determined in advance based on the abnormal warning value of any target vehicle model, or a second abnormal value range for each performance indicator can be determined based on the average of the abnormal warning values ​​of multiple target vehicles, so as to subsequently determine whether the vehicle to be detected has any abnormalities based on the first or second abnormal value range.

[0091] The first outlier range can be a range that includes the abnormal warning values. For example, the abnormal warning values ​​can be used as the threshold values, and values ​​greater than or equal to the abnormal warning values ​​can be defined as the first outlier range. The second outlier range can be a range that includes the average of the abnormal warning values. For example, the average of the abnormal warning values ​​can be used as the threshold values, and values ​​greater than or equal to the average of the abnormal warning values ​​can be defined as the second outlier range.

[0092] During the driving process of any target vehicle of any target model, real vehicle classification data of the target vehicle is acquired, and the second quantile value of the real vehicle classification data of the target vehicle under the corresponding performance index is calculated. Among all performance indices under all categories, when there is a second quantile value of the target vehicle that does not belong to the corresponding target outlier range, the second quantile value that does not belong to the corresponding target outlier range is determined as an outlier, and / or, it is determined that there is an outlier in the real vehicle classification data corresponding to the second quantile value that does not belong to the corresponding target outlier range.

[0093] The target outlier range includes either a first outlier range or a second outlier range. That is, when using the outlier range of the target vehicle's model to perform outlier detection on the target vehicle, the referenced target outlier range is the first outlier range; when using the outlier ranges of multiple target vehicle models that include the target vehicle's model to perform outlier detection on the target vehicle, the referenced target outlier range is the second outlier range.

[0094] This application embodiment uses a first abnormal value range determined by the abnormal warning value of the target vehicle's target model, or a second abnormal value range determined by the average of the abnormal warning values ​​of multiple target models, to judge the second quantile value of the target vehicle, thereby quickly determining in which performance aspects the target vehicle has abnormalities.

[0095] Scenario 3: New Model Approval + Outlier Detection

[0096] If a vehicle model is determined to be ineligible for approval, outlier detection can be performed on the vehicle model to quickly identify the reasons for its ineligibility. This allows for prompt optimization of the vehicle model based on the detected outliers.

[0097] In this application embodiment, for each performance indicator under each category, a first abnormal value range for each performance indicator can be determined in advance based on the abnormal warning value of any target vehicle model, or a second abnormal value range for each performance indicator can be determined based on the average of the abnormal warning values ​​of multiple target vehicles, so as to subsequently determine which abnormalities exist in the unapprovable vehicle models based on the first or second abnormal value range.

[0098] When a vehicle model is determined to be ineligible for approval, the second quantile of the vehicle model's real-vehicle classification data under the corresponding performance index can be calculated first. Among all performance indices under all categories, if the second quantile of the vehicle model does not belong to the corresponding target outlier range, the second quantile that does not belong to the corresponding target outlier range is identified as an outlier. And / or, it is determined that there is an outlier in the real-vehicle classification data corresponding to the second quantile that does not belong to the corresponding target outlier range, so as to optimize the vehicle model based on the outlier.

[0099] The target outlier range includes either a first outlier range or a second outlier range. In this embodiment, when only the first exit range is used for vehicle model exit determination, or when the first and second exit ranges are combined for vehicle model exit determination, the target outlier range is the first outlier range; when only the second exit range is used for vehicle model exit determination, the target outlier range is the second outlier range.

[0100] After detecting anomalies, technicians can analyze the causes of the anomalies in a targeted manner and optimize the models to be approved for release until they meet the approval requirements.

[0101] It should be added that when the first score of the vehicle to be approved is within the range between the approval range and the abnormal value range, the performance index corresponding to the first score can be marked as a warning state so that technicians can pay close attention to the performance index and make further improvements to it.

[0102] In this embodiment of the application, when it is determined that the model to be approved cannot be approved, the second quantile value of the model to be approved can be judged by using a first abnormal value range determined by the abnormal warning value of any target model, or a second abnormal value range determined by the average of the abnormal warning values ​​of multiple target models. This can quickly locate the abnormalities in the performance aspects of the model to be approved, thereby accelerating the optimization efficiency of the model to be approved.

[0103] In one possible implementation, to facilitate technicians in visually viewing the various calculated values, such as the excellent performance value and abnormal warning value of a single vehicle model, the average excellent performance value and average abnormal warning value of multiple vehicle models, the abnormal value of any vehicle, the abnormal value of the vehicle to be approved, the performance comparison between the vehicle to be approved and the target vehicle model, the performance comparison between different target vehicle models, etc., another embodiment of this application may also provide a human-computer interaction tool, such as a Kanban board. Users can input performance statistics requirements through the human-computer interaction interface, and the human-computer interaction tool can query and statistically analyze performance index information, outputting and displaying the performance index statistics results according to a preset display format for user viewing.

[0104] The preset display formats include, but are not limited to, bar charts, scatter plots, and tables. For example, in the scenario of new vehicle model approval, the excellent performance values ​​and abnormal warning values ​​of each performance indicator of the target vehicle, as well as the excellent performance values ​​and abnormal warning values ​​of the corresponding models to be approved for the same performance indicator, can be output in tabular form. It can also show whether each performance indicator meets the requirements, i.e., whether it is within the corresponding approval range. Abnormal performance indicators can be highlighted. In addition, the performance indicators to which the abnormal values ​​belong can be categorized and summarized, for example, if the abnormality is mainly in the horizontal performance indicators.

[0105] When comparing the performance of the vehicle to be approved with that of the target vehicle, performance indicators that meet the approval range can be marked as "passed". For performance indicators that do not meet the approval range, the difference between the two can be used. When the first score of the vehicle to be approved is within the range between the approval range and the outlier range, the performance indicator corresponding to the first score can be marked as "warning status", and the performance indicator that meets the outlier range can be marked as "failed".

[0106] Based on the above method embodiments, another embodiment of this application provides a vehicle control performance evaluation device. This device can be applied to electronic devices or computer devices, specifically to a terminal or server, as shown in FIG2. The device includes:

[0107] The acquisition unit 210 is used to acquire the real vehicle classification data of the target vehicle model, wherein the target vehicle model is a model that has been approved for release, and the real vehicle classification data includes data after classifying the collected original real vehicle data according to multiple evaluation dimensions, wherein the evaluation dimensions include at least performance indicators.

[0108] The calculation unit 220 is used to calculate the first quantile value and / or the second quantile value of the real vehicle classification data of the target vehicle under the corresponding performance index for each performance index under each category.

[0109] The determining unit 230 is used to take the first quantile value as the excellent performance value under the corresponding performance index and the second quantile value as the abnormal warning value under the corresponding performance index, wherein the first quantile is less than the second quantile.

[0110] In one possible implementation, the determining unit 230 is further configured to determine a first clearance range for each performance index under each category based on the excellent performance value of any target vehicle model, and / or to determine a second clearance range for each performance index based on the average of the excellent performance values ​​of multiple target vehicles.

[0111] The acquisition unit 210 is also used to acquire the actual vehicle classification data of the vehicle model to be approved for release;

[0112] The calculation unit 220 is also used to calculate the first quantile value of the actual vehicle classification data of the model to be approved under the corresponding performance index;

[0113] The device further includes:

[0114] The exit determination unit is used to determine the vehicle to be exited as an exitable vehicle when all the first quantile values ​​of all the vehicles to be exited are within their respective first exit ranges and / or within their respective second exit ranges, among all performance indicators under all categories; otherwise, it determines the vehicle to be exited as an unexitable vehicle.

[0115] In one possible implementation, the determining unit 230 is further configured to determine a first abnormal value range for each performance indicator under each category, based on the abnormal warning value of any target vehicle model, or to determine a second abnormal value range for each performance indicator based on the average of the abnormal warning values ​​of multiple target vehicles.

[0116] The calculation unit 220 is also used to calculate the second quantile value of the actual vehicle classification data of the vehicle to be approved under the corresponding performance index when the vehicle to be approved is determined to be a non-approved vehicle.

[0117] The device further includes:

[0118] The first anomaly determination unit is used to determine the second quantile value that does not belong to the corresponding target anomaly range as an anomaly value when there is a second quantile value of the vehicle model to be approved for approval that does not belong to the corresponding target anomaly range among all performance indicators under all categories, and / or to determine that there is an anomaly value in the real vehicle classification data corresponding to the second quantile value that does not belong to the corresponding target anomaly range, so as to optimize the vehicle model to be approved for approval based on the anomaly value, wherein the target anomaly range includes the first anomaly range or the second anomaly range.

[0119] In one possible implementation, the determining unit 230 is further configured to determine a first abnormal value range for each performance indicator under each category, based on the abnormal warning value of any target vehicle model, or to determine a second abnormal value range for each performance indicator based on the average of the abnormal warning values ​​of multiple target vehicles.

[0120] The acquisition unit 210 is also used to acquire the real vehicle classification data of the target vehicle during the driving process of the target vehicle of any target model.

[0121] The calculation unit 220 is also used to calculate the second quantile value of the real vehicle classification data of the target vehicle under the corresponding performance index;

[0122] The device further includes:

[0123] The second anomaly determination unit is used to determine the second quantile value that does not belong to the corresponding target anomaly value range as an anomaly value when there is a second quantile value of the target vehicle that does not belong to the corresponding target anomaly value range among all performance indicators under all categories, and / or to determine that there is an anomaly value in the real vehicle classification data corresponding to the second quantile value that does not belong to the corresponding target anomaly value range, wherein the target anomaly value range includes the first anomaly value range or the second anomaly value range.

[0124] In one possible implementation, the device further includes:

[0125] The statistical unit is used to compile performance indicators according to performance statistics requirements;

[0126] The display unit is used to display the statistical results of performance indicators in a preset display format.

[0127] The vehicle control performance evaluation device provided in this application embodiment can classify the original real vehicle data of approved models to obtain real vehicle classification data under various categories. For each performance index under each category, it calculates the first quantile value and / or the second quantile value of each real vehicle classification data under the corresponding performance index. The first quantile value is used as the excellent performance value under the corresponding performance index, and the second quantile value is used as the abnormal warning value under the corresponding performance index. Thus, the excellent performance value and the abnormal warning value can be used as performance evaluation standards to perform functions such as new model approval judgment and abnormal value detection, making the evaluation of vehicle control performance more objective and accurate. Moreover, the whole process does not require human intervention, saving manpower and improving evaluation efficiency.

[0128] Based on the above method embodiments, another embodiment of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the above embodiments.

[0129] Based on the above method embodiments, another embodiment of this application provides an electronic device or computer device, as shown in FIG3, including:

[0130] One or more processors 310;

[0131] The processor 310 is coupled to a storage device 320, the storage device 320 being used to store one or more programs;

[0132] When the one or more programs are executed by the one or more processors 310, the electronic device or computer device performs the method as described in any of the above embodiments.

[0133] Based on the above embodiments, another embodiment of this application provides a computer program product, which includes instructions that, when executed on a computer or processor, cause the computer or processor to perform the method described in any of the above embodiments.

[0134] The above-described device and system embodiments correspond to the method embodiments and have the same technical effects. For detailed descriptions, please refer to the method embodiments. The device embodiments are derived from the method embodiments; detailed descriptions can be found in the method embodiments section, and will not be repeated here. Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0135] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for evaluating vehicle control performance, characterized in that, The method includes: Obtain real-vehicle classification data for the target vehicle model, wherein the target vehicle model is a model that has been approved for release, and the real-vehicle classification data includes data after classifying the collected raw real-vehicle data according to multiple evaluation dimensions, wherein the evaluation dimensions include at least performance indicators; For each performance index under each category, calculate the first quantile value and / or the second quantile value of the real vehicle classification data of the target vehicle under the corresponding performance index, and take the first quantile value as the excellent performance value under the corresponding performance index, and take the second quantile value as the abnormal warning value under the corresponding performance index, wherein the first quantile is less than the second quantile.

2. The method according to claim 1, characterized in that, The method further includes: For each performance index under each category, a first clearance range for each performance index is determined based on the excellent performance value of any target vehicle model, and / or a second clearance range for each performance index is determined based on the average of the excellent performance values ​​of multiple target vehicles. Obtain the actual vehicle classification data of the models to be approved for release; Calculate the first quantile value of the actual vehicle classification data of the vehicle model to be approved for approval under the corresponding performance index; Among all performance indicators under all categories, if all first quantile values ​​of all the models to be approved for exit are within their respective first exit ranges and / or within their respective second exit ranges, the models to be approved for exit are determined to be approved for exit; otherwise, the models to be approved for exit are determined to be unapproved for exit.

3. The method according to claim 2, characterized in that, For each performance indicator under each category, determine the first abnormal value range for each performance indicator based on the abnormal warning value of any target vehicle model, or determine the second abnormal value range for each performance indicator based on the average of the abnormal warning values ​​of multiple target vehicles. If the vehicle model to be approved for exit is determined to be a vehicle model that cannot be approved for exit, the method further includes: Calculate the second quantile value of the actual vehicle classification data of the vehicle model to be approved for approval under the corresponding performance index; Among all performance indicators under all categories, when the second quantile value of the vehicle model to be approved does not belong to the corresponding target outlier range, the second quantile value that does not belong to the corresponding target outlier range is determined as an outlier, and / or, it is determined that there is an outlier in the real vehicle classification data corresponding to the second quantile value that does not belong to the corresponding target outlier range, so as to optimize the vehicle model to be approved based on the outlier, wherein the target outlier range includes the first outlier range or the second outlier range.

4. The method according to claim 1, characterized in that, For each performance indicator under each category, determine the first abnormal value range for each performance indicator based on the abnormal warning value of any target vehicle model, or determine the second abnormal value range for each performance indicator based on the average of the abnormal warning values ​​of multiple target vehicles. The method further includes: During the driving process of a target vehicle of any target model, acquire the real vehicle classification data of the target vehicle; Calculate the second quantile value of the real vehicle classification data of the target vehicle under the corresponding performance index; Among all performance indicators under all categories, when the second quantile value of the target vehicle does not belong to the corresponding target outlier range, the second quantile value that does not belong to the corresponding target outlier range is determined to be an outlier, and / or, it is determined that there is an outlier in the real vehicle classification data corresponding to the second quantile value that does not belong to the corresponding target outlier range, wherein the target outlier range includes the first outlier range or the second outlier range.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Calculate performance metrics according to performance statistics requirements and display the statistical results of performance metrics in a preset display format.

6. A device for evaluating vehicle control performance, characterized in that, The device includes: The acquisition unit is used to acquire real vehicle classification data of the target vehicle model, wherein the target vehicle model is a model that has been approved for release, and the real vehicle classification data includes data after classifying the collected original real vehicle data according to multiple evaluation dimensions, wherein the evaluation dimensions include at least performance indicators; The calculation unit is used to calculate the first quantile value and / or the second quantile value of the real vehicle classification data of the target vehicle under the corresponding performance index for each performance index under each category. The determining unit is used to take the first quantile value as the excellent performance value under the corresponding performance indicator and the second quantile value as the abnormal warning value under the corresponding performance indicator, wherein the first quantile is less than the second quantile.

7. The apparatus according to claim 6, characterized in that, The determining unit is further configured to, for each performance index under each category, determine a first clearance range for each performance index based on the excellent performance value of any target vehicle, and / or, determine a second clearance range for each performance index based on the average of the excellent performance values ​​of multiple target vehicles. The acquisition unit is also used to acquire the actual vehicle classification data of the vehicle model to be approved for release; The calculation unit is also used to calculate the first quantile value of the actual vehicle classification data of the model to be approved under the corresponding performance index; The device further includes: The exit determination unit is used to determine the vehicle to be exited as an exitable vehicle when all the first quantile values ​​of all the vehicles to be exited are within their respective first exit ranges and / or within their respective second exit ranges, among all performance indicators under all categories; otherwise, it determines the vehicle to be exited as an unexitable vehicle.

8. The apparatus according to claim 7, characterized in that, The determining unit is further configured to, for each performance indicator under each category, determine a first abnormal value range for each performance indicator based on the abnormal warning value of any target vehicle model, or determine a second abnormal value range for each performance indicator based on the average of the abnormal warning values ​​of multiple target vehicles. The calculation unit is also used to calculate the second quantile value of the actual vehicle classification data of the vehicle to be approved under the corresponding performance index when the vehicle to be approved is determined to be a non-approvable vehicle. The device further includes: An anomaly determination unit is used to determine, among all performance indicators under all categories, when there is a second quantile value of the vehicle model to be approved that does not belong to the corresponding target anomaly value range, as an anomaly value, and / or to determine that there is an anomaly value in the real vehicle classification data corresponding to the second quantile value that does not belong to the corresponding target anomaly value range, so as to optimize the vehicle model to be approved based on the anomaly value, wherein the target anomaly value range includes the first anomaly value range or the second anomaly value range.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; The processor is coupled to a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the electronic device performs the method as described in any one of claims 1-5.

Images