Model training method, planning and control information acquisition method, and device

Abstract

A model training method, a planning and control information acquisition method, and a device, applied to the field of intelligent driving. The planning and control information acquisition method comprises: acquiring training data comprising a training sample and desired planning and control information, the training sample comprising driving style information and environment information, the driving style information indicating a driving style of a vehicle, and the desired planning and control information matching the driving style indicated by the driving style information; inputting the training sample into a first model, and obtaining predicted planning and control information by means of the first model; and using a loss function to train the first model, the loss function indicating the similarity between the predicted planning and control information and the desired planning and control information, so that the first model learns the capability of combining the driving style information and the environment information to obtain planning and control information matching the driving style indicated by the driving style information, which facilitates obtaining, on the basis of a user-defined driving style and environment information, planning and control information matching the user-defined driving style, so as to meet the personalized requirements of a user.

Description

A model training method, a method for acquiring regulatory information, and equipment. Technical Field

[0001] This application relates to intelligent driving technology, and more particularly to a model training method, a method for acquiring regulatory information, and a device. Background Technology

[0002] Intelligent driving technology is currently one of the most focused research areas. Early intelligent driving mainly adopted rule-based methods, but these methods have limited capabilities and cannot handle the many complex traffic scenarios encountered by vehicles. With the development of artificial intelligence (AI) technology, end-to-end intelligent driving solutions have gradually become the research frontier in the field. End-to-end intelligent driving solutions can not only reduce the maintenance difficulty of intelligent driving systems, but also handle many complex traffic scenarios that rule-based methods cannot solve.

[0003] Specifically, end-to-end intelligent driving refers to designing a machine learning model, inputting environmental information into the machine learning model, and obtaining the regulatory information output by the machine learning model. However, since the input of the machine learning model only includes environmental information, the regulatory information output by the machine learning models in different vehicles is highly similar and cannot meet the personalized needs of users. Summary of the Invention

[0004] This application provides a model training method, a method for acquiring regulation and control information, and a device, which facilitates the acquisition of regulation and control information that matches the user's customized driving style based on the user's customized driving style and environmental information, thereby meeting the user's personalized needs and greatly optimizing the user's riding experience.

[0005] This application provides the following technical solution:

[0006] Firstly, this application provides a model training method applicable to the field of intelligent driving. In this method, a training device acquires training data (hereinafter referred to as "first training data" for ease of distinction), which includes training samples (hereinafter referred to as "first training samples") and expected regulatory control information corresponding to the first training samples. The first training samples include first driving style information and environmental information. The training device inputs the first training samples into a first model and obtains the predicted regulatory control information corresponding to the first training samples through the first model. Then, a loss function is used to train the first model. The loss function indicates the similarity between the predicted regulatory control information and the expected regulatory control information. The goal of training with the loss function includes improving the similarity between the predicted regulatory control information and the expected regulatory control information.

[0007] The first driving style information indicates the driving style of the vehicle, and the expected regulation information matches the driving style indicated by the first driving style information. For example, the expected regulation information can also be understood as the regulation information adopted by the vehicle to drive in accordance with the driving style indicated by the first driving style information in the traffic environment indicated by the environmental information in the first training sample (optionally, also including the vehicle state indicated by the first state information). The expected regulation information can also be replaced by the correct regulation information.

[0008] The expected control information and the predicted control information contain the same type of information. For example, both the expected control information and the predicted control information include at least one of the following: the planned location of the vehicle, the driving strategy of the vehicle, or the control information of the vehicle, etc.

[0009] For example, the environmental information may include at least one image and / or point cloud data of the traffic environment surrounding the vehicle. The at least one image of the traffic environment surrounding the vehicle may be acquired by a first sensor in the vehicle, and the at least one image may be an image from a perspective view (PV). The point cloud data of the traffic environment surrounding the vehicle may be acquired by a second sensor in the vehicle. The "traffic environment surrounding the vehicle" can be understood as the environment within the field of view of the sensors in the vehicle (such as the aforementioned first or second sensor).

[0010] This application provides a training method for a first model. The input of the first model includes not only environmental information but also driving style information indicating the driving style of the vehicle. When training the first model using training data, the expected regulatory information in the training data matches the driving style indicated by the driving style information. This enables the first model to learn the ability to combine driving style information and environmental information to obtain regulatory information that matches the driving style indicated by the driving style information. This is beneficial for obtaining regulatory information that matches the user-defined driving style based on user-defined driving style and environmental information, thereby meeting the user's personalized needs and greatly optimizing the user's riding experience.

[0011] In one possible implementation, the first training sample also includes the vehicle's state information (hereinafter referred to as "first state information" for ease of distinction). For example, the vehicle's first state information can be understood as the vehicle's current state information, which can reflect the vehicle's true physical state. The vehicle's first state information may include at least one of the following: velocity, acceleration, yaw rate, steering angle, pitch angle, or other physical characteristic information of the vehicle.

[0012] The training device inputs the first training sample into the first model and obtains the predictive control information corresponding to the training sample through the first model, including: the training device extracts features from the first driving style information to obtain the first feature, and extracts features from the first state information of the vehicle to obtain the second feature; based on the environmental information, the first feature and the second feature, the predictive control information corresponding to the training sample is obtained through the first model.

[0013] In this implementation, not only are the first driving style information and environmental information input into the first model, but the first state information of the vehicle is also input into the first model. Since the control information is used to control the vehicle, and the vehicle starts to drive according to the plan indicated by the control information in the state indicated by the first state information, the control information is generated by combining the first state information of the vehicle. This helps to make the driving style of the vehicle when driving according to the plan indicated by the control information as close as possible to the driving style indicated by the first driving style information, and thus helps to more accurately meet the personalized needs of users.

[0014] In one possible implementation, the process of the training device obtaining predictive control information through the first model includes: the training device fusing the first feature and the second feature to obtain the fused feature; for example, the training device obtains predictive control information corresponding to the training sample through the first model based on environmental information, the first feature and the second feature, including: the training device fusing the first feature and the second feature to obtain the fused feature, and obtaining predictive control information corresponding to the training sample through the first model based on environmental information and the fused feature.

[0015] The method also includes: the training device obtains the predicted state information of the vehicle through a first decoder based on the fused features; the loss function also indicates the similarity between the predicted state information and the first state information; the goal of training using the loss function also includes improving the similarity between the predicted state information and the first state information.

[0016] Since subsequent steps are based on the first fused features and environmental information to obtain regulatory information, in order to ensure that the first fused features can still fully carry the vehicle's state information, this embodiment will also use the first decoder to restore the vehicle's state information based on the first fused features, and use a loss function to improve the similarity between the predicted state information and the correct state information. The aforementioned training helps to ensure that the first fused features can still fully retain the vehicle's state information, so that the vehicle's state information can also be used in the subsequent process of obtaining regulatory information based on the first fused features and the third feature. This is beneficial to fully combining the vehicle's state information to generate regulatory information, so that the driving style of the vehicle when driving according to the scheme indicated by the regulatory information can be closer to the desired driving style.

[0017] In one possible implementation, the process of the training device obtaining predictive control information through the first model includes: the training device fusing the first feature and the second feature to obtain the fused feature; for example, the training device obtains predictive control information corresponding to the training sample through the first model based on environmental information, the first feature and the second feature, including: the training device fusing the first feature and the second feature to obtain the fused feature, and obtaining predictive control information corresponding to the training sample through the first model based on environmental information and the fused feature.

[0018] The training data also includes the vehicle's correct behavior information. The method also includes: the training device obtains the vehicle's predicted behavior information through a second decoder based on the fused features. The loss function also indicates the similarity between the predicted behavior information and the correct behavior information. The goal of training using the loss function also includes improving the similarity between the predicted behavior information and the correct behavior information.

[0019] In this application, both the predicted behavior information and the correct behavior information are vehicle behavior information. Behavior information can also be referred to as behavior label. Correct behavior information can be understood as the truth value corresponding to the predicted behavior information. For example, vehicle behavior information may include at least one of the following: whether there is overtaking or cutting in, whether there is driver overtake, whether there is sudden acceleration, whether there is sudden deceleration, or other behavior information.

[0020] Since subsequent steps obtain regulatory information based on the first fused features and environmental information, in order to ensure that the first fused features can still fully carry vehicle behavior information, this embodiment of the application will also infer vehicle behavior information based on the first fused features through a second decoder, and use a loss function to improve the similarity between the predicted behavior information and the correct behavior information. Since it is only possible to accurately infer vehicle behavior information by using the first fused features if the vehicle state information is still fully preserved in the first fused features, the aforementioned training method is conducive to ensuring that the first fused features can still fully preserve vehicle behavior information. Thus, in the subsequent process of obtaining regulatory information based on the first fused features and the third feature, the vehicle behavior information can also be used, which is conducive to fully combining vehicle behavior information to generate regulatory information, so that the driving style of the vehicle when driving according to the scheme indicated by the regulatory information can be closer to the desired driving style.

[0021] In one possible implementation, the method further includes: the training device obtaining predicted driving style information based on the first feature through a third decoder; the loss function also indicates the similarity between the predicted driving style information and the first driving style information, which can be understood as the correct driving style information, or the first driving style information can also be understood as the ground truth corresponding to the predicted driving style information. The goal of training using the loss function also includes improving the similarity between the predicted driving style information and the first driving style information.

[0022] In this implementation, the training device also obtains the predicted driving style information based on the first feature through the third decoder. The loss function also indicates the similarity between the predicted driving style information and the correct driving style information, thereby guiding the first feature extraction module to obtain features that can fully carry driving style information, which is beneficial to obtaining good first features.

[0023] In one possible implementation, the training device extracts features from the first driving style information to obtain a first feature, including: using a first feature extraction module to extract features from the first driving style information to obtain the first feature. The training device extracts features from the first state information of the vehicle to obtain a second feature, including: using a second feature extraction module to extract features from the first state information of the vehicle to obtain the second feature. Based on environmental information, the first feature, and the second feature, the training device obtains predictive control information corresponding to the training samples through a first model, including: using a third feature extraction module to extract features from the environmental information to obtain a third feature, wherein the first feature extraction module, the second feature extraction module, and the third feature extraction module are different modules in the first model; based on the first feature, the second feature, and the third feature, the predictive control information corresponding to the training samples is obtained through the first model.

[0024] In this implementation, three different feature extraction modules are used to extract features from three different types of information: driving style information, vehicle status information, and environmental information. This allows each feature extraction module to extract features specifically for one type of information, which is beneficial for extracting better first, second, and third features. In turn, it is beneficial for obtaining better regulatory control information based on the first, second, and third features.

[0025] In one possible implementation, the training device trains the first model using a loss function, including: training the first model using the loss function until convergence is met, obtaining a trained first model, which includes a trained first feature extraction module. The training device also extracts features from each of the multiple driving style information using the trained first feature extraction module, obtaining features for each of the multiple driving style information, where the first driving style information is one of the multiple driving style information. The second model is deployed to the execution device, and includes a feature set and modules other than the first feature extraction module in the first model. The feature set includes multiple driving style information and features for each of the multiple driving style information, and is used by the execution device to obtain features of the driving style information.

[0026] In this implementation, after the training device obtains the first feature extraction module after training, it can obtain a feature set through the first feature extraction module after training. The feature set is then deployed on the execution device. The execution device can obtain the features of each driving style information by looking up a table. This helps to reduce the latency consumed by the execution device when obtaining the features of driving style information, and also helps to reduce the consumption of computer resources in the process of obtaining the features of driving style information.

[0027] In one possible implementation, the first driving style information includes a first parameter value indicating the vehicle's driving efficiency, which can also be referred to as the vehicle's speed. Clearly defining which parameters are included in the driving style information improves the feasibility of this solution; furthermore, since the driving style information consists of parameter values, it uses a concise way to express the user's driving style requirements, minimizing the amount of information the model needs to process and thus saving computer resources.

[0028] Alternatively, the first driving style information may also include a second parameter value indicating the vehicle's driving comfort level, or the first driving style information may include parameter values ​​of more or fewer parameters. Alternatively, the first driving style information may also include text, for example, the first driving style information includes: high driving efficiency and moderate driving comfort.

[0029] In one possible implementation, the first training data can be obtained based on the second training data. The first training data includes a first training sample and the expected regulatory control information corresponding to the first training sample. The second training data includes a second training sample and the expected regulatory control information corresponding to the second training sample. The second training sample includes environmental information (optionally, it also includes vehicle state information). The training device can determine the first driving style information that matches the expected regulatory control information in the second training sample, and add the first driving style information to the second training sample to obtain the first training sample. The first driving style information that matches the expected regulatory control information is obtained from multiple driving style information based on the score of the expected regulatory control information. The score of the expected regulatory control information is obtained after data analysis of the expected regulatory control information using preset indicators.

[0030] For example, at least one preset indicator may include at least one of the following: maximum positive acceleration, maximum negative acceleration, maximum steering angular velocity, overtaking rate, minimum following distance, average following distance, number of overtaking and lane change times, or other preset indicators.

[0031] In this implementation, since traditional training data often includes environmental information and desired regulatory information, this application embodiment also provides rules for determining the first driving style information that matches the desired regulatory information, which can automatically obtain the first training data and reduce the implementation difficulty of this solution.

[0032] Optionally, the multiple driving style information may also include driving style information indicating that the driving style is the default. For example, the second training data set includes multiple second training data sets. Before determining the driving style information that matches the desired regulation information in each of the second training data sets, the training device may first randomly sample multiple second training data sets from the second training data set, set the driving style information of the sampled multiple second training data sets as the default, and then determine the driving style information that matches the desired regulation information in each of the remaining second training data sets.

[0033] In one possible implementation, different preset indicators are used for different traffic scenarios. For example, at least two different traffic scenarios may include highway scenarios, intersection scenarios, non-highway and non-intersection scenarios, or other scenarios, where other scenarios can be understood as traffic scenarios other than highway scenarios, intersection scenarios, and non-highway and non-intersection scenarios.

[0034] In this implementation, since the driving patterns of vehicles vary greatly under different traffic scenarios, when using preset indicators to analyze the expected traffic control information, using different preset indicators for data analysis in different traffic scenarios is beneficial to matching more accurate driving style information for each expected traffic control information, which is beneficial to obtaining better training data, and thus to making the performance of the first model after training better.

[0035] Secondly, this application provides a method for obtaining regulatory information, which can be applied to the field of intelligent driving. In this method, the vehicle obtains second driving style information and environmental information. The second driving style information indicates the driving style of the vehicle and is obtained based on user operation. The second driving style information and environmental information are input into a machine learning model, and regulatory information is obtained through the machine learning model. The regulatory information is matched with the driving style indicated by the second driving style information.

[0036] In one possible implementation, the method further includes: acquiring vehicle state information, the input of the machine learning model also including vehicle state information; inputting second driving style information and environmental information into the machine learning model, and obtaining regulation information through the machine learning model, including: acquiring a first feature of the second driving style information; extracting features from the vehicle state information to obtain a second feature; and obtaining predicted regulation information corresponding to the training samples through a first model based on environmental information, the first feature and the second feature.

[0037] In one possible implementation, the machine learning model can be a second model, which includes a feature set, comprising multiple driving style information and features of each driving style information, including second driving style information; the vehicle acquires a first feature, which includes: the vehicle acquiring a first feature of the second driving style information from the feature set.

[0038] Alternatively, the machine learning model can be a first model that has undergone training. If the first feature extraction module in the first model uses a VAE, it is more suitable for scenarios where the driving style information includes continuous values. In other words, if the driving style information includes continuous values, using a VAE to extract features from the driving style information is beneficial for obtaining better first features.

[0039] In the second aspect of this application, the vehicle is also used to perform the steps executed by the training device in the first aspect and various possible implementations of the first aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the second aspect can be found in the first aspect, and will not be repeated here.

[0040] Thirdly, this application provides a model training device that can be used in the field of intelligent driving. The device includes: an acquisition module for acquiring training data, the training data including training samples and expected regulation information corresponding to the training samples, the training samples including first driving style information and environmental information, the first driving style information indicating the driving style of the vehicle, and the expected regulation information matching the driving style indicated by the first driving style information; a processing module for inputting the training samples into a first model and obtaining predicted regulation information corresponding to the training samples through the first model; and a training module for training the first model using a loss function, the loss function indicating the similarity between the predicted regulation information and the expected regulation information.

[0041] In the third aspect of this application, the training device for the model is also used to perform the steps executed by the training device in the first aspect and various possible implementations of the first aspect. The specific implementation methods of the steps in the third aspect, the meanings of the terms, and the beneficial effects they bring can all be found in the first aspect, and will not be repeated here.

[0042] Fourthly, this application provides a device for acquiring regulatory information, which can be used in the field of intelligent driving. The device includes: an acquisition module for acquiring second driving style information and environmental information, wherein the second driving style information indicates the driving style of the vehicle and is obtained based on user operation; and a processing module for inputting the second driving style information and the environmental information into a machine learning model, and obtaining regulatory information through the machine learning model, wherein the regulatory information matches the driving style indicated by the second driving style information.

[0043] In the fourth aspect of this application, the device for acquiring regulatory information is also used to perform the steps executed by the vehicle in the second aspect and various possible implementations of the second aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the fourth aspect can be found in the first aspect, and will not be repeated here.

[0044] Fifthly, this application provides an apparatus including a processor and a memory, the processor being coupled to the memory, the memory storing program instructions, which, when executed by the processor, implement the methods described in the first or second aspect.

[0045] In a sixth aspect, this application provides a vehicle including a processor and a memory, the processor being coupled to the memory, the memory storing program instructions, and the method described in the second aspect being implemented when the program instructions stored in the memory are executed by the processor.

[0046] In a seventh aspect, this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first or second aspect.

[0047] Eighthly, this application provides a computer program product comprising a program that, when run on a computer, causes the computer to perform the methods described in the first or second aspect.

[0048] Ninthly, this application provides a chip system including a processor for supporting the implementation of the functions involved in the foregoing aspects, such as transmitting or processing data and / or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the terminal device or communication device. This chip system may be composed of chips or may include chips and other discrete devices.

[0049] The second to ninth aspects of this application correspond to the first aspect or multiple possible ways of the first aspect, and have corresponding beneficial effects. Attached Figure Description

[0050] Figure 1 is a structural diagram of an artificial intelligence main framework provided in this application;

[0051] Figure 2 is a system architecture diagram of a regulatory information acquisition system provided in an embodiment of this application;

[0052] Figure 3 is a flowchart illustrating a training method for a model provided in an embodiment of this application;

[0053] Figure 4 is a schematic diagram of a first model provided in an embodiment of this application;

[0054] Figure 5 is another schematic diagram of the training method of the model provided in the embodiment of this application;

[0055] Figure 6 is a schematic diagram of a second model provided in an embodiment of this application;

[0056] Figure 7 is a schematic diagram of the relationship between the first training data and the second training data provided in an embodiment of this application;

[0057] Figure 8 is a schematic diagram of a method for obtaining a first training data set provided in an embodiment of this application;

[0058] Figure 9 is a schematic diagram of preset indicators used in different traffic scenarios provided in the embodiments of this application;

[0059] Figure 10 is a schematic diagram of a process for obtaining a first training dataset based on a second training dataset according to an embodiment of this application;

[0060] Figure 11 is a flowchart illustrating a method for obtaining regulatory information according to an embodiment of this application.

[0061] Figure 12 is a schematic diagram of an interface for obtaining driving style information provided in an embodiment of this application;

[0062] Figure 13 is a schematic diagram of another interface for obtaining driving style information provided in an embodiment of this application;

[0063] Figure 14 is a schematic diagram illustrating the beneficial effects of the method provided in the embodiments of this application;

[0064] Figure 15 is a schematic diagram of a driving scheme obtained by the method provided in this application according to an embodiment of the present application;

[0065] Figure 16 is a schematic diagram of a training device for a model provided in an embodiment of this application;

[0066] Figure 17 is a schematic diagram of a control information acquisition device provided in an embodiment of this application;

[0067] Figure 18 is a schematic diagram of a device provided in an embodiment of this application;

[0068] Figure 19 is a structural schematic diagram of a vehicle provided in an embodiment of this application. Detailed Implementation

[0069] The embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0070] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0071] In the embodiments of this application, "instruction" can include direct and indirect instructions, as well as explicit and implicit instructions. The information indicated by a certain piece of information (hereinafter referred to as instruction information) is called the information to be instructed. In specific implementation, there are many ways to indicate the information to be instructed, such as, but not limited to, directly indicating the information to be instructed, such as the information to be instructed itself or its index. It can also indirectly indicate the information to be instructed by indicating other information, where there is an association between the other information and the information to be instructed; or it can indicate only a part of the information to be instructed, while the other parts are known or pre-agreed upon. For example, the instruction can be implemented by using a pre-agreed (e.g., protocol predefined) arrangement of various information, thereby reducing the instruction overhead to a certain extent. This application does not limit the specific method of instruction. It is understood that for the sender of the instruction information, the instruction information can be used to indicate the information to be instructed; for the receiver of the instruction information, the instruction information can be used to determine the information to be instructed.

[0072] First, the overall workflow of the artificial intelligence system is described, as shown in Figure 1. Figure 1 is a structural diagram of one aspect of the artificial intelligence framework provided in this application. The framework is then elaborated on from two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0073] (1) Infrastructure

[0074] The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips, which can specifically employ hardware acceleration chips such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), tensor processing units (TPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0075] (2) Data

[0076] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0077] (3) Data processing

[0078] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0079] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0080] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0081] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0082] (4) General ability

[0083] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0084] (5) Smart Products and Industry Applications

[0085] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent terminals, intelligent manufacturing, intelligent transportation, smart homes, intelligent healthcare, intelligent security, intelligent driving, and smart cities.

[0086] The method provided in this application can be applied to the field of intelligent driving. For example, the method provided in this application can be used in application scenarios to obtain vehicle control information. The vehicle can be a car, truck, motorcycle, bus, boat, lawnmower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, airplane, or helicopter, etc., and this application does not impose any particular limitation.

[0087] The vehicle's planning and control information includes at least one of the following: the vehicle's planned location, the vehicle's driving strategy, the vehicle's control information, or other types of information, which can be determined based on the actual application scenario.

[0088] For example, the vehicle's control information may include at least one of the following: the planned position of the vehicle corresponding to each of at least one time after the current time, the driving strategy of the vehicle corresponding to each of at least one time after the current time, and the control information of the vehicle corresponding to each of at least one time after the current time.

[0089] Furthermore, the planned position of a vehicle at each time point can also be understood as the planned trajectory point of the vehicle at each time point. If at least one time point is specifically at least two time points, then the at least two planned trajectory points of the vehicle corresponding to at least two time points can also be understood as the planned trajectory of the vehicle at the aforementioned at least two time points.

[0090] The driving strategy for a vehicle at each moment can include the lateral driving strategy and / or longitudinal driving strategy corresponding to each moment. The lateral driving strategy corresponding to a certain moment can be left turn, straight, right turn, lane change or lateral avoidance, etc., and the longitudinal driving strategy corresponding to a certain moment can be acceleration, constant speed or deceleration, etc. The specific manifestation of the driving strategy can be determined in combination with the actual application scenario.

[0091] The vehicle control information corresponding to each moment may include: control signals of at least one component in the vehicle at each moment, and / or, the planned vehicle state information at each moment. For example, at least one component may include a steering wheel, engine, brakes, clutch, turn signals, or other components used during vehicle operation; the planned vehicle state information corresponding to each moment after the current moment can be understood as the state information that the vehicle needs to reach.

[0092] In related technologies, environmental information can be input into the machine learning model to obtain the regulatory information output by the machine learning model. However, since the input of the machine learning model only includes environmental information, the regulatory information output by the machine learning model in different vehicles is highly similar and cannot meet the personalized needs of users.

[0093] To address the aforementioned issues, this application discloses the following: Acquiring training data, which includes training samples and corresponding expected regulatory control information. The training samples include first driving style information and environmental information. The first driving style information indicates the vehicle's driving style, and the expected regulatory control information matches the driving style indicated by the first driving style information. The training samples are input into a first model, which then obtains predicted regulatory control information corresponding to the training samples. A loss function is used to train the first model, indicating the similarity between the predicted and expected regulatory control information. This provides a training method for the first model. The input to the first model includes not only environmental information but also driving style information indicating the vehicle's driving style. When training the first model using the training data, the expected regulatory control information in the training data matches the driving style indicated by the driving style information. This enables the first model to learn the ability to combine driving style information and environmental information to obtain regulatory control information matching the driving style indicated by the driving style information. This facilitates obtaining regulatory control information matching the user-defined driving style based on user-defined driving style and environmental information, thus meeting the user's personalized needs and greatly optimizing the user's riding experience.

[0094] Before detailing the method provided in this application, the architecture of the regulatory information acquisition system provided in this application will be described first. Please refer to Figure 2, which is a system architecture diagram of the regulatory information acquisition system provided in an embodiment of this application. In Figure 2, the regulatory information acquisition system 200 includes a training device 210, a database 220, an execution device 230, and a data storage system 240. The execution device 230 includes a computing module 231. The database 220 stores a training dataset. During the training phase, the training device 210 can use the training data in the training dataset to train the first model 201, obtaining the trained first model 201. The training device 210 can obtain a machine learning model 202 based on the trained first model 201. The specific implementation of obtaining the machine learning model 202 based on the first model 201 will be described in detail later when the method provided in this application is described in detail; it will not be elaborated here.

[0095] In the application phase, the inference process of the machine learning model 201 can be executed by the computing module 231 of the execution device 230. Optionally, as shown in Figure 2, the execution device 230 can be integrated into the vehicle, allowing users to directly interact with the vehicle where the execution device 230 is deployed. For example, the execution device 230 can be a module in the vehicle's host CPU that uses the machine learning model for data processing. The execution device 230 can also be a graphics processing unit (GPU), neural network processing unit (NPU), or tensor processing unit (TPU) in the vehicle, etc. The aforementioned GPU, NPU, or TPU is mounted as a coprocessor on the vehicle's host processor, and the host processor allocates tasks, etc.

[0096] The execution device 230 can access data, code, etc., in the data storage system 240, and can also store data, instructions, etc., in the data storage system 240. The data storage system 240 can be located within the execution device 230, or it can be an external memory relative to the execution device 230.

[0097] It should be noted that Figure 2 is merely a schematic diagram of one architecture of the regulatory information acquisition system provided in this application embodiment, and the positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, in some other embodiments of this application, the execution device 230 and the vehicle can be separate and independent devices. The execution device 230 is configured with an input / output (I / O) interface, through which the execution device 230 can interact with the vehicle for data. For example, in the application phase, the intelligent driving system in the vehicle can send driving style information and environmental information to the execution device 230 through the I / O interface. After obtaining the regulatory information through the machine learning model 201 deployed in the computing module 231, the execution device 230 can send the regulatory information to the intelligent driving system in the vehicle through the I / O interface.

[0098] For example, in some other embodiments of this application, the training device 210 and the execution device 230 may also be integrated into the same device, and the specific architecture of the regulatory information acquisition system can be determined according to the actual application scenario. The specific implementation processes of the training phase and the application phase are described below.

[0099] I. Training Phase

[0100] Please refer to Figure 3, which is a flowchart illustrating a model training method provided in an embodiment of this application. The model training method provided in an embodiment of this application may include:

[0101] 301. Obtain training data, which includes training samples and expected control information corresponding to the training samples. The training samples include first driving style information and environmental information. The first driving style information indicates the driving style of the vehicle, and the expected control information matches the driving style indicated by the first driving style information.

[0102] For example, the training device may deploy a first training dataset, which includes multiple training data sets (hereinafter referred to as "first training data" for ease of distinction). Each training data set includes a training sample (hereinafter referred to as "first training sample") and the expected control information corresponding to the first training sample. The first training sample includes first driving style information and environmental information; optionally, the first training sample may also include first state information of the vehicle.

[0103] Optionally, the first driving style information may include a first parameter value indicating the vehicle's driving efficiency, which can also be referred to as the vehicle's speed. Clearly defining which parameters are included in the driving style information improves the feasibility of this solution; furthermore, since the driving style information includes parameter values, it uses a concise way to express the user's driving style requirements, minimizing the amount of information the model needs to process and thus saving computer resources.

[0104] For example, during the training phase, the range of values ​​for the first parameter can include N values, where N is an integer greater than or equal to 1, and the N values ​​represent N levels of driving efficiency. Optionally, the aforementioned N values ​​can include N values ​​that increase sequentially, where a larger first parameter value indicates higher driving efficiency, and a smaller first parameter value indicates lower driving efficiency.

[0105] Optionally, the range of values ​​for the first parameter may include N+1 values. The N+1 values ​​include not only the aforementioned N values, but also a first value indicating the default driving efficiency. The first value is a value other than the aforementioned N values. For example, the N values ​​are N integers from 1 to N, the first value can be negative, or the first value can be 0, or the N values ​​and the first value can be non-integers, etc. The specific values ​​of the N values ​​and the first value can be determined in combination with the actual application scenario.

[0106] Alternatively, the first driving style information may also include a second parameter value indicating the vehicle's driving comfort level. For example, during the training phase, the range of values ​​for the second parameter value may include M values, where M is an integer greater than or equal to 1, and the M values ​​represent M levels of driving comfort. Optionally, the aforementioned M values ​​may include M values ​​that increase sequentially, with a larger second parameter value representing higher driving comfort and a smaller second parameter value representing lower driving comfort. Optionally, the range of values ​​for the second parameter value may include M+1 values, which not only include the aforementioned M values ​​but also a second value indicating the default driving comfort level, and this second value is a value other than the aforementioned M values.

[0107] Alternatively, the first driving style information can include more or fewer parameter values; the specific parameter values ​​included in the first driving style information can be determined based on the actual application scenario. Alternatively, the first driving style information can also include text; for example, the first driving style information could include: high driving efficiency and moderate driving comfort. The specific information included in the first driving style information can be determined based on the actual application scenario.

[0108] For example, the environmental information may include at least one image and / or point cloud data of the traffic environment around the vehicle. At least one image of the traffic environment around the vehicle may be acquired by a first sensor in the vehicle, and the at least one image may be an image under a perspective view (PV). The point cloud data of the traffic environment around the vehicle may be acquired by a second sensor in the vehicle.

[0109] For example, "traffic environment around the vehicle" can be understood as the environment within the field of view of the sensors in the vehicle (such as the first or second sensor mentioned above). Exemplarily, the first sensor can be a photoelectric sensor, such as a camera or event camera; for example, the second sensor can be an ultrasonic sensor, a lidar sensor, a millimeter-wave radar sensor, or other sensors capable of measuring and obtaining point cloud data, etc., which are not exhaustively listed in the embodiments of this application.

[0110] For example, the vehicle's first state information can be understood as the vehicle's current state information. The vehicle's first state information may include at least one of the following: velocity, acceleration, yaw rate, steering angle, pitch angle, or other physical characteristic information of the vehicle. Furthermore, the yaw rate may include yaw rate and pitch rate. The vehicle's first state information can reflect the vehicle's true physical state, and the specific information included can be determined in combination with the actual application scenario.

[0111] For example, the expected regulation information can also be understood as: the regulation information adopted by the vehicle if it wants to drive in the driving style indicated by the first driving style information in the traffic environment indicated by the environmental information in the first training sample (optionally, also including the vehicle state indicated by the first state information). The expected regulation information can also be replaced by the correct regulation information. The information contained in the expected regulation information is consistent with the information contained in the regulation information. For details, please refer to the above description for understanding, and it will not be repeated here.

[0112] 302. Input the training samples into the first model, and obtain the prediction and control information corresponding to the training samples through the first model.

[0113] The first model can be a machine learning model, such as a convolutional neural network, a fully connected neural network, an attention-based neural network, a residual neural network, or other types of neural networks, which can be determined based on the actual application scenario. The information contained in the predicted regulation information is consistent with the information contained in the regulation information; for details, please refer to the above description for understanding, which will not be repeated here. For example, step 302 may include: the training device inputs the first driving style information and environmental information into the first model, and obtains the expected regulation information corresponding to the first training sample through the first model.

[0114] Optionally, in one scenario, if the first training sample further includes the vehicle's first state information, then step 302 may include: the training device inputs the first driving style information, environmental information, and the vehicle's first state information into the first model, and obtains the predictive control information corresponding to the first training sample through the first model. For example, after inputting the first driving style information, environmental information, and the vehicle's first state information into the first model, the training device can extract features from the first driving style information using the first model to obtain a first feature, extract features from the vehicle's first state information using the first model to obtain a second feature, and then, based on the environmental information, the first feature, and the second feature, obtain the predictive control information corresponding to the training sample through the first model. Optionally, the first feature, the second feature, and the third feature can all be in matrix form; or, the first feature can be in vector form, and the second and third features can be in matrix form, etc.

[0115] In this embodiment, not only are the first driving style information and environmental information input into the first model, but the first state information of the vehicle is also input into the first model. Since the control information is used to control the vehicle, and the vehicle starts to drive according to the plan indicated by the control information in the state indicated by the first state information, the control information is generated by combining the first state information of the vehicle. This is beneficial to make the driving style of the vehicle when driving according to the plan indicated by the control information as close as possible to the driving style indicated by the first driving style information, and thus more accurately meet the personalized needs of the user.

[0116] Optionally, the training device uses a first feature extraction module in the first model to extract features from the first driving style information to obtain a first feature, and uses a second feature extraction module in the first model to extract features from the first state information of the vehicle to obtain a second feature. Based on the environmental information, the first feature, and the second feature, the training device obtains predictive control information corresponding to the training sample through the first model. This may include: the training device uses a third feature extraction module in the first model to extract features from the environmental information to obtain a third feature, and then, based on the first feature, the second feature, and the third feature, obtains predictive control information corresponding to the training sample through the first model; wherein the first feature extraction module, the second feature extraction module, and the third feature extraction module are different modules in the first model.

[0117] Optionally, the first feature extraction module can be a first encoder, which can be a variational autoencoder (VAE). For example, the first feature extraction module may include at least one of the following: a multilayer perceptron (MLP), a support vector machine, a recurrent neural network layer, a fully connected neural network layer, an attention-based neural network layer, or a residual neural network layer, etc.

[0118] Optionally, the second feature extraction module can be a second encoder. The second feature module and the first feature module can adopt the same or different model structures. For example, the second feature extraction module can include at least one of the following: multilayer perceptron, support vector machine, recurrent neural network layer, fully connected neural network layer, attention-based neural network layer or residual neural network layer, etc.

[0119] Optionally, the third feature extraction module can be understood as a backbone network for perception. For example, the third feature extraction module may include at least one of the following: a convolutional neural network layer, an attention-based neural network layer, a fully connected neural network layer, a multilayer perceptron, a support vector machine, or a residual neural network layer, etc.

[0120] In this embodiment, three different feature extraction modules are used to extract features from three different types of information: driving style information, vehicle status information, and environmental information. This allows each feature extraction module to extract features specifically for one type of information, which is beneficial for extracting better first, second, and third features. In turn, it is beneficial for obtaining better regulatory control information based on the first, second, and third features.

[0121] Optionally, the training device, based on the first feature, the second feature, and the third feature, obtains the prediction and control information corresponding to the training sample through the first model. This can include: the training device fusing the first feature and the second feature to obtain a fused feature (hereinafter referred to as the "first fused feature" for ease of distinction), and then obtaining the prediction and control information corresponding to the training sample through the first model based on the first fused feature and the third feature. Optionally, the fused feature can be in matrix form.

[0122] Optionally, the training device can fuse the first feature and the second feature using a fusion module to obtain the first fused feature. For example, the fusion module can be a fusion unit, and can include at least one of the following: a multilayer perceptron, a support vector machine, a fully connected neural network layer, an attention-based neural network layer, or a residual neural network layer. Alternatively, the fusion method can be concatenation or other fusion methods, which can be determined based on the actual application scenario.

[0123] For example, the first model may further include a planning and control backbone network and a planning and control head network; then the training device, based on the first fused features and the third features, obtains the predicted planning and control information corresponding to the training samples through the first model, which may include: the training device uses the planning and control backbone network to process the first fused features and the third features to obtain processed features, and generates the predicted planning and control information through the head network based on the processed features.

[0124] To better understand this solution, please refer to Figure 4. Figure 4 is a schematic diagram of a first model provided in an embodiment of this application. After inputting driving style information (e.g., first driving style information in the training samples), vehicle state information (e.g., first state information in the training samples), and environmental information into the first model, the first feature extraction module in the first model extracts features from the driving style information to obtain a first feature. The second feature extraction module in the first model extracts features from the vehicle state information to obtain a second feature. The first feature and the second feature are fused by a fusion module to obtain a first fused feature. The third feature extraction module in the first model extracts features from the environmental information to obtain a third feature. Then, based on the third feature and the first fused feature, the main network of planning and control is used to process the third feature to obtain the processed feature. Based on the processed feature, planning and control information is generated through the head network of planning and control. It should be understood that the example in Figure 4 is only for the convenience of understanding this solution and is not intended to limit this solution.

[0125] Alternatively, the training device may obtain the prediction and control information corresponding to the training sample through the first model based on the first feature, the second feature, and the third feature. This may include: the training device fusing the first feature and the third feature to obtain the second fused feature, and then obtaining the prediction and control information corresponding to the training sample through the first model based on the second fused feature and the third feature. The specific implementation of the aforementioned steps is similar to the specific implementation of "obtaining the prediction and control information corresponding to the training sample through the first model based on the first fused feature and the third feature". The difference is that the first fused feature in the above description is replaced with the second fused feature, which will not be described again here.

[0126] Alternatively, the training device may obtain predictive control information corresponding to the training samples through the first model based on the first feature, the second feature, and the third feature. This may include: the training device inputting the first feature, the second feature, and the third feature into the backbone network of planning and control; fusing the first feature, the second feature, and the third feature through the backbone network of planning and control to obtain the third fused feature; and generating predictive control information through the head network of planning and control based on the third fused feature.

[0127] In another scenario, if the first training sample does not include vehicle state information, the training device inputs the first driving style information and environmental information into the first model. It can then extract the first feature from the first driving style information through the first feature extraction module, extract the third feature from the environmental information through the third feature extraction module, fuse the first and third features through the planning and control backbone network to obtain the fourth fused feature, and generate predictive planning and control information based on the fourth fused feature through the planning and control head network.

[0128] 303. The first model is trained using a loss function, which indicates the similarity between the predicted regulatory information and the expected regulatory information.

[0129] For example, after obtaining the prediction and control information corresponding to the training samples generated by the first model, the training device can calculate the function value of the loss function based on the prediction and control information and the expected control information corresponding to the training samples; based on the function value of the loss function, the weight parameters in the first model are updated using the backpropagation algorithm to complete one training of the first model.

[0130] The training device can repeatedly execute steps 301 to 303 multiple times to iteratively train the first model until the convergence condition is met, thus obtaining the trained first model. For example, the convergence condition may include at least one of the following: satisfying the convergence condition of the loss function, training the first model a preset number of times, or other convergence conditions.

[0131] The loss function at least indicates the similarity between the predicted and expected regulatory information. The goal of training the first model using this loss function is to improve the similarity between the predicted and expected regulatory information. For example, the similarity between the predicted and expected regulatory information can be obtained by calculating the L1 distance, L2 distance, cosine similarity, or other methods, and the specific method can be determined based on the actual application scenario.

[0132] Optionally, before executing step 303, the training device can also obtain the predicted state information of the vehicle through the first decoder based on the first fused features. The aforementioned loss function can also indicate the similarity between the predicted state information and the first state information. The first state information can be understood as the correct state information of the vehicle, or it can be understood as the ground truth corresponding to the predicted state information. The goal of training using this loss function also includes improving the similarity between the predicted state information and the first state information. For example, the loss function can include a first loss term and a second loss term. The first loss term indicates the similarity between the predicted regulatory information and the expected regulatory information, and the second loss term indicates the similarity between the predicted state information and the first state information. For example, the similarity between the predicted state information and the first state information can be obtained by calculating the L1 distance, L2 distance, cosine similarity, Euclidean distance, or other methods between the predicted state information and the first state information, and the specific method can be determined based on the actual application scenario.

[0133] The types of information contained in the predicted state information are similar to those contained in the first state information. Please refer to the above description of the first state information for understanding, and it will not be repeated here. It should be noted that the types of information contained in the predicted state information may be less than or equal to the types of information contained in the first state information, as long as it is ensured that each type of information in the predicted state information can find a corresponding truth value in the first state information.

[0134] Optionally, the training device can obtain the predicted state information of the vehicle through the first decoder based on the first fused features and the first feature; in other words, the training device can input the first fused features and the first feature into the first decoder to obtain the predicted state information of the vehicle generated by the first decoder.

[0135] For example, the first decoder reconstructs the vehicle's state information based on the first fused features (optionally, it also includes the first features). For example, the first decoder may include at least one of the following: a multilayer perceptron, a support vector machine, a fully connected neural network layer, a residual neural network layer, or other model structures.

[0136] Since subsequent steps are based on the first fused features and environmental information to obtain regulatory information, in order to ensure that the first fused features can still fully carry the vehicle's state information, this embodiment will also use the first decoder to restore the vehicle's state information based on the first fused features, and use a loss function to improve the similarity between the predicted state information and the correct state information. The aforementioned training helps to ensure that the first fused features can still fully retain the vehicle's state information, so that the vehicle's state information can also be used in the subsequent process of obtaining regulatory information based on the first fused features and the third feature. This is beneficial to fully combining the vehicle's state information to generate regulatory information, so that the driving style of the vehicle when driving according to the scheme indicated by the regulatory information can be closer to the desired driving style.

[0137] Optionally, the training data also includes the vehicle's correct behavior information. Before executing step 303, the training device can also obtain the vehicle's predicted behavior information through a second decoder based on the first fused features. The aforementioned loss function can also indicate the similarity between the predicted behavior information and the correct behavior information. The goal of training using this loss function also includes improving the similarity between the predicted behavior information and the correct behavior information. For example, the loss function can also include a third loss term, which indicates the similarity between the predicted behavior information and the correct behavior information. Exemplarily, the similarity between the predicted behavior information and the correct behavior information can be obtained by calculating the L1 distance, L2 distance, cosine similarity, Euclidean distance, or other methods between the predicted behavior information and the correct behavior information, which can be determined in conjunction with the actual application scenario.

[0138] In this application, both the predicted behavior information and the correct behavior information are vehicle behavior information. Behavior information can also be referred to as behavior label. Correct behavior information can be understood as the truth value corresponding to the predicted behavior information. For example, vehicle behavior information may include at least one of the following: whether there is overtaking or cutting in, whether there is driver overtake, whether there is sudden acceleration, whether there is sudden deceleration, or other behavior information, etc. The specifics can be determined in combination with the actual application scenario.

[0139] Optionally, the training device can obtain the predicted behavior information of the vehicle through the second decoder based on the first fused features and the first feature; in other words, the training device can input the first fused features and the first feature into the second decoder to obtain the predicted behavior information of the vehicle generated by the second decoder.

[0140] For example, the second decoder infers the vehicle's behavior information based on the first fused features (optionally, it also includes the first features). For example, the second decoder may include at least one of the following: a multilayer perceptron, a support vector machine, a fully connected neural network layer, a residual neural network layer, or other model structures.

[0141] Since subsequent steps obtain regulatory information based on the first fused features and environmental information, in order to ensure that the first fused features can still fully carry vehicle behavior information, this embodiment of the application will also infer vehicle behavior information based on the first fused features through a second decoder, and use a loss function to improve the similarity between the predicted behavior information and the correct behavior information. Since it is only possible to accurately infer vehicle behavior information by using the first fused features if the vehicle state information is still fully preserved in the first fused features, the aforementioned training method is conducive to ensuring that the first fused features can still fully preserve vehicle behavior information. Thus, in the subsequent process of obtaining regulatory information based on the first fused features and the third feature, the vehicle behavior information can also be used, which is conducive to fully combining vehicle behavior information to generate regulatory information, so that the driving style of the vehicle when driving according to the scheme indicated by the regulatory information can be closer to the desired driving style.

[0142] Optionally, the method further includes: training the device to obtain predicted driving style information based on the first feature through a third decoder; the loss function also indicates the similarity between the predicted driving style information and the first driving style information, where the first driving style information can be understood as the correct driving style information, or it can also be understood as the ground truth corresponding to the predicted driving style information. The goal of training using this loss function also includes improving the similarity between the predicted driving style information and the first driving style information. For example, the loss function may also include a fourth loss term, which indicates the similarity between the predicted driving style information and the first driving style information. Exemplarily, the similarity between the predicted driving style information and the first driving style information can be obtained by calculating the L1 distance, L2 distance, cosine similarity, Euclidean distance, or other methods between the predicted driving style information and the first driving style information, which can be determined in conjunction with the actual application scenario.

[0143] Optionally, the training device can obtain the predicted driving style information through a third decoder based on the first feature, the predicted state information, and the predicted behavior information; in other words, the training device can input the first feature, the predicted state information, and the predicted behavior information into the third decoder to obtain the predicted driving style information generated by the third decoder.

[0144] For example, the third decoder reconstructs driving style information based on the first feature (optionally, it also includes predicted state information and predicted behavior information). For example, the third decoder may include at least one of the following: multilayer perceptron, support vector machine, fully connected neural network layer, residual neural network layer or other model structure.

[0145] In this embodiment of the application, the training device also obtains the predicted driving style information through the third decoder based on the first feature. The loss function also indicates the similarity between the predicted driving style information and the correct driving style information, thereby guiding the first feature extraction module to obtain features that can fully carry driving style information, which is beneficial to obtaining good first features.

[0146] To understand this solution more intuitively, please refer to Figure 5. Figure 5 is another schematic diagram of the training method of the model provided in the embodiment of this application. Figure 5 can be understood in conjunction with the above description of Figure 4. In Figure 5, the first feature extraction module is taken as the first encoder and the second feature extraction module is taken as the second encoder. As shown in Figure 5, after the training device generates the first feature of the first driving style information through the first encoder and generates the first fused feature through the fusion unit, it can obtain the predicted state information of the vehicle through the first decoder based on the first fused feature and the first feature; obtain the predicted behavior information of the vehicle through the second decoder based on the first fused feature and the first feature; and obtain the predicted driving style information of the vehicle through the third decoder based on the first feature, the predicted state information of the vehicle, and the predicted behavior information of the vehicle. The training device can calculate the value of a loss function, which may include a first loss term, a second loss term, a third loss term, and a fourth loss term. The first loss term indicates the similarity between the predicted control information and the expected control information. The second loss term indicates the similarity between the predicted state information and the first state information. The third loss term indicates the similarity between the predicted behavior information and the correct behavior information. The fourth loss term indicates the similarity between the predicted driving style information and the first driving style information. It should be understood that the example in Figure 5 is only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0147] Optionally, the trained first model may include a trained first feature extraction module. After obtaining the trained first model, the training device can further extract features from each of the multiple driving style information using the trained first feature extraction module, obtaining features for each of the multiple driving style information, where the first driving style information is one of the multiple driving style information. Then, a second model is obtained based on the trained first model. The second model is deployed to the execution device and includes a feature set and modules other than the first feature extraction module in the first model. The feature set includes multiple driving style information and the first feature of each of the multiple driving style information, and the feature set is used by the execution device to obtain the first feature of the driving style information.

[0148] In this embodiment, after the training device obtains the first feature extraction module after training, it can obtain a feature set through the first feature extraction module after training and deploy the feature set on the execution device. Then, the execution device can obtain the features of each driving style information by looking up a table. This helps to reduce the latency consumed by the execution device when obtaining the features of driving style information, and also helps to reduce the consumption of computer resources in the process of obtaining the features of driving style information.

[0149] To understand this solution more intuitively, please refer to Figure 6. Figure 6 is a schematic diagram of a second model provided in an embodiment of this application. The second model shown in Figure 6 can be understood in conjunction with the above description of Figure 4. The difference between the second model shown in Figure 6 and the first model shown in Figure 4 is that the first feature extraction module in the first model is replaced by the feature set in the second model. After obtaining the driving style information, the first feature corresponding to the input driving style information can be queried from the feature set. The repeated parts of Figure 6 and Figure 4 will not be described again here. It should be understood that the example in Figure 6 is only for the convenience of understanding this solution and is not intended to limit this solution.

[0150] In this embodiment, a training method for a first model is provided. The input of the first model includes not only environmental information but also driving style information indicating the driving style of the vehicle. When training the first model using training data, the expected regulation information in the training data matches the driving style indicated by the driving style information. This enables the first model to learn the ability to combine driving style information and environmental information to obtain regulation information that matches the driving style indicated by the driving style information. This is beneficial for obtaining regulation information that matches the user-defined driving style based on user-defined driving style and environmental information, thereby meeting the user's personalized needs and greatly optimizing the user's riding experience.

[0151] Optionally, embodiments of this application also provide a method for obtaining training data. For example, the first training data can be obtained based on the second training data. The first training data includes a first training sample and the expected regulation information corresponding to the first training sample. The second training data includes a second training sample and the expected regulation information corresponding to the second training sample. The second training sample includes environmental information (optionally, it also includes vehicle state information). The training device can determine the first driving style information that matches the expected regulation information in the second training sample and add the first driving style information to the second training sample to obtain the first training sample.

[0152] It should be noted that each second training data is a second training data that meets the screening criteria. The screening criteria may include: no traffic violations and no collisions. In other words, the driving scheme indicated by the expected regulatory information in each second training data is a legal and safe driving scheme.

[0153] To more intuitively understand the relationship between the first training data and the second training data, please refer to Figure 7. Figure 7 is a schematic diagram of the relationship between the first training data and the second training data provided in an embodiment of this application. In Figure 7, the first training data i and the first training data j represent two different first training data. As shown in Figure 7, the first training data i is obtained by adding driving style information i that matches the expected regulation information in the second training data i to the second training data i. The first training data j is obtained by adding driving style information j that matches the expected regulation information in the second training data j to the second training data. That is, a first training data can be obtained by adding driving style information that matches the expected regulation information in the second training data to the second training data. The first model is trained using multiple first training data to obtain the trained first model. Then, the feature set can be obtained based on the trained first feature extraction module in the trained first model. It should be understood that the example in Figure 7 is only for the convenience of understanding this solution and is not intended to limit this solution.

[0154] Optionally, the multiple driving style information may also include driving style information indicating that the driving style is the default. For example, the second training data set includes multiple second training data sets. Before determining the driving style information that matches the desired regulation information in each of the second training data sets, the training device may first randomly sample multiple second training data sets from the second training data set, set the driving style information of the aforementioned sampled multiple second training data sets as the default, and then determine the driving style information that matches the desired regulation information in each of the remaining second training data sets. To understand this solution more intuitively, please refer to Figure 8. Figure 8 is a schematic diagram of a method for obtaining the first training data set provided in an embodiment of this application. The driving dataset in Figure 8 can be regarded as the second training data set. The training device can first sample multiple second training data from the second training data set, and set the driving style information of the sampled multiple second training data to the driving style information when the driving style is the default. Then, determine the driving style information that the expected control information matches in each second training data, thereby obtaining P+1 subsets. After merging and shuffling the P+1 subsets, the first training data set is obtained. It should be understood that the example in Figure 9 is only for the convenience of understanding this solution and is not intended to limit this solution.

[0155] For example, in one case, the first driving style information that matches the expected regulatory information in each second training data can be determined manually by manually labeling the information.

[0156] In another scenario, the training device can input the desired control information (optionally including vehicle state information) from each second training data point into the third model to obtain first driving style information generated by the third model that matches the desired control information. The third model is a machine learning model, such as a recurrent neural network, an attention-based neural network, a fully connected neural network, or other types of neural networks, which can be determined based on the actual application scenario.

[0157] In another scenario, the first driving style information matched with the desired control information in each second training dataset is obtained from multiple driving style information based on the score of the desired control information. The score of the desired control information is obtained by analyzing the data using at least one preset indicator. For example, the training device can analyze the data of the desired control information using each of the at least one preset indicator to obtain the value of each preset indicator and determine the second score corresponding to the value of each preset indicator. For instance, the at least one preset indicator may include at least one of the following: maximum positive acceleration, maximum negative acceleration, maximum steering angular velocity, overtaking rate, minimum following distance, average following distance, number of overtaking and lane change attempts, or other preset indicators.

[0158] The training device obtains the first score of the desired regulatory information based on the second score corresponding to each of the at least one preset index. For example, the training device can perform a weighted summation of the second scores corresponding to each of the at least one preset index to obtain the first score of the desired regulatory information.

[0159] The training device can obtain a first score for each expected control information among multiple expected control information using the aforementioned method. Based on the first score of each expected control information, a first driving style information matching each expected control information is obtained. For example, the multiple driving style information may include P types of driving style information. In one implementation, the expected control information in all second training data uses the same preset index. The training device can sort all expected control information according to the first score of each expected control information, thereby splitting the multiple expected control information into P subsets. Each of the P subsets corresponds to a type of driving style information, and all expected control information in the same subset matches the same type of driving style information.

[0160] Alternatively, in another implementation, the first driving style information includes the parameter value of a first parameter indicating the driving efficiency of the vehicle. The training device can divide all the second training data into second training data in at least two different traffic scenarios. Different preset indicators are used in different traffic scenarios. Correspondingly, based on the first score of the expected regulation information in all the second training data in the same traffic scenario, the driving style information matched by each expected regulation information is determined.

[0161] For example, at least two different traffic scenarios may include highway scenarios, intersection scenarios, non-highway and non-intersection scenarios, or other scenarios, where other scenarios can be understood as traffic scenarios other than highway scenarios, intersection scenarios, and non-highway and non-intersection scenarios.

[0162] To more intuitively understand this solution, please refer to Figures 9 and 10. Figure 9 is a schematic diagram of preset indicators used in different traffic scenarios provided by the embodiments of this application. As shown in Figure 9, the preset indicators used in the highway scenario include: maximum speeding percentage, maximum steering angular velocity, maximum positive acceleration, relative speed with other vehicles, maximum negative acceleration, overtaking rate, the percentage between average vehicle speed and speed limit (i.e., average vehicle speed / speed limit percentage in Figure 9), maximum overtaking relative speed and average overtaking relative speed (i.e., maximum / average overtaking relative speed in Figure 9), minimum lane change time, and... The average lane change time (i.e., the minimum / average lane change time in Figure 9), minimum following distance, and average following distance (i.e., the minimum / average following distance in Figure 9), as well as overtaking and lane changing (cutin), are preset indicators used in non-highway and non-intersection scenarios and in intersection scenarios, which can be referred to in Figure 9. They will not be described in detail here. It should be noted that the average / minimum VRU distance shown in Figure 9 represents the average distance between the vehicle and other vehicles and the minimum distance between the vehicle and other vehicles. It should be understood that the examples in Figure 9 are only for the convenience of understanding this solution and are not intended to limit this solution.

[0163] Figure 10 is a flowchart illustrating a process for obtaining a first training dataset based on a second training dataset according to an embodiment of this application. In Figure 10, taking the first driving style information as an example, which includes the parameter value of a first parameter indicating the driving efficiency of the vehicle, P is equal to N. The first training dataset includes N+1 subsets. As shown in Figure 10, the second training dataset is first sampled to obtain a subset corresponding to the default driving style information (i.e., subset-1 in Figure 10). The training device then divides the remaining second training data into scenarios, assigning them to highway scenarios, intersection scenarios in urban scenarios, non-intersection scenarios in urban scenarios, and other scenarios. The expected traffic control information in each traffic scenario is analyzed according to the preset indicators used in that traffic scenario to obtain the value of each preset indicator for each expected traffic control information. Based on the value of each preset indicator, a first score for each expected traffic control information is obtained. Based on the first score of each expected traffic control information in the same traffic scenario, all expected traffic control information in the same traffic scenario is sorted to divide all second training data in each traffic scenario into N subsets. The driving style information subsets in different traffic scenarios are merged to obtain N subsets from subset 1 to subset N. Subsets from subset 1 to subset N and subset -1 constitute the first training dataset. It should be understood that the example in Figure 10 is only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0164] It should be noted that the execution device for the above-mentioned acquisition of the first training dataset can also be a device other than the training device, so that the training device can receive the first training dataset sent by other devices. In this embodiment of the application, only the execution device is the training device as an example for illustration.

[0165] Since traditional training data often includes environmental information and desired regulatory information, this application embodiment also provides rules for determining the first driving style information that matches the desired regulatory information, which can automatically obtain the first training data and reduce the implementation difficulty of this solution. In addition, since the driving modes of vehicles vary greatly in different traffic scenarios, when using preset indicators to analyze the data of desired regulatory information, using different preset indicators for data analysis in different traffic scenarios is beneficial to matching more accurate driving style information for each desired regulatory information, that is, to obtaining better training data, and thus to improving the performance of the first model after training.

[0166] II. Application Phase

[0167] Please refer to Figure 11. Figure 11 is a flowchart illustrating a method for obtaining regulatory information provided in an embodiment of this application. The method for obtaining regulatory information provided in an embodiment of this application may include:

[0168] 1101. Obtain the second driving style information. The second driving style information indicates the driving style of the vehicle and is obtained based on user operation.

[0169] The intelligent driving system in the vehicle can obtain second driving style information based on user operations; for example, if the user does not input any configuration operation for driving style information, the driving style of the vehicle indicated by the second driving style information can be the default.

[0170] For example, the intelligent driving system in a vehicle can obtain the user's input of driving style configuration operations through a display interface; or it can obtain the user's input of driving style configuration operations through a voice module, etc. The specific implementation method can be determined based on the actual application product.

[0171] To more intuitively understand this solution, please refer to Figures 12 and 13. Referring first to Figure 12, which is a schematic diagram of an interface for obtaining driving style information according to an embodiment of this application, the second driving style information includes parameter values ​​for multiple parameters, including a first parameter used to indicate driving efficiency. The parameter values ​​for each parameter are discrete integer values. It should be understood that the example in Figure 12 is only for ease of understanding this solution and is not intended to limit this solution. Referring then to Figure 13, which is another schematic diagram of an interface for obtaining driving style information according to an embodiment of this application, the second driving style information includes parameter values ​​for multiple parameters, including a first parameter used to indicate driving efficiency. The parameter values ​​for each parameter can be continuous values. It should be understood that the example in Figure 13 is only for ease of understanding this solution and is not intended to limit this solution.

[0172] 1102. Obtain environmental information.

[0173] For example, the intelligent driving system in the vehicle can collect environmental information through sensors in the vehicle (such as the first and second sensors in the embodiment corresponding to Figure 3). The meaning of the environmental information and the collection method can be referred to the description in the embodiment corresponding to Figure 3 above, and will not be repeated here.

[0174] Optionally, the intelligent driving system in the vehicle can also acquire the vehicle's status information. The meaning of the vehicle's status information can be found in the above description of the first status information, and will not be repeated here.

[0175] 1103. Input the second driving style information and environmental information into the machine learning model, obtain the control information through the machine learning model, and match the control information with the driving style indicated by the second driving style information.

[0176] For example, the machine learning model can be a first model that has undergone training. Optionally, if the first feature extraction module in the first model uses a VAE, it is more suitable for scenarios where the driving style information includes continuous values. In other words, if the driving style information includes continuous values, using a VAE to extract features from the driving style information is beneficial for obtaining better first features. Alternatively, the machine learning model can also be a second model.

[0177] Optionally, the intelligent driving system in the vehicle inputs the second driving style information, environmental information, and vehicle state information into a machine learning model to obtain control information. It should be noted that the meanings of the terms in step 1103 and the specific implementation methods of the steps can be found in the description of the embodiment corresponding to Figure 3 above, except that the first driving style information is replaced with the second driving style information, and the first state information is replaced with the vehicle state information; these will not be repeated here.

[0178] In this embodiment, the control information that matches the driving style specified by the user can be generated, which is beneficial for the vehicle to drive according to the driving style specified by the user, to meet the user's personalized needs, and to improve the user experience.

[0179] To more intuitively understand the beneficial effects of the method provided in this application, please refer to Figure 14. Figure 14 is a schematic diagram of the beneficial effects of the method provided in the embodiment of this application. In Figure 14, the driving style information includes the parameter value of the first parameter indicating driving efficiency as an example. As shown in Figure 14, in a traffic scenario where a vehicle passes through an intersection and there are other vehicles ahead, when the value of the first parameter indicating driving efficiency is 10, that is, when the driving style information is efficiency-type, the vehicle accelerates away under the premise of safety, thus achieving high-speed passage; when the value of the first parameter indicating driving efficiency is 1, that is, when the driving style information is comfort-type, the vehicle adopts a more conservative passage mode under the premise of safety, choosing to actively give way. Thus, different driving schemes can be given in the same traffic scenario by combining driving style information, which is conducive to meeting the personalized needs of users. It should be noted that the method provided in this application is under the premise of ensuring safety. For example, please refer to Figure 15. Figure 15 is a schematic diagram of a driving scheme obtained by using the method provided in this application according to an embodiment of this application. As shown in Figure 15, when there is a pedestrian crossing the zebra crossing ahead, regardless of whether the value of the first parameter indicating driving efficiency is 10 or 1, the driving scheme of actively yielding is adopted to ensure the safety of the driving process.

[0180] Based on the embodiments corresponding to Figures 1 to 15, in order to better implement the above-described solutions of the embodiments of this application, related equipment for implementing the above-described solutions is also provided below. Specifically, referring to Figure 16, Figure 16 is a schematic diagram of a model training device provided in the embodiments of this application. The model training device 1600 includes: an acquisition module 1601, used to acquire training data, the training data including training samples and expected regulation information corresponding to the training samples, the training samples including first driving style information and environmental information, the first driving style information indicating the driving style of the vehicle, and the expected regulation information matching the driving style indicated by the first driving style information; a processing module 1602, used to input the training samples into a first model, and obtain the predicted regulation information corresponding to the training samples through the first model; and a training module 1603, used to train the first model using a loss function, the loss function indicating the similarity between the predicted regulation information and the expected regulation information.

[0181] Optionally, the training samples also include vehicle state information; the processing module 1602 is specifically used to extract features from the first driving style information to obtain a first feature, extract features from the vehicle state information to obtain a second feature, and obtain the prediction and control information corresponding to the training samples through the first model based on the environmental information, the first feature and the second feature.

[0182] Optionally, the process of obtaining the predicted control information through the first model by the processing module 1602 includes: fusing the first feature and the second feature to obtain the fused feature; the processing module 1602 is also used to obtain the predicted state information of the vehicle through the first decoder based on the fused feature, and the loss function also indicates the similarity between the predicted state information and the state information.

[0183] Optionally, the process of obtaining the predicted control information through the first model by the processing module 1602 includes: fusing the first feature and the second feature to obtain the fused feature; the training data also includes the correct behavior information of the vehicle, and the processing module 1602 is also used to obtain the predicted behavior information of the vehicle through the second decoder based on the fused feature, and the loss function also indicates the similarity between the predicted behavior information and the correct behavior information.

[0184] Optionally, the processing module 1602 is further configured to obtain predicted driving style information based on the first feature through a third decoder, and the loss function further indicates the similarity between the predicted driving style information and the first driving style information.

[0185] Optionally, the processing module 1602 is specifically used for: extracting features from the first driving style information using a first feature extraction module to obtain a first feature; extracting features from the vehicle's state information using a second feature extraction module to obtain a second feature; and extracting features from the environmental information using a third feature extraction module to obtain a third feature, wherein the first feature extraction module, the second feature extraction module, and the third feature extraction module are different modules in the first model; and obtaining the prediction and control information corresponding to the training samples through the first model based on the first feature, the second feature, and the third feature.

[0186] Optionally, the training module 1603 is specifically used to train the first model using a loss function until the convergence condition is met, to obtain the trained first model, which includes a trained first feature extraction module; the processing module 1602 is further used to extract features from each of the multiple driving style information through the trained first feature extraction module, to obtain the features of each of the multiple driving style information, where the first driving style information is one of the multiple driving style information; wherein, the second model is deployed to the execution device, and the second model includes a feature set and modules other than the first feature extraction module in the first model, the feature set including multiple driving style information and the features of each of the multiple driving style information, and the feature set is used for the execution device to obtain the features of the driving style information.

[0187] Optionally, the first driving style information includes parameter values ​​that indicate the driving efficiency of the vehicle.

[0188] Optionally, the first driving style information matched with the desired control information is obtained from multiple driving style information based on the score of the desired control information, and the score of the desired control information is obtained after data analysis of the desired control information using preset indicators.

[0189] Optionally, different preset indicators may be used in different traffic scenarios.

[0190] It should be noted that the information interaction and execution process between the modules / units in the model training device 1600 are based on the same concept as the various method embodiments corresponding to Figures 1 to 15 in this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0191] Please refer to Figure 17. Figure 17 is a schematic diagram of a control information acquisition device provided in an embodiment of this application. The control information acquisition device 1700 includes: an acquisition module 1701, used to acquire second driving style information and environmental information, wherein the second driving style information indicates the driving style of the vehicle and is obtained based on user operation; and a processing module 1702, used to input the second driving style information and environmental information into a machine learning model, and obtain control information through the machine learning model, wherein the control information matches the driving style indicated by the second driving style information.

[0192] Optionally, the acquisition module 1701 is also used to acquire vehicle state information, and the input of the machine learning model also includes vehicle state information; the processing module 1702 is specifically used to acquire the first feature of the second driving style information, extract the second feature from the vehicle state information, and obtain the prediction and control information corresponding to the training sample through the first model based on the environmental information, the first feature and the second feature.

[0193] Optionally, the machine learning model includes a feature set, which includes multiple driving style information and features of each driving style information, including second driving style information; the processing module 1702 is specifically used to obtain the first feature of the second driving style information from the feature set.

[0194] It should be noted that the information interaction and execution process between the modules / units in the regulatory information acquisition device 1700 are based on the same concept as the various method embodiments corresponding to Figures 1 to 15 in this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0195] The following describes a device provided in an embodiment of this application. Please refer to Figure 18, which is a schematic diagram of the structure of a device provided in an embodiment of this application. Optionally, the device 1800 performs the functions of the training device or vehicle in the various method embodiments corresponding to Figures 1 to 15.

[0196] Device 1800 includes a memory 1802 and at least one processor 1801. Optionally, device 1800 further includes at least one accelerator 1803. Optionally, processor 1801 implements the method in the above embodiments by reading program instructions stored in memory 1802; or, processor 1801 reads program instructions stored in memory 1802 and implements the steps executed by the machine learning model in the method in the above embodiments through accelerator 1803; or, processor 1801 may also implement the method in the above embodiments by reading program instructions stored internally; or, processor 1801 may also read program instructions stored internally and implement the steps executed by the machine learning model in the method in the above embodiments through accelerator 1803.

[0197] When the processor 1801 reads the program instructions stored in the memory 1802 to implement the method in the above embodiments, the memory 1802 stores the program instructions that implement the method provided in the above embodiments of this application.

[0198] Optionally, at least one processor 1801 is one or more CPUs, either a single-core CPU or a multi-core CPU. For example, the memory 1802 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, or optical memory. The memory 1802 stores program instructions for the operating system. For example, at least one accelerator 1803 may include at least one of the following: GPU, NPU, TPU, ASIC, FPGA, or other types of accelerators. After the program instructions stored in the memory 1802 are read by the at least one processor 1801, the device 1800 executes the corresponding operations in the foregoing embodiments.

[0199] Optionally, the device 1800 also includes a network interface 1804, which can be a wired interface or a wireless interface. The network interface 1804 is used to send and receive data in the various method embodiments corresponding to Figures 1 to 15.

[0200] It should be understood that network interface 1804 has the functions of receiving and sending data. The functions of "receiving data" and "sending data" can be integrated into the same transceiver interface, or the functions of "receiving data" and "sending data" can be implemented in different interfaces, without limitation here. In other words, network interface 1804 may include one or more interfaces for implementing the functions of "receiving data" and "sending data".

[0201] After the processor 1801 reads the program instructions from the memory 1802, other functions that the device 1800 can perform are described in the preceding method embodiments.

[0202] Optionally, the device 1800 also includes a bus 1805, through which the processor 1801 and memory 1802 are typically interconnected, or in other ways.

[0203] The device 1800 provided in this application embodiment is used to execute the methods of training devices or vehicles in the above-described method embodiments and achieve the corresponding beneficial effects. The specific implementation of the device 1800 shown in Figure 18 can be referred to the descriptions in the foregoing method embodiments, and will not be repeated here.

[0204] This application also provides a vehicle, as shown in Figure 19. Figure 19 is a structural schematic diagram of a vehicle provided in this application embodiment. The vehicle 100 is configured for fully or partially automated driving mode. For example, the vehicle 100 can control itself while in automated driving mode, and can determine the current state of the vehicle and its surrounding environment through human operation, determine the possible behavior of at least one other vehicle in the surrounding environment, and determine the confidence level corresponding to the probability of other vehicles performing possible behaviors, and control the vehicle 100 based on the determined information. When the vehicle 100 is in automated driving mode, the vehicle 100 can also be set to operate without human interaction.

[0205] Vehicle 100 may include various subsystems, such as a mobility system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 112, and a user interface 116. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0206] The mobility system 102 may include components that provide powered motion to the vehicle 100. In one embodiment, the mobility system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels / tires 121.

[0207] Engine 118 can be an internal combustion engine, an electric motor, an air-compressed engine, or other combinations of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. Engine 118 converts energy source 119 into mechanical energy. Examples of energy source 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 119 can also provide energy to other systems of vehicle 100. Transmission 120 transmits mechanical power from engine 118 to wheels 121. Transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, transmission 120 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 121.

[0208] Sensor system 104 may include several sensors for sensing information about the environment surrounding vehicle 100. For example, sensor system 104 may include a positioning system 122 (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU) 124, a radar 126, a laser rangefinder 128, and a camera 130. Sensor system 104 may also include sensors for the internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensing data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of the autonomous vehicle 100.

[0209] The positioning system 122 can be used to estimate the geographical location of the vehicle 100. An IMU 124 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In one embodiment, the IMU 124 can be a combination of an accelerometer and a gyroscope. A radar 126 can use radio signals to sense objects in the surrounding environment of the vehicle 100, specifically millimeter-wave radar or lidar. In some embodiments, in addition to sensing objects, the radar 126 can also be used to sense the speed and / or direction of travel of objects. A laser rangefinder 128 can use lasers to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. A camera 130 can be used to capture multiple images of the surrounding environment of the vehicle 100. The camera 130 can be a still camera or a video camera.

[0210] The control system 106 controls the operation of the vehicle 100 and its components. The control system 106 may include various components, including a steering system 132, a throttle 134, a braking unit 136, a computer vision system 140, a trajectory control system 142, and an obstacle avoidance system 144.

[0211] The steering system 132 is operable to adjust the forward direction of the vehicle 100. For example, in one embodiment, it may be a steering wheel system. The throttle 134 controls the operating speed of the engine 118 and thus the speed of the vehicle 100. The braking unit 136 controls the deceleration of the vehicle 100. The braking unit 136 may use friction to slow down the wheels 121. In other embodiments, the braking unit 136 may convert the kinetic energy of the wheels 121 into electrical current. The braking unit 136 may also take other forms to slow down the rotational speed of the wheels 121 to control the speed of the vehicle 100. The computer vision system 140 is operable to process and analyze images captured by the camera 130 to identify objects and / or features in the environment surrounding the vehicle 100. The objects and / or features may include traffic signals, road boundaries, and obstacles. The computer vision system 140 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 140 may be used to map the environment, track objects, estimate the speed of objects, etc. The route control system 142 is used to determine the driving route and speed of the vehicle 100. In some embodiments, the route control system 142 may include a lateral planning module 1421 and a longitudinal planning module 1422, which are respectively used to combine data from the obstacle avoidance system 144, GPS 122, and one or more predetermined maps to determine the driving route and speed for the vehicle 100. The obstacle avoidance system 144 is used to identify, evaluate, and avoid or otherwise traverse obstacles in the environment of the vehicle 100, which may specifically be physical obstacles and virtual moving bodies that may collide with the vehicle 100. In one example, the control system 106 may add or alternatively include components other than those shown and described. Alternatively, some of the components shown above may be reduced.

[0212] Vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users via peripheral device 108. Peripheral device 108 may include wireless communication system 146, on-board computer 148, microphone 150, and / or speaker 152. In some embodiments, peripheral device 108 provides a means for a user of vehicle 100 to interact with user interface 116. For example, on-board computer 148 may provide information to a user of vehicle 100. User interface 116 may also operate on-board computer 148 to receive user input. On-board computer 148 may be operated via a touchscreen. In other cases, peripheral device 108 may provide a means for vehicle 100 to communicate with other devices located within the vehicle. For example, microphone 150 may receive audio (e.g., voice commands or other audio input) from a user of vehicle 100. Similarly, speaker 152 may output audio to a user of vehicle 100. Wireless communication system 146 may communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. The wireless communication system 146 may utilize a wireless local area network (WLAN) for communication. In some embodiments, the wireless communication system 146 may utilize an infrared link, Bluetooth, or ZigBee to communicate directly with the device. Other wireless protocols, such as various vehicle communication systems, may also be used. For example, the wireless communication system 146 may include one or more dedicated short-range communications (DSRC) devices that can enable public and / or private data communication between the vehicle and / or a roadside station.

[0213] Power source 110 can provide power to various components of vehicle 100. In one embodiment, power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more such battery packs can be configured to provide power to various components of vehicle 100. In some embodiments, power source 110 and energy source 119 can be implemented together, as is the case in some fully electric vehicles.

[0214] Some or all of the functions of vehicle 100 are controlled by computer system 112. Computer system 112 may include at least one processor 113, which executes program instructions 115 stored in a non-transitory computer-readable medium such as memory 114. Computer system 112 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner. Processor 113 may include any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, processor 113 may also include a dedicated device such as a GPU, NPU, TPU, ASIC, FPGA, or other hardware-based processor. Although Figure 19 functionally illustrates the processor, memory, and other components of computer system 112 in the same block, those skilled in the art will understand that the processor or memory may actually include multiple processors or memories not stored in the same physical housing. For example, memory 114 may be a hard disk drive or other storage medium located in a housing different from that of computer system 112. Therefore, references to processor 113 or memory 114 will be understood to include a collection of processors or memories that may or may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as steering and deceleration components, can each have their own processor that performs only calculations related to the component's specific function.

[0215] In all the aspects described herein, processor 113 may be located remotely from vehicle 100 and may communicate wirelessly with vehicle 100. In other aspects, some of the processes described herein are executed on processor 113 located within vehicle 100, while others are executed by remote processor 113, including taking the necessary steps to perform a single operation.

[0216] In some embodiments, memory 114 may contain instructions 115 (e.g., program logic) that can be executed by processor 113 to perform various functions of vehicle 100, including those described above. Memory 114 may also contain additional program instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the mobility system 102, sensor system 104, control system 106, and peripheral devices 108. In addition to instructions 115, memory 114 may also store data such as road maps, route information, vehicle position, direction, speed, and other such vehicle data, as well as other information. This information may be used by vehicle 100 and computer system 112 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes. A user interface 116 is provided to or receives information from a user of vehicle 100. Optionally, user interface 116 may include one or more input / output devices within the set of peripheral devices 108, such as wireless communication system 146, on-board computer 148, microphone 150, and speaker 152.

[0217] Computer system 112 can control the functions of vehicle 100 based on input received from various subsystems (e.g., driving system 102, sensor system 104, and control system 106) and from user interface 116. For example, computer system 112 can utilize input from control system 106 to control steering system 132 to avoid obstacles detected by sensor system 104 and obstacle avoidance system 144. In some embodiments, computer system 112 is operable to provide control over many aspects of vehicle 100 and its subsystems.

[0218] Alternatively, one or more of these components may be installed separately from or associated with vehicle 100. For example, memory 114 may exist partially or completely separately from vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0219] Optionally, the above components are merely examples. In practical applications, components in each of the above modules may be added or removed as needed. Figure 19 should not be construed as a limitation on the embodiments of this application. A vehicle traveling on a road, such as vehicle 100 above, can identify objects in its surrounding environment to determine adjustments to its current speed. These objects can be other vehicles, traffic control equipment, or other types of objects. In some examples, each identified object can be considered independently, and based on the object's individual characteristics, such as its current speed, acceleration, and distance from the vehicle, the speed adjustment to be made by the vehicle can be determined.

[0220] Optionally, vehicle 100 or computing devices associated with vehicle 100, such as computer system 112, computer vision system 140, and memory 114 as shown in Figure 19, can predict the behavior of the identified objects based on the characteristics of the identified objects and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of each other, so all identified objects can also be considered together to predict the behavior of a single identified object. Vehicle 100 can adjust its speed based on the predicted behavior of the identified objects. In other words, vehicle 100 can determine what steady state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the objects. In this process, other factors can also be considered in determining the speed of vehicle 100, such as the lateral position of vehicle 100 in the road, the curvature of the road, the proximity of static and dynamic objects, etc. In addition to providing program instructions to adjust the speed of the vehicle, the computing device may also provide program instructions to modify the steering angle of the vehicle 100 so that the vehicle 100 follows a given trajectory and / or maintains a safe lateral and longitudinal distance from objects near the vehicle 100 (e.g., cars in adjacent lanes on the road).

[0221] In this embodiment, the processor 113 in the vehicle 100 is used to execute the methods performed by the vehicle in the embodiments corresponding to Figures 1 to 15. It should be noted that the specific manner in which the processor 113 executes the aforementioned steps is based on the same concept as the method embodiments corresponding to Figures 1 to 15 in this application, and the resulting technical effects are the same as those in the method embodiments corresponding to Figures 1 to 15 in this application. For details, please refer to the descriptions in the method embodiments shown above in this application; they will not be repeated here.

[0222] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the steps performed by the training device or vehicle in the methods described in the embodiments shown in Figures 1 to 15.

[0223] This application also provides a computer program product that includes a program that, when run on a computer, causes the computer to perform the steps performed by the training device or vehicle in the methods described in the embodiments shown in Figures 1 to 15 above.

[0224] This application also provides a circuit system including a processing circuit configured to perform the steps performed by the training device or vehicle in the methods described in the embodiments shown in Figures 1 to 15 above.

[0225] The training device, model training apparatus, or control information acquisition device provided in this application embodiment can specifically be a chip. The chip includes a processing unit, which may be, for example, a processor. Optionally, the chip also includes a communication unit, which may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip to execute the methods described in the embodiments shown in Figures 1 to 15 above. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. The storage unit can also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0226] The processor mentioned above can be a general-purpose central processing unit, microprocessor, GPU, NPU, TPU, ASIC, FPGA, or one or more integrated circuits used to control the execution of the program in the first aspect of the above method.

[0227] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0228] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CLUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0229] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0230] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A method for training a model, characterized in that, The method includes: Acquire training data, which includes training samples and expected regulation information corresponding to the training samples. The training samples include first driving style information and environmental information. The first driving style information indicates the driving style of the vehicle, and the expected regulation information matches the driving style indicated by the first driving style information. The training samples are input into the first model, and the prediction and control information corresponding to the training samples is obtained through the first model. The first model is trained using a loss function that indicates the similarity between the predicted regulatory information and the expected regulatory information.

2. The method according to claim 1, characterized in that, The training samples also include vehicle state information. The step of inputting the training samples into a first model and obtaining prediction and control information corresponding to the training samples through the first model includes: The first feature is obtained by extracting features from the first driving style information; The second feature is obtained by extracting features from the vehicle's state information; Based on the environmental information, the first feature, and the second feature, the predictive control information corresponding to the training sample is obtained through the first model.

3. The method according to claim 2, characterized in that, The process of obtaining the prediction and control information through the first model includes: fusing the first feature and the second feature to obtain the fused feature; The method further includes: obtaining the predicted state information of the vehicle through a first decoder based on the fused features, wherein the loss function also indicates the similarity between the predicted state information and the state information.

4. The method according to claim 2, characterized in that, The process of obtaining the prediction and control information through the first model includes: fusing the first feature and the second feature to obtain the fused feature; The training data also includes correct behavioral information of the vehicle, and the method further includes: Based on the fused features, the predicted behavior information of the vehicle is obtained through a second decoder. The loss function also indicates the similarity between the predicted behavior information and the correct behavior information.

5. The method according to any one of claims 2 to 4, characterized in that, The method further includes: Based on the first feature, the predicted driving style information is obtained through a third decoder. The loss function also indicates the similarity between the predicted driving style information and the first driving style information.

6. The method according to any one of claims 2 to 4, characterized in that, The step of extracting features from the first driving style information to obtain the first feature includes: using a first feature extraction module to extract features from the first driving style information to obtain the first feature; The step of extracting features from the vehicle's state information to obtain the second feature includes: using a second feature extraction module to extract features from the vehicle's state information to obtain the second feature; The step of obtaining prediction and control information corresponding to the training samples through the first model based on the environmental information, the first feature, and the second feature includes: A third feature is obtained by using a third feature extraction module to extract features from the environmental information, wherein the first feature extraction module, the second feature extraction module and the third feature extraction module are different modules in the first model; Based on the first feature, the second feature, and the third feature, the prediction and control information corresponding to the training sample is obtained through the first model.

7. The method according to claim 6, characterized in that, The step of training the first model using a loss function includes: training the first model using the loss function until the convergence condition is met, thereby obtaining a trained first model, wherein the trained first model includes a trained first feature extraction module. The method further includes: extracting features from each type of driving style information in the multiple driving style information through the trained first feature extraction module to obtain the features of each type of driving style information in the multiple driving style information, wherein the first driving style information is one of the multiple driving style information; The second model is deployed in the execution device. The second model includes a feature set and modules other than the first feature extraction module in the first model. The feature set includes the multiple driving style information and the features of each driving style information. The feature set is used to provide the execution device with the features of the driving style information.

8. The method according to any one of claims 1 to 4, characterized in that, The first driving style information includes parameter values ​​that indicate the driving efficiency of the vehicle.

9. The method according to any one of claims 1 to 4, characterized in that, The first driving style information that matches the expected control information is obtained from multiple driving style information based on the score of the expected control information. The score of the expected control information is obtained by performing data analysis on the expected control information using preset indicators.

10. The method according to claim 9, characterized in that, The preset indicators used vary depending on the traffic scenario.

11. A method for acquiring regulatory information, characterized in that, The method includes: Acquire second driving style information and environmental information, wherein the second driving style information indicates the vehicle's driving style and is obtained based on user operation; The second driving style information and the environmental information are input into a machine learning model, and the control information is obtained through the machine learning model. The control information is matched with the driving style indicated by the second driving style information.

12. The method according to claim 11, characterized in that, The method further includes: The machine learning model also includes the vehicle's status information as input to the machine learning model. The step of inputting the second driving style information and the environmental information into a machine learning model, and obtaining regulatory information through the machine learning model, includes: The first feature for obtaining the second driving style information; The second feature is obtained by extracting features from the vehicle's state information; Based on the environmental information, the first feature, and the second feature, the predictive control information corresponding to the training sample is obtained through the first model.

13. The method according to claim 12, characterized in that, The machine learning model includes a feature set, which includes multiple driving style information and features of each driving style information, including the second driving style information. The step of obtaining the first feature includes: obtaining the first feature of the second driving style information from the feature set.

14. The method according to claim 12 or 13, characterized in that, The step of extracting features from the vehicle's state information to obtain the second feature includes: using a second feature extraction module to extract features from the vehicle's state information to obtain the second feature; The step of obtaining prediction and control information corresponding to the training samples through the first model based on the environmental information, the first feature, and the second feature includes: A third feature is obtained by using a third feature extraction module to extract features from the environmental information, wherein the first feature extraction module, the second feature extraction module and the third feature extraction module are different modules in the first model; Based on the first feature, the second feature, and the third feature, the prediction and control information corresponding to the training sample is obtained through the first model.

15. A training device for a model, characterized in that, The device includes: An acquisition module is used to acquire training data, the training data including training samples and expected regulation information corresponding to the training samples, the training samples including first driving style information and environmental information, the first driving style information indicating the driving style of the vehicle, and the expected regulation information matching the driving style indicated by the first driving style information. The processing module is used to input the training samples into the first model and obtain the prediction and control information corresponding to the training samples through the first model; The training module is used to train the first model using a loss function, which indicates the similarity between the predicted regulatory information and the expected regulatory information.

16. The apparatus according to claim 15, characterized in that, The training samples also include vehicle status information; The processing module is specifically used to extract features from the first driving style information to obtain a first feature, extract features from the vehicle state information to obtain a second feature, and obtain prediction and control information corresponding to the training sample through the first model based on the environmental information, the first feature and the second feature.

17. The apparatus according to claim 16, characterized in that, The process by which the processing module obtains the prediction and control information through the first model includes: fusing the first feature and the second feature to obtain the fused feature; The processing module is further configured to obtain the predicted state information of the vehicle through a first decoder based on the fused features, and the loss function further indicates the similarity between the predicted state information and the state information.

18. The apparatus according to claim 16, characterized in that, The process by which the processing module obtains the prediction and control information through the first model includes: fusing the first feature and the second feature to obtain the fused feature; The training data also includes the vehicle's correct behavior information. The processing module is further configured to obtain the vehicle's predicted behavior information through a second decoder based on the fused features. The loss function also indicates the similarity between the predicted behavior information and the correct behavior information.

19. The apparatus according to any one of claims 16 to 18, characterized in that, The processing module is further configured to obtain predicted driving style information based on the first feature through a third decoder, and the loss function further indicates the similarity between the predicted driving style information and the first driving style information.

20. The apparatus according to any one of claims 16 to 18, characterized in that, The processing module is specifically used for: The first feature is obtained by extracting features from the first driving style information using the first feature extraction module; The second feature is obtained by extracting features from the vehicle's state information using a second feature extraction module. A third feature is obtained by using a third feature extraction module to extract features from the environmental information, wherein the first feature extraction module, the second feature extraction module and the third feature extraction module are different modules in the first model; Based on the first feature, the second feature, and the third feature, the prediction and control information corresponding to the training sample is obtained through the first model.

21. The apparatus according to claim 20, characterized in that, The training module is specifically used to train the first model using the loss function until the convergence condition is met, to obtain the trained first model, which includes a trained first feature extraction module. The processing module is further configured to extract features from each type of driving style information in the multiple driving style information through the trained first feature extraction module, thereby obtaining the features of each type of driving style information in the multiple driving style information, wherein the first driving style information is one of the multiple driving style information; The second model is deployed in the execution device. The second model includes a feature set and modules other than the first feature extraction module in the first model. The feature set includes the multiple driving style information and the features of each driving style information. The feature set is used to provide the execution device with the features of the driving style information.

22. The apparatus according to any one of claims 15 to 18, characterized in that, The first driving style information includes parameter values ​​that indicate the driving efficiency of the vehicle.

23. The apparatus according to any one of claims 15 to 18, characterized in that, The first driving style information that matches the expected control information is obtained from multiple driving style information based on the score of the expected control information. The score of the expected control information is obtained by performing data analysis on the expected control information using preset indicators.

24. The apparatus according to claim 23, characterized in that, The preset indicators used vary depending on the traffic scenario.

25. A device for acquiring regulatory information, characterized in that, The device includes: The acquisition module is used to acquire second driving style information and environmental information. The second driving style information indicates the driving style of the vehicle and is obtained based on user operation. The processing module is used to input the second driving style information and the environmental information into a machine learning model, and obtain regulation information through the machine learning model. The regulation information is matched with the driving style indicated by the second driving style information.

26. The apparatus according to claim 25, characterized in that, The acquisition module is also used to acquire vehicle status information, and the input of the machine learning model also includes the vehicle status information; The processing module is specifically used to obtain the first feature of the second driving style information, extract the second feature from the vehicle's state information, and obtain the prediction and control information corresponding to the training sample through the first model based on the environmental information, the first feature, and the second feature.

27. The apparatus according to claim 26, characterized in that, The machine learning model includes a feature set, which includes multiple driving style information and features of each driving style information, including the second driving style information. The processing module is specifically used to obtain the first feature of the second driving style information from the feature set.

28. A device, characterized in that, The method includes a processor coupled to a memory storing program instructions, which, when executed by the processor, implement the method of any one of claims 1 to 14.

29. A vehicle, characterized in that, The method includes a processor coupled to a memory storing program instructions that, when executed by the processor, implement the method of any one of claims 11 to 14.

30. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 14.

31. A computer program product, characterized in that, The computer program product includes a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 14.

32. A chip, characterized in that, The chip includes a processor for performing the steps of the method according to any one of claims 1 to 14.

Images