Prediction information acquisition method, model training method, and device

Abstract

A prediction information acquisition method, a model training method, and a device. The acquisition method comprises: on the basis of information of an object in a traffic environment, obtaining prediction information of a first vehicle by means of a model, wherein the first vehicle is the ego vehicle, and the prediction information of the vehicle indicates a trajectory and / or a driving strategy of the vehicle (302); on the basis of the prediction information of the first vehicle, obtaining prediction information of a second vehicle by means of the model, the second vehicle being another vehicle other than the ego vehicle (303); and, on the basis of the prediction information of the second vehicle, obtaining updated prediction information of the first vehicle by means of the model (304). The prediction information of other vehicles is present in an explicit state, such that during model training, the model can be guided by means of a loss function to learn how to obtain correct prediction information of the other vehicles, and in addition, during model inference, interaction processes between the ego vehicle and the other vehicles can also be represented explicitly, thereby enabling the model to obtain better prediction information of the ego vehicle by comprehensively considering the interactions between the ego vehicle and the other vehicles.

Description

A method for acquiring prediction information, a method for training a model, and a device. Technical Field

[0001] This application relates to intelligent driving technology, and more particularly to a method for acquiring predictive information, a method for training a model, and a device. Background Technology

[0002] End-to-end intelligent driving technology helps to raise the upper limit of intelligent driving capabilities. For example, an end-to-end model can directly output the vehicle's prediction information, which includes the vehicle's decisions and / or planned trajectories. However, in the process of generating the vehicle's prediction information through an end-to-end model, the prediction information of other vehicles outside the vehicle often exists in the form of hidden states in the end-to-end model. This implicit representation makes it difficult to guide the model to learn how to obtain the correct prediction information of other vehicles during training through a loss function. Summary of the Invention

[0003] This application provides a method for obtaining prediction information, a method for training a model, and an apparatus. Since the prediction information of other vehicles is explicitly present, it is beneficial to enable the model to learn how to obtain the correct prediction information of other vehicles through the loss function during the training process. Furthermore, it explicitly expresses the interaction process between the self-vehicle and other vehicles, which is beneficial for the model to obtain better prediction information of the self-vehicle after comprehensively considering the situation of other vehicles and the interaction between the self-vehicle and other vehicles.

[0004] This application provides the following technical solution:

[0005] Firstly, this application provides a method for obtaining predictive information, which can be applied in the field of intelligent driving. In this method, after a first vehicle acquires first information including information about objects in the traffic environment, it can obtain predictive information of the first vehicle through a model based on the first information; then, based on the predictive information of the first vehicle, it obtains predictive information of each of at least one second vehicle through a model; finally, based on the predictive information of all the second vehicles, it obtains updated predictive information of the first vehicle through a model. Here, the first vehicle is the vehicle itself, and the second vehicles are different from the first vehicle; in other words, the second vehicles are other vehicles besides the vehicle itself.

[0006] For example, in one scenario, the information about objects in the traffic environment may include attribute information of the objects in the traffic environment, which may include location information of the objects in the traffic environment. In another scenario, the information about objects in the traffic environment may include images and / or point cloud data of the objects in the traffic environment.

[0007] The vehicle prediction information indicates the vehicle's trajectory and / or driving strategy. For example, the prediction information for a first vehicle may include the planned trajectory and / or planned driving strategy for the first vehicle, and the prediction information for each second vehicle may include the predicted trajectory and / or predicted driving strategy for each second vehicle.

[0008] Optionally, the prediction information for the first vehicle may include a planned trajectory of the first vehicle, a score for the planned trajectory, at least one candidate trajectory, and a score for each candidate trajectory. The prediction information for each second vehicle may include a predicted trajectory for each second vehicle. Alternatively, the prediction information for the first vehicle may include a planned driving strategy of the first vehicle, a score for the planned driving strategy, at least one candidate driving strategy, and a score for each candidate driving strategy. The prediction information for each second vehicle may include a predicted driving strategy for each second vehicle.

[0009] In this implementation, after acquiring first information including information about objects in the traffic environment, the model obtains prediction information for the first vehicle (i.e., the vehicle itself) based on the first information. Based on the prediction information of the first vehicle, the model obtains prediction information for the second vehicle (i.e., other vehicles). Based on the prediction information of other vehicles, the model obtains updated prediction information for the vehicle itself. Since the prediction information of other vehicles is explicitly present, it facilitates the model's learning of how to obtain correct prediction information for other vehicles during training through a loss function. Furthermore, in the inference process of the model provided in this application, the prediction information of other vehicles is first obtained based on the prediction information of the vehicle itself, that is, the reactions of other vehicles to the prediction information of the vehicle itself are explicitly shown. Then, based on the reactions of other vehicles to the prediction information of the vehicle itself, the prediction information of the vehicle itself is determined again. In other words, in the inference process of the model provided in this application, the interaction process between the vehicle itself and other vehicles is explicitly expressed, enabling the model to understand the interactive game between the vehicle itself and other vehicles. This helps the model obtain better prediction information for the vehicle itself after comprehensively considering the situation of other vehicles and the interaction between the vehicle itself and other vehicles.

[0010] In one possible implementation, the method provided in this application includes executing K rounds through the model, where K is an integer greater than or equal to 1. Each of the K rounds includes: obtaining prediction information for a second vehicle based on prediction information for a first vehicle, and obtaining prediction information for the first vehicle based on prediction information for the second vehicle. In this implementation, each of the K rounds reflects the interaction and game process between the vehicle and other vehicles. When the value of K is greater than 1, it means that multiple rounds of interaction and game can be executed through the first module, enabling the first model to understand the interaction and game situation of multiple rounds. This facilitates the exploration of better prediction information for the first vehicle after understanding the interaction and game situation of multiple rounds, thereby enabling better control over the first vehicle and improving the user experience for passengers in the first vehicle.

[0011] In one possible implementation, the model includes a first module and a second module. A first vehicle obtains predicted information about itself through the model based on first information, including: the first vehicle extracting features from the first information using the model to obtain first features; based on the first features, making decisions on the driving strategy and / or trajectory of the first vehicle through the first module in the model to obtain predicted information about itself; and updating the input features through the first module to obtain updated first features. The first vehicle obtains updated predicted information about itself through the model based on predicted information from a second vehicle, including: the first vehicle making decisions on the driving strategy and / or trajectory of itself through the first module in the model based on the updated first features and predicted information from the second vehicle to obtain updated predicted information about itself.

[0012] In this implementation, since the first module is used not only to generate prediction information for the first vehicle but also to update features to obtain the updated first feature, when the value of K is greater than 1, the experience of the first and second vehicles in the historical rounds of interaction and game play over K rounds can be passed down by updating the first feature, which is beneficial for obtaining better prediction information for the first vehicle. Furthermore, compared to directly passing down the prediction information of the first and second vehicles used in historical rounds, features often carry richer information. Therefore, passing down the accumulated historical experience of interaction and game play by updating the first feature is beneficial for obtaining better prediction information for the first vehicle.

[0013] In one possible implementation, the first vehicle extracts features from the first information using a model to obtain a first feature, including: the first vehicle extracts features from the first information using a model to obtain the first feature and an initial second feature. Optionally, the method further includes: the second vehicle can also generate an updated second feature based on the initial second feature through a second module. The first vehicle obtains the prediction information of the second vehicle based on the prediction information of the first vehicle through a model, which can include: the first vehicle obtains the prediction information of the second vehicle based on the second feature (the initial second feature or the updated second feature) and the prediction information of the first vehicle through a second module.

[0014] In this implementation, when generating the prediction information of the second vehicle, not only the prediction information of the first vehicle is considered, but also the second feature obtained by feature extraction of the information of objects in the traffic environment is considered. That is, when generating the prediction information of the second vehicle, not only the interaction with the first vehicle is considered, but also the situation of objects in the traffic environment is considered, which is conducive to obtaining more accurate prediction information of the second vehicle.

[0015] Optionally, since the first module is used not only to generate prediction information for the second vehicle but also to update features to obtain updated second features, when K is greater than 1, the experience of the first and second vehicles interacting and playing against each other in K historical rounds can be passed down by updating the second features. This helps the second module provide more accurate prediction information for the second vehicle based on historical interactive game situations, thereby assisting the first module in obtaining better prediction information for the first vehicle. Furthermore, compared to directly transmitting the prediction information of the first and second vehicles used in historical rounds, features often carry richer information. Therefore, transmitting the accumulated historical experience of interactive games by updating the second features helps the second module obtain richer information, which in turn helps the first module obtain better prediction information for the first vehicle.

[0016] In one possible implementation, the first vehicle obtains prediction information for the second vehicle through a second module based on the updated second feature and the prediction information of the first vehicle. This includes: in the k-th round, the first vehicle obtains prediction information for the second vehicle in the k-th round through the second module based on the second feature of the (k-1)-th round and the prediction information of the first vehicle in the (k-1)-th round, where k is any value from 1 to K. The first vehicle then obtains updated prediction information for the first vehicle through a model based on the prediction information of the second vehicle. This includes: in the k-th round, the first vehicle obtains prediction information for the first vehicle in the k-th round through the first module based on the first feature of the (k-1)-th round and the prediction information of the second vehicle in the k-th round.

[0017] In this implementation, in the k-th round (i.e., any one of the K rounds), the prediction information of the first vehicle is obtained through the first module based on the prediction information of the second vehicle in the k-th round. This allows for the generation of prediction information for the vehicle by combining more information about other vehicles besides the vehicle itself, which is more conducive to exploring prediction information for the vehicle with better safety and higher traffic efficiency.

[0018] In one possible implementation, the first vehicle obtains prediction information for the second vehicle through a second module based on the updated second feature and the prediction information of the first vehicle. This includes: in the k-th round, the first vehicle obtains prediction information for the second vehicle in the k-th round based on the second feature of the (k-1)-th round and the prediction information of the first vehicle in the (k-1)-th round, where k is any value from 1 to K. The method further includes: in the k-th round, the first vehicle obtains the first feature of the k-th round based on the first feature of the (k-1)-th round and the prediction information of the second vehicle in the (k-1)-th round, through the second module. The first vehicle then obtains updated prediction information for the first vehicle based on the prediction information of the second vehicle through a model. This includes: in the k+1-th round, the first vehicle obtains prediction information for the first vehicle in the k+1-th round based on the first feature of the k-th round and the prediction information of the second vehicle in the k-th round, through the first module.

[0019] In this implementation, since the vehicle and the surrounding vehicle take actions synchronously during actual driving, in the kth round, the first module and the second module generate their respective prediction information symmetrically. This reasoning method is closer to the interaction mode in the actual driving scenario, which helps to make the interaction game between the first vehicle and the second vehicle closer to the real interaction situation. In turn, it is beneficial to explore better prediction information of the first vehicle based on a more accurate interaction game situation.

[0020] In one possible implementation, the value of K is obtained based on first information. Exemplarily, the value of K is obtained based on the first information and a preset rule; optionally, the preset rule indicates that the value of K is obtained based on at least one of the following: the type of traffic environment, the number of obstacles, or the type of obstacles, etc. Optionally, the preset rule may include second information. After obtaining a specific type of traffic environment, a specific number of obstacles, and / or a specific type of obstacles based on the first information, the value of K corresponding to the aforementioned specific type of traffic environment, specific number of obstacles, and / or specific type of obstacles can be obtained by querying the second information.

[0021] In this implementation, the value of K can be adaptively determined based on the traffic environment, which is beneficial for obtaining a smaller K value in simple traffic environments and a larger K value in complex traffic environments. This, in turn, facilitates the rapid acquisition of better performance prediction information for the first vehicle in simple traffic environments, and even better prediction information for the first vehicle in complex traffic environments, thereby further improving the user experience for passengers in the first vehicle.

[0022] In one possible implementation, the attribute information of objects in the traffic environment may include the location information of the objects in the traffic environment. Objects in the traffic environment may include dynamic objects and static objects. For example, dynamic objects may include dynamic obstacles, and dynamic obstacles include at least a second vehicle. Static objects may include roads. For example, the attribute information of dynamic objects in the traffic environment may include the location information of the dynamic objects. The attribute information of static objects in the traffic environment may include the location information of the static objects; the location information of roads in the traffic environment may include the locations of points in the road topology that make up the road, and the location information of roads can also be understood as the topological information of roads.

[0023] This implementation clarifies the meaning of information about objects in the traffic environment, which helps to improve the integration of this solution with specific application scenarios.

[0024] Secondly, this application provides a model training method applicable to the field of intelligent driving. In this method, a training device acquires first information, including information about objects in a traffic environment; based on the first information, a model is used to obtain prediction information for a first vehicle, wherein the first vehicle is a self-driving vehicle, and the vehicle's prediction information indicates the vehicle's trajectory and / or driving strategy; based on the first vehicle's prediction information, a model is used to obtain prediction information for a second vehicle, which is different from the first vehicle; based on the second vehicle's prediction information, a model is used to obtain prediction information for the first vehicle; the training device uses a loss function to train the model, wherein the loss function includes a first loss term and a second loss term, the first loss term indicating the similarity between the first vehicle's prediction information and the first vehicle's ground truth value, the second loss term indicating the similarity between the second vehicle's prediction information and the second vehicle's ground truth value, and the vehicle's ground truth value indicating the vehicle's correct trajectory and / or the vehicle's correct driving strategy.

[0025] In one possible implementation, the loss function further includes a third loss term, which indicates at least one of the following obtained based on the prediction information of the first vehicle and / or the prediction information of the second vehicle: the safety of the first vehicle, the traffic efficiency of the first vehicle, the safety of the second vehicle, or the traffic efficiency of the second vehicle.

[0026] In this implementation, the loss function also includes a third loss term. The purpose of training the first model using the third loss term is to enable the first model to learn the ability to generate prediction information that is safer and more efficient in traffic. Then, the prediction information of the first vehicle obtained by the trained model can control the first vehicle to drive in a safer and more efficient manner.

[0027] In the second aspect of this application, the training equipment is also used to perform the steps performed by the first vehicle in the first aspect and various possible implementations of the first aspect. The specific implementations of the steps in the second aspect, the meanings of the terms, and the beneficial effects are all described in the first aspect and will not be repeated here.

[0028] Thirdly, this application provides a predictive information acquisition device that can be used in the field of intelligent driving. The device includes: an acquisition unit for acquiring first information, the first information including information about objects in a traffic environment; a processing unit for obtaining predictive information of a first vehicle through a model based on the first information, wherein the first vehicle is a self-driving vehicle, and the predictive information of the vehicle indicates the vehicle's trajectory and / or the vehicle's driving strategy; the processing unit is further configured to obtain predictive information of a second vehicle through a model based on the predictive information of the first vehicle, the second vehicle being different from the first vehicle; and the processing unit is further configured to obtain updated predictive information of the first vehicle through a model based on the predictive information of the second vehicle.

[0029] In the third aspect of this application, the predictive information acquisition device is also used to perform the steps performed by the first vehicle in the first aspect and various possible implementations of the first aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the third aspect can all be found in the first aspect, and will not be repeated here.

[0030] Fourthly, this application provides a model training device for use in the field of intelligent driving. The device includes: an acquisition unit for acquiring first information, the first information including information about objects in a traffic environment; a processing unit for obtaining prediction information of a first vehicle through a model based on the first information, wherein the first vehicle is a self-driving vehicle, and the prediction information of the vehicle indicates the vehicle's trajectory and / or the vehicle's driving strategy; the processing unit is further configured to obtain prediction information of a second vehicle through a model based on the prediction information of the first vehicle, the second vehicle being different from the first vehicle; the processing unit is further configured to obtain prediction information of the first vehicle through a model based on the prediction information of the second vehicle; and a training unit for training the model using a loss function, wherein the loss function includes a first loss term and a second loss term, the first loss term indicating the similarity between the prediction information of the first vehicle and the ground truth value of the first vehicle, the second loss term indicating the similarity between the prediction information of the second vehicle and the ground truth value of the second vehicle, and the ground truth value of the vehicle indicating the correct trajectory of the vehicle and / or the correct driving strategy of the vehicle.

[0031] In the fourth aspect of this application, the training device for the model is also used to perform the steps executed by the training device in the second aspect and various possible implementations of the second aspect. The specific implementation methods, the meanings of the terms, and the beneficial effects of the steps in the fourth aspect can be found in the first aspect, and will not be repeated here.

[0032] Fifthly, this application provides an apparatus including a processor and a memory, the processor being coupled to the memory, the memory storing program instructions, which, when executed by the processor, implement the methods described in the first or second aspect.

[0033] In a sixth aspect, this application provides a vehicle including a processor and a memory, the processor being coupled to the memory, the memory storing program instructions, and the method described in the first aspect being implemented when the program instructions stored in the memory are executed by the processor.

[0034] In a seventh aspect, this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first or second aspect.

[0035] Eighthly, this application provides a computer program product comprising a program that, when run on a computer, causes the computer to perform the methods described in the first or second aspect.

[0036] Ninthly, this application provides a chip system including a processor for supporting the implementation of the functions involved in the foregoing aspects, such as transmitting or processing data and / or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the terminal device or communication device. This chip system may be composed of chips or may include chips and other discrete devices.

[0037] The second to ninth aspects of this application correspond to the first aspect or multiple possible ways of the first aspect, and have corresponding beneficial effects. Attached Figure Description

[0038] Figure 1 is a structural diagram of an artificial intelligence main framework provided in this application;

[0039] Figure 2 is a system architecture diagram of the prediction information acquisition system provided in an embodiment of this application;

[0040] Figure 3 is a flowchart illustrating a method for obtaining prediction information provided in an embodiment of this application.

[0041] Figure 4 is a schematic diagram of the second information provided in an embodiment of this application;

[0042] Figure 5 is a schematic diagram of a process for obtaining the value of K based on the first information according to an embodiment of this application;

[0043] Figure 6 is a schematic diagram of a first model provided in an embodiment of this application;

[0044] Figure 7 is a schematic diagram of a first model provided in an embodiment of this application;

[0045] Figure 8 is another schematic diagram of the method for obtaining prediction information provided in the embodiments of this application;

[0046] Figure 9 is another schematic diagram of the first model provided in the embodiment of this application;

[0047] Figure 10 is another schematic diagram of the method for obtaining prediction information provided in the embodiments of this application;

[0048] Figure 11 is a flowchart illustrating a training method for a model provided in an embodiment of this application;

[0049] Figure 12 is another schematic diagram of the training method of the model provided in the embodiment of this application;

[0050] Figure 13 is a schematic diagram of a predictive information acquisition device provided in an embodiment of this application;

[0051] Figure 14 is a schematic diagram of a training device for a model provided in an embodiment of this application;

[0052] Figure 15 is a schematic diagram of a device provided in an embodiment of this application;

[0053] Figure 16 is a structural schematic diagram of a vehicle provided in an embodiment of this application. Detailed Implementation

[0054] The embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0055] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0056] In the embodiments of this application, "instruction" can include direct and indirect instructions, as well as explicit and implicit instructions. The information indicated by a certain piece of information (hereinafter referred to as instruction information) is called the information to be instructed. In specific implementation, there are many ways to indicate the information to be instructed, such as, but not limited to, directly indicating the information to be instructed, such as the information to be instructed itself or its index. It can also indirectly indicate the information to be instructed by indicating other information, where there is an association between the other information and the information to be instructed; or it can indicate only a part of the information to be instructed, while the other parts are known or pre-agreed upon. For example, the instruction can be implemented by using a pre-agreed (e.g., protocol predefined) arrangement of various information, thereby reducing the instruction overhead to a certain extent. This application does not limit the specific method of instruction. It is understood that for the sender of the instruction information, the instruction information can be used to indicate the information to be instructed; for the receiver of the instruction information, the instruction information can be used to determine the information to be instructed.

[0057] First, the overall workflow of the artificial intelligence system is described, as shown in Figure 1. Figure 1 is a structural diagram of one aspect of the artificial intelligence framework provided in this application. The framework is then elaborated on from two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0058] (1) Infrastructure

[0059] The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips, which can specifically employ hardware acceleration chips such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), tensor processing units (TPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0060] (2) Data

[0061] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0062] (3) Data processing

[0063] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0064] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0065] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0066] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0067] (4) General ability

[0068] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0069] (5) Smart Products and Industry Applications

[0070] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent terminals, intelligent manufacturing, intelligent transportation, smart homes, intelligent healthcare, intelligent security, intelligent driving, and smart cities.

[0071] The method provided in this application can be applied to the field of intelligent driving. For example, the method provided in this application can be used in application scenarios to obtain predictive information about vehicles. The vehicles can be cars, trucks, motorcycles, buses, boats, lawnmowers, recreational vehicles, amusement park vehicles, construction equipment, trams, golf carts, trains, airplanes, and helicopters, etc., and this application does not impose any particular limitation.

[0072] The vehicle prediction information in this application may include: the vehicle's trajectory and / or the vehicle's driving strategy. For example, the vehicle's trajectory may include: the vehicle's position corresponding to each of at least one time points after the current time point, where the vehicle's position can also be understood as the vehicle's trajectory points. The vehicle's driving strategy may include: the vehicle's driving strategy corresponding to each of at least one time points after the current time point.

[0073] The driving strategy for a vehicle at each moment can include the lateral driving strategy and / or longitudinal driving strategy corresponding to each moment. The lateral driving strategy corresponding to a certain moment can be left turn, straight, right turn, lane change or lateral avoidance, etc., and the longitudinal driving strategy corresponding to a certain moment can be acceleration, constant speed or deceleration, etc. The specific manifestation of the driving strategy can be determined in combination with the actual application scenario.

[0074] In related technologies, the prediction information of the first vehicle can be directly output through an end-to-end model. In this application, the first vehicle can be understood as the vehicle itself. However, in the process of generating the prediction information of the first vehicle through an end-to-end model, the prediction information of the second vehicle other than the first vehicle often exists in the form of a hidden state in the end-to-end model. This implicit representation makes it difficult to guide the model to learn how to obtain the correct prediction information of the second vehicle through the loss function during training. In this application, the second vehicle can be understood as other vehicles besides the vehicle itself. In other words, the second vehicle and the first vehicle are different vehicles.

[0075] To address the aforementioned issues, this application discloses the following: After acquiring information about objects in the traffic environment, based on the first information, a model (hereinafter referred to as the "first model") is used to obtain the prediction information of the first vehicle (i.e., the vehicle itself). Based on the prediction information of the first vehicle, the model is used to obtain the prediction information of the second vehicle (i.e., other vehicles). Based on the prediction information of other vehicles, the model is used to obtain the updated prediction information of the vehicle itself. Since the prediction information of other vehicles is explicitly present, it is beneficial to enable the model to learn how to obtain the correct prediction information of other vehicles through a loss function during training. Furthermore, in the reasoning process of the model provided in this application, the prediction information of other vehicles is first obtained based on the prediction information of the self-vehicle, that is, the reaction of other vehicles to the prediction information of the self-vehicle is displayed. Then, based on the reaction of other vehicles to the prediction information of the self-vehicle, the prediction information of the self-vehicle is determined again. In other words, in the reasoning process of the model provided in this application, the interaction process between the self-vehicle and other vehicles is also expressed, so that the model has the ability to understand the interactive game between the self-vehicle and other vehicles. This is conducive to the model obtaining better prediction information of the self-vehicle after comprehensively considering the situation of other vehicles and the interaction between the self-vehicle and other vehicles.

[0076] Before detailing the method provided in this application, the architecture of the prediction information acquisition system provided in this application will be described first. Please refer to Figure 2, which is a system architecture diagram of the prediction information acquisition system provided in an embodiment of this application. In Figure 2, the prediction information acquisition system 200 includes a training device 210, a database 220, an execution device 230, and a data storage system 240. The execution device 230 includes a computing module 231. The database 220 stores a training data set. During the training phase, the training device 210 can use the training data in the training data set to train the first model 201, thereby obtaining the trained first model 201.

[0077] During the application phase, the reasoning process of the first model 201 can be executed by the computing module 231 of the execution device 230. Optionally, as shown in Figure 2, the execution device 230 can be integrated into the vehicle, and the user can directly interact with the vehicle on which the execution device 230 is deployed.

[0078] For example, the execution device 230 can be a module in the vehicle's host CPU that uses the model to process data. The execution device 230 can also be a graphics processing unit (GPU), neural network processing unit (NPU), or tensor processing unit (TPU) in the vehicle, etc. The aforementioned GPU, NPU, or TPU is mounted on the vehicle's host CPU as a coprocessor, and the vehicle's host CPU allocates tasks, etc.

[0079] The execution device 230 can access data, code, etc., in the data storage system 240, and can also store data, instructions, etc., in the data storage system 240. The data storage system 240 can be located within the execution device 230, or it can be an external memory relative to the execution device 230.

[0080] It should be noted that Figure 2 is merely a schematic diagram of one architecture of the prediction information acquisition system provided in this application embodiment, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in some other embodiments of this application, the execution device 230 and the vehicle can be separate and independent devices. The execution device 230 is configured with an input / output (I / O) interface, through which the execution device 230 can interact with the vehicle. For example, in the application phase, the intelligent driving system in the first vehicle can send environmental information or first information to the execution device 230 through the I / O interface. After obtaining the prediction information of the first vehicle through the first model 201 deployed in the calculation module 231, the execution device 230 can send the prediction information to the intelligent driving system in the first vehicle through the I / O interface.

[0081] For example, in some other embodiments of this application, the training device 210 and the execution device 230 may also be integrated into the same device, and the specific architecture of the prediction information acquisition system can be determined according to the actual application scenario. The specific implementation processes of the application stage and the training stage are described below.

[0082] I. Application Phase

[0083] Please refer to Figure 3, which is a flowchart illustrating a method for obtaining prediction information provided in an embodiment of this application. The method for obtaining prediction information provided in an embodiment of this application may include steps 301 to 304, wherein...

[0084] 301. Obtain first information, which includes information about objects in the traffic environment.

[0085] For example, in one scenario, information about objects in a traffic environment may include attribute information about the objects in the traffic environment. The attribute information about objects in the traffic environment may include location information about the objects in the traffic environment.

[0086] For example, objects in a traffic environment can include dynamic objects and static objects. For example, dynamic objects can include dynamic obstacles, which include at least a second vehicle, which is a vehicle other than the first vehicle (i.e., the vehicle itself). Optionally, dynamic objects can also include cyclists, pedestrians, or other types of dynamic obstacles, as determined by the specific application scenario. For example, static objects can include roads; optionally, static objects can also include at least one of the following: road barriers, traffic lights, traffic signs, or other types of static objects, as determined by the specific application scenario.

[0087] The attribute information of dynamic objects in a traffic environment can include the position information of the dynamic objects. Optionally, the attribute information of dynamic objects in a traffic environment can also include the speed, orientation, size, or other attribute information that reflects the motion of the dynamic objects.

[0088] The attribute information of static objects in a traffic environment can include the location information of the static objects. For example, the location information of a road in a traffic environment can include the locations of points in the road topology that make up the road; the location information of a road can also be understood as the topological information of the road. Optionally, the attribute information of a traffic sign can also include the content indicated by the traffic sign, etc. The specific information included in the attribute information of objects in a traffic environment can be determined based on the actual application scenario.

[0089] Optionally, the attribute information of objects in the traffic environment can be obtained by using a second model based on environmental information collected by sensors in the first vehicle. The environmental information includes images and / or point cloud data of the traffic environment surrounding the first vehicle. The images of the traffic environment surrounding the first vehicle can be collected by a first sensor in the first vehicle, and these images can be images from a perspective view (PV). The point cloud data of the traffic environment surrounding the first vehicle can be collected by a second sensor in the first vehicle.

[0090] "Traffic environment around the vehicle" can be understood as the environment within the field of view of the sensors in the vehicle (such as the aforementioned first or second sensor). For example, the first sensor can be a photoelectric sensor, such as a camera or event camera; the second sensor can be an ultrasonic sensor, a lidar sensor, a millimeter-wave radar sensor, or other sensors capable of measuring and obtaining point cloud data, etc., and this application does not exhaustively list them all.

[0091] For example, the first vehicle can input environmental information into the second model, which then processes the environmental information to obtain attribute information of objects in the traffic environment generated by the second model. For example, the second model can be an attention-based neural network, a convolutional neural network, a fully connected neural network, a residual neural network, or other types of neural networks.

[0092] Optionally, the tasks performed by the second model may include: detecting and tracking dynamic objects in the traffic environment based on environmental information, and identifying static objects in the traffic environment.

[0093] Optionally, the task head of the second model may include a first task head and a second task head. The first task head is used to detect and track dynamic objects in the traffic environment and to generate attribute information of dynamic objects in the traffic environment. The second task head is used to identify static objects in the traffic environment and to generate attribute information of static objects in the traffic environment.

[0094] In this embodiment of the application, the meaning of the information of objects in the traffic environment is clarified, which is conducive to improving the degree of integration between this solution and specific application scenarios.

[0095] In another scenario, information about objects in the traffic environment may include images and / or point cloud data of objects in the traffic environment, and this information can be acquired by sensors in the first vehicle.

[0096] Optionally, the first information may further include the state information of the first vehicle, which reflects the physical state of the first vehicle. For example, the state information of the first vehicle includes at least one of the following: velocity, acceleration, yaw rate, steering angle, or pitch angle. Further, the yaw rate may include yaw rate and pitch rate; the specific information included in the state information of the first vehicle can be determined based on the actual application scenario.

[0097] Optionally, step 301 may further include: the first vehicle acquiring navigation information. For example, the navigation information may include navigation instructions, such as: turn left, turn right, make a U-turn, enter a ramp, or other types of instructions, etc., which are not exhaustive examples here.

[0098] Optionally, step 301 may further include: the first vehicle acquiring prediction information of the second vehicle generated by the third model (hereinafter referred to as "reference prediction information of the second vehicle" for ease of distinction). For example, before performing step 301, the first vehicle can input environmental information into the third model to obtain the reference prediction information of the second vehicle output by the third model. For example, the third model can be an attention-based neural network, a convolutional neural network, a fully connected neural network, a residual neural network, or other types of neural networks.

[0099] The tasks performed by the third model may include: predicting the trajectories of other vehicles in the traffic environment based on environmental information, and / or predicting the driving strategies of other vehicles in the traffic environment.

[0100] It should be noted that in this embodiment of the application, the execution subject is described as the first vehicle, and the steps in this embodiment of the application can be understood as being executed by the intelligent driving system in the first vehicle.

[0101] 302. Based on the first information, the prediction information of the first vehicle is obtained through the first model, wherein the first vehicle is a self-driving vehicle, and the prediction information of the vehicle indicates the trajectory of the vehicle and / or the decision of the vehicle.

[0102] For example, the first model may include a backbone module, a first module, and a second model, wherein the backbone module is used to extract features from the first information; the first module is used to generate prediction information for the first vehicle; and the second module is used to generate prediction information for each of at least one second vehicle. Optionally, the task performed by the first module may include making decisions on the driving strategy and / or trajectory of the first vehicle (i.e., the vehicle itself). The task performed by the second module may include predicting the driving strategy and / or trajectory of the second vehicle (i.e., other vehicles). The model structure adopted by the first module and the second module can be determined in combination with the actual application scenario, and will not be exemplified here.

[0103] For example, in one scenario, the prediction information for the first vehicle may include a planned trajectory for the first vehicle, a score for the planned trajectory, at least one candidate trajectory, and a score for each candidate trajectory. The prediction information for each second vehicle may include a predicted trajectory for each second vehicle. In another scenario, the prediction information for the first vehicle may include a planned trajectory for the first vehicle; and the prediction information for each second vehicle may include a predicted trajectory for each second vehicle.

[0104] For example, in one scenario, the prediction information for the first vehicle may include a planned driving strategy for the first vehicle, a score for the planned driving strategy, at least one candidate driving strategy, and a score for each candidate driving strategy. The prediction information for each second vehicle may include a predicted driving strategy for each second vehicle. In another scenario, the prediction information for the first vehicle may include a planned driving strategy for the first vehicle; and the prediction information for each second vehicle may include a predicted driving strategy for each second vehicle.

[0105] The specific information included in the prediction information for the first vehicle and the prediction information for the second vehicle can be determined based on the actual application scenario.

[0106] For example, step 302 may include: the first vehicle extracting features from the first information using the backbone module to obtain a first feature; optionally, the first vehicle extracting features from the first information using the backbone module to obtain an initial first feature and an initial second feature. The difference between the first feature and the second feature includes: the first feature is a feature input to the first module, and the second feature is a feature input to the second module; optionally, the difference between the first feature and the second feature also includes: the first feature and the second feature have different dimensions.

[0107] The first vehicle can make decisions on its driving strategy and / or trajectory based on initial first features (optionally, also including navigation information and reference prediction information of the second vehicle) through the first module, thereby obtaining prediction information for the first vehicle. Optionally, the task performed by the first module may also include updating the input features, so the first vehicle can also generate updated first features based on the initial first features through the first module.

[0108] Optionally, after obtaining the initial first feature and the initial second feature through the backbone module, step 302 may further include: the second vehicle predicting its driving strategy and / or trajectory based on the initial second feature through the second module, thereby obtaining prediction information for the second vehicle. Optionally, the task performed by the second module may further include: updating the input features, so the second vehicle may also generate updated second features based on the initial second feature through the second module.

[0109] 303. Based on the prediction information of the first vehicle, the prediction information of the second vehicle is obtained through the first model. The second vehicle is different from the first vehicle.

[0110] For example, step 303 may include: the first vehicle obtaining the prediction information of the second vehicle through the second module in the first model based on the prediction information of the first vehicle.

[0111] Optionally, step 303 may include: the first vehicle predicting the driving strategy and / or trajectory of the second vehicle through the second module in the first model based on the updated second features and the prediction information of the first vehicle, thereby obtaining the prediction information of the second vehicle; optionally, the first vehicle updating its features through the second module to obtain the updated second features again.

[0112] Alternatively, step 303 may include: the first vehicle predicting the driving strategy and / or trajectory of the second vehicle through the second module in the first model based on the initial second feature and the prediction information of the first vehicle, thereby obtaining the prediction information of the second vehicle.

[0113] In this embodiment of the application, when generating the prediction information of the second vehicle, not only the prediction information of the first vehicle is considered, but also the second feature obtained by feature extraction of the information of objects in the traffic environment is considered. That is, when generating the prediction information of the second vehicle, not only the interaction with the first vehicle is considered, but also the situation of objects in the traffic environment is considered, which is conducive to obtaining more accurate prediction information of the second vehicle.

[0114] Optionally, since the first module is used not only to generate prediction information for the second vehicle but also to update features to obtain updated second features, when K is greater than 1, the experience of the first and second vehicles interacting and playing against each other in K historical rounds can be passed down by updating the second features. This helps the second module provide more accurate prediction information for the second vehicle based on historical interactive game situations, thereby assisting the first module in obtaining better prediction information for the first vehicle. Furthermore, compared to directly transmitting the prediction information of the first and second vehicles used in historical rounds, features often carry richer information. Therefore, transmitting the accumulated historical experience of interactive games by updating the second features helps the second module obtain richer information, which in turn helps the first module obtain better prediction information for the first vehicle.

[0115] Alternatively, if the task performed by the second module does not include updating the input features, step 304 may also include: the first vehicle obtaining the prediction information of the second vehicle through the second module based on the initial second features and the prediction information of the first vehicle.

[0116] 304. Based on the prediction information of the second vehicle, the updated prediction information of the first vehicle is obtained through the first model.

[0117] For example, step 304 may include: the first vehicle obtaining updated prediction information of the first vehicle through the first module in the first model based on the prediction information of the second vehicle (optionally, also including navigation information and reference prediction information of the second vehicle).

[0118] Optionally, step 304 may include: the first vehicle making a decision on the driving strategy and / or trajectory of the first vehicle based on the updated first feature and the prediction information of the second vehicle (optionally, also including navigation information and reference prediction information of the second vehicle), through the first module in the first model, to obtain the updated prediction information of the first vehicle; optionally, the first vehicle performs feature update through the first module to obtain the updated first feature again.

[0119] In this embodiment, since the first module is used not only to generate prediction information for the first vehicle but also to update features to obtain updated first features, when the value of K is greater than 1, the experience of the first vehicle and the second vehicle in interactive games over K rounds can be passed down by updating the first features, which is beneficial for obtaining better prediction information for the first vehicle. Furthermore, compared to directly passing down the prediction information of the first vehicle and the second vehicle used in historical rounds, features often carry richer information. Therefore, passing down the accumulated historical experience of interactive games by updating the first features is beneficial for obtaining better prediction information for the first vehicle.

[0120] Alternatively, if the task performed by the first module does not include updating the input features, step 304 may also include: the first vehicle obtaining the updated prediction information of the first vehicle through the first module based on the initial first features and the prediction information of the second vehicle (optionally, also including navigation information and reference prediction information of the second vehicle).

[0121] For example, the method provided in this application includes executing K rounds through a first model, where K is an integer greater than or equal to 1. Each of the K rounds includes: obtaining prediction information for a second vehicle through a second module based on prediction information for a first vehicle; and obtaining prediction information for a first vehicle through a first module based on prediction information for the second vehicle. For example, the prediction information for the first vehicle obtained in the Kth round can be used as the final updated prediction information for the first vehicle; or, the prediction information for the first vehicle in one round can be selected from the prediction information for all rounds included in the K rounds as the final updated prediction information for the first vehicle. In other words, in the method provided in this application, K rounds of interactive game can be executed through the first model to obtain the final updated prediction information for the first vehicle, which is used to control the first vehicle.

[0122] Optionally, in one case, the value of K is obtained based on the first information. For example, the value of K is obtained based on the first information and a preset rule; alternatively, the preset rule indicates that the value of K is obtained based on at least one of the following: the type of traffic environment, the number of obstacles, or the type of obstacles, etc. For example, the type of traffic environment may include ramp merging, intersection, multi-lane, single-lane, or other types, etc. The number of obstacles may include multiple ranges; for example, multiple ranges may include 1, 2-5, and more than 5 (three ranges); or, for example, multiple ranges may include 1-2, 3-4, 5-6, and more than 6 (four ranges), etc., and the specific situation can be determined according to the actual application scenario. The categories of obstacles may include vehicles, cyclists, pedestrians, mixed categories, or other categories. A mixed category represents obstacles that include at least two of the following: vehicles, cyclists, and pedestrians. For example, a cyclist may include a person riding a two-wheeled electric vehicle and an electric vehicle, or a cyclist may include a person riding a bicycle and a bicycle, or a cyclist may include a person riding a motorcycle and a motorcycle, etc.

[0123] For example, a first vehicle can obtain the category of the traffic environment, the number of obstacles, and the category of the obstacles based on the first information. In one case, if the first information includes attribute information of objects in the traffic environment, the category of the traffic environment can be obtained based on the topological information of the roads in the traffic environment included in the first information; the number of obstacles can be obtained based on the attribute information of the obstacles included in the first information; and the category of obstacles can be obtained based on the attribute information of the obstacles included in the first information. In another case, if the first information includes images and / or point cloud data of objects in the traffic environment, the first vehicle can identify the category of the traffic environment, the number of obstacles, and the category of the obstacles through a neural network layer based on the first information, thereby obtaining the category of the traffic environment, the number of obstacles, and the category of the obstacles.

[0124] Optionally, the preset rules may include second information. After obtaining a specific category of traffic environment, a specific number of obstacles, and / or a specific category of obstacles based on the first information, the value of K corresponding to the aforementioned specific category of traffic environment, specific number of obstacles, and / or specific category of obstacles can be obtained by querying the second information.

[0125] Optionally, if the preset rule indicates that the value of K is based on the traffic environment category, the number of obstacles, and the category of obstacles, the traffic environment category, the number of obstacles, and the category of obstacles can be regarded as keys, the value of K can be regarded as a value, and the second information can be regarded as a key-value table. After obtaining the specific category of traffic environment, the specific number of obstacles, and / or the specific category of obstacles, the value of K corresponding to the aforementioned specific category of traffic environment, the specific number of obstacles, and / or the specific category of obstacles can be obtained by querying the key-value table. To understand this solution more intuitively, please refer to Figures 4 and 5. First, refer to Figure 4. Figure 4 is a schematic diagram of the second information provided in the embodiment of this application. As shown in Figure 4, the second information can be a three-dimensional table. The first dimension is the category of traffic environment. Each category of traffic environment corresponds to a two-dimensional table. In Figure 4, the categories of traffic environment include ramp merging, intersections, and multi-lane roads as examples. Each category of the traffic environment corresponds to a two-dimensional table. The horizontal axis includes multiple ranges of the number of obstacles, and the vertical axis includes multiple categories of obstacles. A value for K is determined by three variables: the category of the traffic environment, the number of obstacles, and the category of obstacles. Figure 4 uses examples where multiple ranges can include a single obstacle, 2-5 obstacles, and more than 5 obstacles, and where obstacle categories can include vehicles, cyclists, and a mix. It should be understood that the examples in Figure 4 are only for illustrative purposes and are not intended to limit the scope of this solution.

[0126] Referring again to Figure 5, which is a flowchart illustrating how to obtain the value of K based on the first information according to an embodiment of this application, as shown in Figure 5, the first vehicle can sequentially determine the specific category of the traffic environment, the specific number of obstacles, and the specific category of obstacles based on the first information. Then, based on the specific category of the traffic environment, the specific number of obstacles, and the specific category of obstacles, a specific value of K can be obtained by querying the second information. It should be understood that the example in Figure 5 is only for the convenience of understanding this solution and is not intended to limit this solution.

[0127] Alternatively, in another scenario, K may be set to a preset value. Optionally, if K is a learnable parameter during the training phase of the first model, then the value of K in the application phase can be obtained from training during the training phase of the first model.

[0128] In this embodiment, the first model executes K rounds, each round including: obtaining prediction information for the second vehicle based on the prediction information of the first vehicle, and obtaining prediction information for the first vehicle based on the prediction information of the second vehicle. That is, each round reflects the interaction and game process between the vehicle and other vehicles. When K is greater than 1, it means that multiple rounds of interaction and game can be executed through the first module, allowing the first model to understand the interaction and game situation across multiple rounds. This understanding helps to explore better prediction information for the first vehicle, thereby enabling better control over the first vehicle and improving the user experience for passengers in the first vehicle.

[0129] Optionally, the value of K can be determined based on first information, which includes information about objects in the traffic environment. That is, this application can adaptively determine the value of K based on the traffic environment, which is beneficial for obtaining a smaller K value in simple traffic environments and a larger K value in complex traffic environments. This further facilitates obtaining better predictive information about the performance of the first vehicle quickly in simple traffic environments and even better predictive information about the first vehicle in complex traffic environments, thereby further improving the user experience for passengers in the first vehicle.

[0130] The relationship between the first module and the second module can be master-slave, or symmetrical, or other relationships, which are not exhaustively listed in this embodiment.

[0131] To understand this solution more intuitively, please refer to Figure 6. Figure 6 is a schematic diagram of a first model provided in an embodiment of this application. Figure 6 uses the vehicle's predicted information, including its trajectory, as an example. The case where the predicted information includes the vehicle's driving strategy can be understood by referring to Figure 6. As shown in Figure 6, the first model may include a backbone module, a first module, a second module, and an interaction iteration number determination module (which can also be understood as a module for determining the value of K). The backbone module is used to extract features from the first information to obtain a first feature and a second feature. The first feature is input to the first module, and the second feature is input to the second module, as shown in Figure 6. The input to the first module may also include navigation information and the reference predicted trajectory of the second vehicle. The interaction iteration number determination module in the first model is used to determine the value of K based on the first information, thereby determining that K rounds need to be executed through the first and second modules. In each of the K rounds, the first module generates the planned trajectory of the first vehicle (i.e., the self-vehicle) based on the predicted trajectory of the second vehicle, and the second module generates the predicted trajectory of the second vehicle (i.e., the other vehicle) based on the planned trajectory of the first vehicle. After executing K rounds, the final planned trajectory of the self-vehicle can be obtained. Therefore, the control module of the first vehicle can control the first vehicle based on the final planned trajectory of the vehicle. It should be understood that the example in Figure 6 is only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0132] In one scenario, the relationship between the first module and the second module is master-slave. Optionally, step 303 may include: in the k-th round, the first vehicle obtains the prediction information of the second vehicle in the k-th round through the second module in the first model based on the second feature of the (k-1)-th round and the prediction information of the first vehicle in the (k-1)-th round, where k is any value from 1 to K, and the k-th round is any one of the K rounds. Step 304 may include: in the k-th round, the first vehicle obtains the prediction information of the first vehicle in the k-th round through the first module in the first model based on the first feature of the (k-1)-th round and the prediction information of the second vehicle in the k-th round (optionally, also including navigation information and reference prediction information of the second vehicle).

[0133] For example, when k is 1, the second feature of the (k-1)th round (i.e. the 0th round) can be the updated second feature or the initial second feature obtained in step 302, and the first feature of the (k-1)th round can be the updated first feature or the initial first feature obtained in step 302.

[0134] For example, the second feature of each round from the first round to the Kth round can be understood as the updated second feature in each round; the first feature of each round from the first round to the Kth round can be understood as the updated first feature in each round.

[0135] To understand this solution more intuitively, please refer to Figures 7 and 8. First, refer to Figure 7, which is a schematic diagram of a first model provided in an embodiment of this application. In Figure 7, the relationship between the first module and the second module is master-slave as an example. As shown in Figure 7, after the first vehicle inputs the first information into the first model, the first feature and the second feature are obtained by feature extraction of the first information through the main module in the first model. Based on the first feature, the planning trajectory of the first vehicle in the 0th round and the first feature of the 0th round are generated by the first module.

[0136] In the first round, the first vehicle, based on its planned trajectory and second feature from the first vehicle in the 0th round, generates the predicted trajectory and second feature of the second vehicle in the first round through the second module. The first vehicle, based on the predicted trajectory and first feature of the second vehicle in the first round and the first feature from the 0th round, generates the planned trajectory and first feature of the first vehicle in the first round through the first module.

[0137] In the k-th round, the first vehicle, based on its planned trajectory and second features from the (k-1)-th round, generates the predicted trajectory and second features for the second vehicle in the k-th round through the second module. The first vehicle, based on the predicted trajectory and first features from the second vehicle in the k-th round, generates its planned trajectory and first features for the k-th round through the first module.

[0138] The first vehicle executes K rounds through the first module and the second module to obtain the planned trajectory of the first vehicle in the Kth round, output by the first model. It should be understood that the example in Figure 7 is only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0139] Referring again to Figure 8, which is another schematic diagram of the prediction information acquisition method provided in the embodiments of this application, Figure 8 takes the master-slave relationship between the first module and the second module as an example. As shown in Figure 8, the first vehicle (i.e., the autonomous vehicle) can input the first information from the perception module into the first model. After the first vehicle extracts features from the first information through the backbone module of the first model, it can obtain the initial first feature and the initial second feature. Based on the first information, the first vehicle determines the value of K through the interaction iteration number determination module of the first model. The K iterations include the first iteration to the Kth iteration.

[0140] In the initial stage (i.e., the 0th round), the first vehicle generates prediction information and the first feature of the 0th round based on the initial first feature through the first module in the first model. The prediction information of the first vehicle in the 0th round includes the planned trajectory of the first vehicle in the 0th round, the score of the planned trajectory, at least one candidate trajectory, and the score of each candidate trajectory.

[0141] In the first round, the first vehicle, based on the prediction information of the first vehicle in round 0 and the initial second features, generates prediction information and second features for each of at least one second vehicle (i.e., other vehicles) in round 1 through the second module of the first model. The prediction information for each second vehicle in round 1 includes the predicted trajectory of each second vehicle in round 1. The first vehicle, based on the prediction information of all second vehicles in round 1 and the first features from round 0, generates prediction information and first features for the first vehicle in round 1 through the first module of the first model.

[0142] In the second round, the first vehicle, based on the prediction information and second features of the first vehicle in the first round, generates prediction information and second features for each of at least one second vehicle in the second round through the first module. The prediction information for each second vehicle in the second round includes the predicted trajectory of each second vehicle in the second round. The first vehicle also generates prediction information for the second round based on the prediction information of all second vehicles in the second round and the updated first features from the first round, through the first module in the first model.

[0143] In the k-th round, the first vehicle, based on the prediction information of the first vehicle in the (k-1)-th round and the second feature of the (k-1)-th round, generates prediction information and the second feature of each of at least one second vehicle in the k-th round through the first module. The prediction information of each second vehicle in the k-th round includes the predicted trajectory of each second vehicle in the k-th round. The first vehicle, based on the prediction information of all second vehicles in the k-th round and the updated first feature of the (k-1)-th round, generates prediction information of the first vehicle in the k-th round through the first module of the first model.

[0144] When the k-th round is the K-th round, the first vehicle can obtain its planned trajectory for the K-th round based on the predicted information generated in the K-th round. This planned trajectory can be output as the first model and sent to the downstream control module. The control module in the first vehicle can control the first vehicle based on its planned trajectory in the K-th round. For example, the first vehicle can generate steering commands, braking commands, and / or drive commands to its chassis and electric drive system based on the planned trajectory in the K-th round. It should be understood that the example in Figure 8 is for illustrative purposes only and is not intended to limit the scope of this solution.

[0145] In this embodiment of the application, in the k-th round (that is, any one of the K rounds), the prediction information of the first vehicle is obtained by the first module based on the prediction information of the second vehicle in the k-th round. This allows for the generation of prediction information of the self-vehicle by comprehensively considering more information about other vehicles besides the self-vehicle, which is more conducive to exploring prediction information of the self-vehicle with better safety and higher traffic efficiency.

[0146] In another scenario, the relationship between the first module and the second module is symmetrical. Optionally, step 303 may include: in the k-th round, the first vehicle obtains the prediction information of the second vehicle in the k-th round through the second module based on the second feature of the (k-1)-th round and the prediction information of the first vehicle in the (k-1)-th round; optionally, the second feature of the k-th round is also obtained through the second module. The method further includes: in the k-th round, the first vehicle obtains the first feature of the k-th round through the first module based on the first feature of the (k-1)-th round and the prediction information of the second vehicle in the (k-1)-th round (optionally, navigation information and reference prediction information of the second vehicle are also included); optionally, the prediction information of the first vehicle in the k-th round is also obtained through the first module.

[0147] Step 304 may include: in the (k+1)th round, the first vehicle, based on the first feature of the kth round and the prediction information of the second vehicle in the kth round (optionally, also including navigation information and reference prediction information of the second vehicle), obtains the prediction information of the first vehicle in the (k+1)th round through the first module; optionally, the first feature of the (k+1)th round is also obtained through the first module. The method may further include: in the (k+1)th round, the first vehicle, based on the second feature of the kth round and the prediction information of the first vehicle in the kth round, obtains the prediction information of the second vehicle in the (k+1)th round and the second feature of the (k+1)th round through the second module.

[0148] To understand this solution more intuitively, please refer to Figures 9 and 10. First, refer to Figure 9, which is another schematic diagram of the first model provided in the embodiment of this application. In Figure 9, the relationship between the first module and the second module is arranged in parallel. As shown in Figure 9, after the first vehicle inputs the first information into the first model, the first feature and the second feature are obtained by feature extraction of the first information through the main module in the first model. Based on the first feature, the planning trajectory of the first vehicle in the 0th round and the first feature of the 0th round are generated by the first module. Based on the second feature, the planning trajectory of the second vehicle in the 0th round and the second feature of the 0th round are generated by the second module.

[0149] In the k-th round, the first vehicle, based on its planned trajectory and candidate trajectory from the (k-1)-th round and the second feature from the (k-1)-th round, generates the predicted trajectory and second feature of the second vehicle in the k-th round through the second module. The first vehicle, based on the predicted trajectory and first feature of the second vehicle in the (k-1)-th round, generates its planned trajectory and candidate trajectory in the k-th round and the first feature from the k-th round through the first module.

[0150] The first vehicle executes K rounds through the first module and the second module to obtain the planned trajectory and candidate trajectory of the first vehicle in the Kth round, output by the first model. It should be understood that the example in Figure 9 is only for the convenience of understanding this scheme and is not intended to limit this scheme.

[0151] Please refer to Figure 10, which is another schematic diagram of the prediction information acquisition method provided in the embodiments of this application. Figure 10 uses a symmetrical relationship between the first module and the second module as an example. As shown in Figure 10, the first vehicle (i.e., the autonomous vehicle) can input the first information from the perception module into the first model. After the first vehicle extracts features from the first information through the main module of the first model, it can obtain initial first features and initial second features. Based on the first information, the first vehicle determines the value of K through the interaction iteration number determination module of the first model. The K iterations include the first to the Kth iterations.

[0152] In the initial stage (which can also be understood as round 0), the first vehicle, based on the initial first feature, generates prediction information and first features for round 0 through the first module of the first model. The prediction information for round 0 includes the planned trajectory, the score of the planned trajectory, at least one candidate trajectory, and the score of each candidate trajectory. Based on the initial second feature, the first vehicle generates prediction information and second features for each of at least one second vehicle (i.e., other vehicles) in round 0 through the second module of the second model. The prediction information for each second vehicle in round 0 includes its predicted trajectory.

[0153] In the first round, the first vehicle, based on the second feature from the first round and the prediction information of the first vehicle in the first round, obtains the prediction information of the second vehicle in the first round and the second feature from the first round through the second module. The first vehicle, based on the first feature from the first round and the prediction information of the second vehicle in the first round, obtains the prediction information of the first vehicle in the first round and the first feature from the first round through the first module.

[0154] In the k-th round, the first vehicle, based on the second feature from the (k-1)-th round and the prediction information of the first vehicle from the (k-1)-th round, obtains the prediction information of the second vehicle from the k-th round and the second feature from the k-th round through the second module. The first vehicle, based on the first feature from the (k-1)-th round and the prediction information of the second vehicle from the (k-1)-th round, obtains the prediction information of the first vehicle from the k-th round and the first feature from the k-th round through the first module.

[0155] When the k-th round is the K-th round, the first vehicle can obtain its planned trajectory for the K-th round based on the predicted information generated in the K-th round. This planned trajectory can be output as the first model and sent to the downstream control module. The control module in the first vehicle can control the first vehicle based on its planned trajectory in the K-th round. For example, the first vehicle can generate steering commands, braking commands, and / or drive commands to its chassis and electric drive system based on the planned trajectory in the K-th round. It should be understood that the example in Figure 10 is for illustrative purposes only and is not intended to limit the scope of this solution.

[0156] In this embodiment, since the vehicle and the surrounding vehicle also take actions synchronously during actual driving, in the kth round, the first module and the second module generate their respective prediction information symmetrically. This reasoning method is closer to the interaction mode in the actual driving scenario, which is conducive to making the interaction game between the first vehicle and the second vehicle closer to the real interaction situation. In turn, it is conducive to exploring better prediction information of the first vehicle based on a more accurate interaction game situation.

[0157] II. Training Phase

[0158] Please refer to Figure 11, which is a flowchart illustrating a model training method provided in an embodiment of this application. The model training method provided in this embodiment may include steps 1101 to 1104, wherein...

[0159] 1101. Obtain first information, which includes information about objects in the traffic environment.

[0160] For example, the training device may deploy a training dataset, which includes multiple training data. Each training data may include first information and the true value of the first vehicle and the true value of the second vehicle corresponding to the first information. The training device can then obtain the first information from the training data. The meaning of the first information can be referred to the description in the above embodiments, and will not be repeated here.

[0161] 1102. Based on the first information, the prediction information of the first vehicle is obtained through the first model, wherein the first vehicle is a self-driving vehicle, and the prediction information of the vehicle indicates the trajectory of the vehicle and / or the driving strategy of the vehicle.

[0162] 1103. Based on the prediction information of the first vehicle, the prediction information of the second vehicle is obtained through the first model. The second vehicle is different from the first vehicle.

[0163] 1104. Based on the prediction information of the second vehicle, the prediction information of the first vehicle is obtained through the first model.

[0164] For example, the specific implementation of steps 1102 to 1104 by the training device, and the specific meanings of the terms in steps 1102 to 1104 can be referred to the description in the embodiment corresponding to Figure 3 above, and will not be repeated here.

[0165] 1105. The first model is trained using a loss function, wherein the loss function includes a first loss term and a second loss term. The first loss term indicates the similarity between the predicted information of the first vehicle and the ground truth of the first vehicle, and the second loss term indicates the similarity between the predicted information of the second vehicle and the ground truth of the second vehicle. The ground truth of the vehicle indicates the correct trajectory of the vehicle and / or the correct driving strategy of the vehicle.

[0166] For example, in this application, the correct trajectory can also be referred to as the expected trajectory, and the correct driving strategy can also be referred to as the expected driving strategy. If the predicted information of the first vehicle indicates the planned trajectory of the first vehicle, then the ground truth value of the first vehicle indicates the correct trajectory of the first vehicle; if the predicted information of the first vehicle indicates the planned driving strategy of the first vehicle, then the ground truth value of the first vehicle indicates the correct driving strategy of the first vehicle. For example, the goal of training using the first loss term includes improving the similarity between the predicted information of the first vehicle and the ground truth value of the first vehicle. In one case, the first loss term indicates the similarity between the predicted information of each of the multiple predicted information of the first vehicle generated in steps 1102 to 1104 and the ground truth value of the first vehicle. In another case, the first loss term indicates the similarity between the final predicted information of the first vehicle in steps 1102 to 1104 and the ground truth value of the first vehicle.

[0167] If the predicted information of the second vehicle indicates the predicted trajectory of the second vehicle, then the ground truth value of the second vehicle indicates the correct trajectory of the second vehicle; if the predicted information of the second vehicle indicates the predicted driving strategy of the second vehicle, then the ground truth value of the second vehicle indicates the correct driving strategy of the second vehicle. For example, the objective of training using the second loss term includes improving the similarity between the predicted information of the second vehicle and the ground truth value of the second vehicle. For example, the second loss term indicates the similarity between the predicted information of each second vehicle generated in at least one of the predicted information of the second vehicle in steps 1102 to 1104 above and the ground truth value of the second vehicle.

[0168] For example, the training device can obtain the value of the first loss term based on the prediction information of the first vehicle and the true value of the first vehicle obtained in steps 1102 to 1104, and obtain the value of the second loss term based on the prediction information of the second vehicle and the true value of the second vehicle.

[0169] Optionally, the loss function further includes a third loss term, which indicates at least one of the following obtained based on the prediction information of the first vehicle and / or the prediction information of the second vehicle: the safety of the first vehicle, the traffic efficiency of the first vehicle, the safety of the second vehicle, or the traffic efficiency of the second vehicle. The purpose of training using the third loss term includes improving at least one of the following: the safety of the first vehicle, the traffic efficiency of the first vehicle, the safety of the second vehicle, or the traffic efficiency of the second vehicle.

[0170] For example, the training device can evaluate the safety and / or traffic efficiency of the second vehicle under each prediction information of the at least one second vehicle generated in steps 1102 to 1104 above, and obtain a first score corresponding to the prediction information of each second vehicle.

[0171] For example, in one case, the training device can evaluate the safety and / or traffic efficiency of the first vehicle under each prediction information of the multiple first vehicles generated in steps 1102 to 1104 above, and obtain a second score corresponding to the prediction information of each first vehicle.

[0172] The training device can obtain the value of the third loss term based on the first score corresponding to the prediction information of each second vehicle, and / or the second score corresponding to the prediction information of each first vehicle.

[0173] In another scenario, the training device can evaluate the safety and / or traffic efficiency of the first vehicle based on the final prediction information of the first vehicle generated in steps 1102 to 1104 above, and obtain a third score corresponding to the final prediction information of the first vehicle.

[0174] The training device can obtain the value of the third loss term based on the first score corresponding to the prediction information of each second vehicle, and / or the third score corresponding to the final prediction information of the first vehicle.

[0175] For example, the training device can obtain the value of the loss function based on the values ​​of the first loss term and the second loss term (optionally, also including the value of the third loss term). The training device performs gradient differentiation on the loss function and uses the backpropagation algorithm to update the parameters in the first model; optionally, the values ​​in the second information are also learnable parameters to complete one training of the first model. The training device can repeat steps 1101 to 1105 to iteratively train the first model until the convergence condition is met, obtaining the trained first model.

[0176] For example, the convergence condition may include: satisfying the convergence condition of the loss function, and / or, the number of iterations of training the first model reaches a preset threshold.

[0177] To more intuitively understand this solution, please refer to Figure 12. Figure 12 is another schematic diagram of the training method of the model provided in the embodiment of this application. In Figure 12, the first module and the second module are symmetrical, and the vehicle prediction information includes the vehicle trajectory as an example. The structure of the first model, the meaning of K, and the specific situations of the 0th round and the Kth round can be understood in conjunction with the description in Figure 3 above, and will not be described in detail here. As shown in Figure 12, the first loss term indicates the similarity between the planned trajectory of the first vehicle generated in each of the 0th round and the Kth round and the correct trajectory of the first vehicle. The second loss term indicates the similarity between the planned trajectory of the second vehicle generated in each of the 0th round and the Kth round and the correct trajectory of the second vehicle. The third loss term indicates at least one of the following obtained based on the prediction information of the first vehicle and the prediction information of the second vehicle generated in each of the 0th round and the Kth round: the safety of the first vehicle, the traffic efficiency of the first vehicle, the safety of the second vehicle, or the traffic efficiency of the second vehicle. It should be understood that the example in Figure 12 is only for the convenience of understanding this solution and is not intended to limit this solution.

[0178] In this embodiment of the application, the loss function further includes a third loss term. The purpose of training the first model using the third loss term is to enable the first model to learn the ability to generate prediction information that is safer and more efficient in traffic. Then, the prediction information of the first vehicle obtained by the trained model can control the first vehicle to drive in a safer and more efficient manner.

[0179] Based on the embodiments corresponding to Figures 1 to 12, in order to better implement the above-described solutions of the embodiments of this application, related equipment for implementing the above-described solutions is also provided below. Specifically, referring to Figure 13, Figure 13 is a structural schematic diagram of a prediction information acquisition device provided in an embodiment of this application. The prediction information acquisition device includes: an acquisition unit 1301, used to acquire first information, the first information including information about objects in a traffic environment; a processing unit 1302, used to obtain prediction information of a first vehicle through a model based on the first information, wherein the first vehicle is a self-driving vehicle, and the vehicle's prediction information indicates the vehicle's trajectory and / or driving strategy; the processing unit 1302 is also used to obtain prediction information of a second vehicle through a model based on the prediction information of the first vehicle, the second vehicle being different from the first vehicle; the processing unit 1302 is also used to obtain updated prediction information of the first vehicle through a model based on the prediction information of the second vehicle.

[0180] Optionally, the apparatus includes performing K rounds through a model, where K is an integer greater than or equal to 1, and each of the K rounds includes: obtaining prediction information for a second vehicle based on prediction information for a first vehicle, and obtaining prediction information for a first vehicle based on prediction information for the second vehicle.

[0181] Optionally, the model includes a first module and a second module. The processing unit 1302 is specifically used to extract features from the first information through the model to obtain a first feature, and based on the first feature, to obtain the prediction information of the first vehicle and the updated first feature through the first module. The processing unit 1302 is specifically used to obtain the updated prediction information of the first vehicle through the first module based on the updated first feature and the prediction information of the second vehicle.

[0182] Optionally, the processing unit 1302 is specifically used to extract features from the first information through a model to obtain a first feature and a second feature; the processing unit 1302 is specifically used to obtain the prediction information of the second vehicle through the second module based on the second feature and the prediction information of the first vehicle.

[0183] Optionally, the processing unit 1302 is specifically used to obtain the prediction information of the second vehicle in the k-th round through the second module based on the second feature of the (k-1)-th round and the prediction information of the first vehicle in the (k-1)-th round, where k is any value from 1 to K; the processing unit 1302 is specifically used to obtain the prediction information of the first vehicle in the k-th round through the first module based on the first feature of the (k-1)-th round and the prediction information of the second vehicle in the k-th round.

[0184] Optionally, processing unit 1302 is specifically used to obtain the prediction information of the second vehicle in the k-th round through a second module based on the second feature of the (k-1)-th round and the prediction information of the first vehicle in the (k-1)-th round, where k is any value from 1 to K; processing unit 1302 is also used to obtain the first feature of the k-th round through a second module based on the first feature of the (k-1)-th round and the prediction information of the second vehicle in the (k-1)-th round; processing unit 1302 is specifically used to obtain the prediction information of the first vehicle in the (k+1)-th round through a first module based on the first feature of the k-th round and the prediction information of the second vehicle in the k-th round.

[0185] Optionally, the value of K is obtained based on the first information.

[0186] Optionally, the information about objects in the traffic environment includes the location information of dynamic objects in the traffic environment and the topological information of roads in the traffic environment.

[0187] It should be noted that the information interaction and execution process between the modules / units in the prediction information acquisition device 1300 are based on the same concept as the various method embodiments corresponding to Figures 1 to 12 in this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0188] Please refer to Figure 14, which is a schematic diagram of a model training device provided in an embodiment of this application. The model training device 1400 includes: an acquisition unit 1401, used to acquire first information, the first information including information about objects in a traffic environment; a processing unit 1402, used to obtain prediction information of a first vehicle through a model based on the first information, wherein the first vehicle is a vehicle, and the vehicle prediction information indicates the vehicle's trajectory and / or the vehicle's driving strategy; the processing unit 1402 is also used to obtain prediction information of a second vehicle through a model based on the prediction information of the first vehicle, the second vehicle being different from the first vehicle; the processing unit 1402 is also used to obtain prediction information of the first vehicle through a model based on the prediction information of the second vehicle; and a training unit 1403, used to train the model using a loss function, wherein the loss function includes a first loss term and a second loss term, the first loss term indicating the similarity between the prediction information of the first vehicle and the ground truth value of the first vehicle, the second loss term indicating the similarity between the prediction information of the second vehicle and the ground truth value of the second vehicle, and the ground truth value of the vehicle indicating the correct trajectory and / or the correct driving strategy of the vehicle.

[0189] Optionally, the loss function may further include a third loss term, which indicates at least one of the following obtained based on the prediction information of the first vehicle and / or the prediction information of the second vehicle: the safety of the first vehicle, the traffic efficiency of the first vehicle, the safety of the second vehicle, or the traffic efficiency of the second vehicle.

[0190] It should be noted that the information interaction and execution process between the modules / units in the training device 1400 of the model are based on the same concept as the various method embodiments corresponding to Figures 1 to 12 in this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0191] The following describes a device provided in an embodiment of this application. Please refer to Figure 15, which is a schematic diagram of the structure of a device provided in an embodiment of this application. Optionally, the device 1500 performs the functions of the first vehicle or training device in the various method embodiments corresponding to Figures 1 to 12.

[0192] Device 1500 includes a memory 1502 and at least one processor 1501. Optionally, device 1500 further includes at least one accelerator 1503. Optionally, processor 1501 implements the method in the above embodiments by reading program instructions stored in memory 1502; or, processor 1501 reads program instructions stored in memory 1502 and implements the steps executed by the machine learning model in the method in the above embodiments through accelerator 1503; or, processor 1501 may also implement the method in the above embodiments by reading program instructions stored internally; or, processor 1501 may also read program instructions stored internally and implement the steps executed by the machine learning model in the method in the above embodiments through accelerator 1503.

[0193] When the processor 1501 reads the program instructions stored in the memory 1502 to implement the method in the above embodiments, the memory 1502 stores the program instructions that implement the method provided in the above embodiments of this application.

[0194] Optionally, at least one processor 1501 is one or more CPUs, either a single-core CPU or a multi-core CPU. For example, memory 1502 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, or optical memory. Memory 1502 stores program instructions for the operating system. For example, at least one accelerator 1503 may include at least one of the following: GPU, NPU, TPU, ASIC, FPGA, or other types of accelerators. After the program instructions stored in memory 1502 are read by the at least one processor 1501, device 1500 executes the corresponding operations in the foregoing embodiments.

[0195] Optionally, the device 1500 also includes a network interface 1504, which can be a wired interface or a wireless interface. The network interface 1504 is used to send and receive data in the various method embodiments corresponding to Figures 1 to 12.

[0196] It should be understood that network interface 1504 has the functions of receiving and sending data. The functions of "receiving data" and "sending data" can be integrated into the same transceiver interface, or the functions of "receiving data" and "sending data" can be implemented in different interfaces, which is not limited here. In other words, network interface 1504 may include one or more interfaces for implementing the functions of "receiving data" and "sending data".

[0197] After the processor 1501 reads the program instructions from the memory 1502, other functions that the device 1500 can perform are described in the preceding method embodiments.

[0198] Optionally, the device 1500 also includes a bus 1505, through which the processor 1501 and memory 1502 are typically interconnected, or in other ways.

[0199] The device 1500 provided in this application embodiment is used to execute the methods executed by the first vehicle or training device in the above-described method embodiments, and to achieve the corresponding beneficial effects. The specific implementation of the device 1500 shown in Figure 15 can be referred to the descriptions in the foregoing method embodiments, and will not be repeated here.

[0200] This application also provides a vehicle, as shown in Figure 16. Figure 16 is a structural schematic diagram of a vehicle provided in this application embodiment. The vehicle 100 is configured for fully or partially automated driving mode. For example, the vehicle 100 can control itself while in automated driving mode, and can determine the current state of the vehicle and its surrounding environment through human operation, determine the possible behavior of at least one other vehicle in the surrounding environment, and determine the confidence level corresponding to the probability of other vehicles performing possible behaviors, and control the vehicle 100 based on the determined information. When the vehicle 100 is in automated driving mode, the vehicle 100 can also be set to operate without human interaction.

[0201] Vehicle 100 may include various subsystems, such as a mobility system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 112, and a user interface 116. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0202] The mobility system 102 may include components that provide powered motion to the vehicle 100. In one embodiment, the mobility system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels / tires 121.

[0203] Engine 118 can be an internal combustion engine, an electric motor, an air-compressed engine, or other combinations of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. Engine 118 converts energy source 119 into mechanical energy. Examples of energy source 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 119 can also provide energy to other systems of vehicle 100. Transmission 120 transmits mechanical power from engine 118 to wheels 121. Transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, transmission 120 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 121.

[0204] Sensor system 104 may include several sensors for sensing information about the environment surrounding vehicle 100. For example, sensor system 104 may include a positioning system 122 (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU) 124, a radar 126, a laser rangefinder 128, and a camera 130. Sensor system 104 may also include sensors for the internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensing data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of the autonomous vehicle 100.

[0205] The positioning system 122 can be used to estimate the geographical location of the vehicle 100. An IMU 124 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In one embodiment, the IMU 124 can be a combination of an accelerometer and a gyroscope. A radar 126 can use radio signals to sense objects in the surrounding environment of the vehicle 100, specifically millimeter-wave radar or lidar. In some embodiments, in addition to sensing objects, the radar 126 can also be used to sense the speed and / or direction of travel of objects. A laser rangefinder 128 can use lasers to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. A camera 130 can be used to capture multiple images of the surrounding environment of the vehicle 100. The camera 130 can be a still camera or a video camera.

[0206] The control system 106 controls the operation of the vehicle 100 and its components. The control system 106 may include various components, including a steering system 132, a throttle 134, a braking unit 136, a computer vision system 140, a trajectory control system 142, and an obstacle avoidance system 144.

[0207] The steering system 132 is operable to adjust the forward direction of the vehicle 100. For example, in one embodiment, it may be a steering wheel system. The throttle 134 controls the operating speed of the engine 118 and thus the speed of the vehicle 100. The braking unit 136 controls the deceleration of the vehicle 100. The braking unit 136 may use friction to slow down the wheels 121. In other embodiments, the braking unit 136 may convert the kinetic energy of the wheels 121 into electrical current. The braking unit 136 may also take other forms to slow down the rotational speed of the wheels 121 to control the speed of the vehicle 100. The computer vision system 140 is operable to process and analyze images captured by the camera 130 to identify objects and / or features in the environment surrounding the vehicle 100. The objects and / or features may include traffic signals, road boundaries, and obstacles. The computer vision system 140 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 140 may be used to map the environment, track objects, estimate the speed of objects, etc. The route control system 142 is used to determine the driving route and speed of the vehicle 100. In some embodiments, the route control system 142 may include a lateral planning module 1421 and a longitudinal planning module 1422, which are respectively used to combine data from the obstacle avoidance system 144, GPS 122, and one or more predetermined maps to determine the driving route and speed for the vehicle 100. The obstacle avoidance system 144 is used to identify, evaluate, and avoid or otherwise traverse obstacles in the environment of the vehicle 100, which may specifically be physical obstacles and virtual moving bodies that may collide with the vehicle 100. In one example, the control system 106 may add or alternatively include components other than those shown and described. Alternatively, some of the components shown above may be reduced.

[0208] Vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users via peripheral device 108. Peripheral device 108 may include wireless communication system 146, on-board computer 148, microphone 150, and / or speaker 152. In some embodiments, peripheral device 108 provides a means for a user of vehicle 100 to interact with user interface 116. For example, on-board computer 148 may provide information to a user of vehicle 100. User interface 116 may also operate on-board computer 148 to receive user input. On-board computer 148 may be operated via a touchscreen. In other cases, peripheral device 108 may provide a means for vehicle 100 to communicate with other devices located within the vehicle. For example, microphone 150 may receive audio (e.g., voice commands or other audio input) from a user of vehicle 100. Similarly, speaker 152 may output audio to a user of vehicle 100. Wireless communication system 146 may communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. The wireless communication system 146 may utilize a wireless local area network (WLAN) for communication. In some embodiments, the wireless communication system 146 may utilize an infrared link, Bluetooth, or ZigBee to communicate directly with the device. Other wireless protocols, such as various vehicle communication systems, may also be used. For example, the wireless communication system 146 may include one or more dedicated short-range communications (DSRC) devices that can enable public and / or private data communication between the vehicle and / or a roadside station.

[0209] Power source 110 can provide power to various components of vehicle 100. In one embodiment, power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more such battery packs can be configured to provide power to various components of vehicle 100. In some embodiments, power source 110 and energy source 119 can be implemented together, as is the case in some fully electric vehicles.

[0210] Some or all of the functions of vehicle 100 are controlled by computer system 112. Computer system 112 may include at least one processor 113, which executes program instructions 115 stored in a non-transitory computer-readable medium such as memory 114. Computer system 112 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner. Processor 113 may include any conventional processor, such as a commercially available central processing unit (CPU). Optionally, processor 113 may also include a dedicated device such as a GPU, NPU, TPU, ASIC, FPGA, or other hardware-based processor. Although FIG16 functionally illustrates the processor, memory, and other components of computer system 112 in the same block, those skilled in the art will understand that the processor or memory may actually include multiple processors or memories not stored in the same physical housing. For example, memory 114 may be a hard disk drive or other storage medium located in a housing different from that of computer system 112. Therefore, references to processor 113 or memory 114 will be understood to include references to a collection of processors or memories that may or may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as steering and deceleration components, can each have their own processor that performs only calculations related to the component's specific function.

[0211] In all the aspects described herein, processor 113 may be located remotely from vehicle 100 and may communicate wirelessly with vehicle 100. In other aspects, some of the processes described herein are executed on processor 113 located within vehicle 100, while others are executed by remote processor 113, including taking the necessary steps to perform a single operation.

[0212] In some embodiments, memory 114 may contain instructions 115 (e.g., program logic) that can be executed by processor 113 to perform various functions of vehicle 100, including those described above. Memory 114 may also contain additional program instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the mobility system 102, sensor system 104, control system 106, and peripheral devices 108. In addition to instructions 115, memory 114 may also store data such as road maps, route information, vehicle position, direction, speed, and other such vehicle data, as well as other information. This information may be used by vehicle 100 and computer system 112 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes. A user interface 116 is provided to or receives information from a user of vehicle 100. Optionally, user interface 116 may include one or more input / output devices within the set of peripheral devices 108, such as wireless communication system 146, on-board computer 148, microphone 150, and speaker 152.

[0213] Computer system 112 can control the functions of vehicle 100 based on input received from various subsystems (e.g., driving system 102, sensor system 104, and control system 106) and from user interface 116. For example, computer system 112 can utilize input from control system 106 to control steering system 132 to avoid obstacles detected by sensor system 104 and obstacle avoidance system 144. In some embodiments, computer system 112 is operable to provide control over many aspects of vehicle 100 and its subsystems.

[0214] Alternatively, one or more of these components may be installed separately from or associated with vehicle 100. For example, memory 114 may exist partially or completely separately from vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0215] Optionally, the above components are merely examples. In practical applications, components in each of the above modules may be added or removed according to actual needs. Figure 16 should not be construed as a limitation on the embodiments of this application. A vehicle traveling on a road, such as vehicle 100 above, can identify objects in its surrounding environment to determine an adjustment to its current speed. These objects can be other vehicles, traffic control equipment, or other types of objects. In some examples, each identified object can be considered independently, and based on the object's individual characteristics, such as its current speed, acceleration, and distance from the vehicle, the speed adjustment to be made by the vehicle can be determined.

[0216] Optionally, vehicle 100 or computing devices associated with vehicle 100, such as computer system 112, computer vision system 140, and memory 114 as shown in Figure 16, can predict the behavior of the identified objects based on the characteristics of the identified objects and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of each other, so all identified objects can also be considered together to predict the behavior of a single identified object. Vehicle 100 can adjust its speed based on the predicted behavior of the identified objects. In other words, vehicle 100 can determine what steady state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the objects. In this process, other factors can also be considered in determining the speed of vehicle 100, such as the lateral position of vehicle 100 in the road, the curvature of the road, the proximity of static and dynamic objects, etc. In addition to providing program instructions to adjust the speed of the vehicle, the computing device may also provide program instructions to modify the steering angle of the vehicle 100 so that the vehicle 100 follows a given trajectory and / or maintains a safe lateral and longitudinal distance from objects near the vehicle 100 (e.g., cars in adjacent lanes on the road).

[0217] In this embodiment, the processor 113 in the vehicle 100 is used to execute the method executed by the first vehicle in the embodiments corresponding to Figures 1 to 12. It should be noted that the specific manner in which the processor 113 executes the aforementioned steps is based on the same concept as the method embodiments corresponding to Figures 1 to 12 in this application, and the resulting technical effects are the same as those in the method embodiments corresponding to Figures 1 to 12 in this application. For details, please refer to the descriptions in the aforementioned method embodiments of this application, which will not be repeated here.

[0218] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the steps performed by the first vehicle or training device in the methods described in the embodiments shown in Figures 1 to 12 above.

[0219] This application also provides a computer program product, which includes a program that, when run on a computer, causes the computer to perform the steps performed by the first vehicle or training device in the methods described in the embodiments shown in Figures 1 to 12 above.

[0220] This application also provides a circuit system including a processing circuit configured to perform the steps executed by the first vehicle or training device in the method described in the embodiments shown in Figures 1 to 12 above.

[0221] The training device, model training apparatus, or prediction information acquisition apparatus provided in this application embodiment can specifically be a chip. The chip includes a processing unit, such as a processor. Optionally, the chip also includes a communication unit, such as an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip to execute the methods described in the embodiments shown in Figures 1 to 12. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0222] The processor mentioned above can be a general-purpose central processing unit, microprocessor, GPU, NPU, TPU, ASIC, FPGA, or one or more integrated circuits used to control the execution of the program in the first aspect of the above method.

[0223] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0224] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CLUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0225] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0226] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A method for obtaining predictive information, characterized in that, The method includes: Obtain first information, which includes information about objects in the traffic environment; Based on the first information, the prediction information of the first vehicle is obtained through the model, wherein the first vehicle is a self-driving vehicle, and the prediction information of the vehicle indicates the trajectory of the vehicle and / or the driving strategy of the vehicle. Based on the prediction information of the first vehicle, the prediction information of the second vehicle is obtained through the model, and the second vehicle is different from the first vehicle; Based on the prediction information of the second vehicle, the updated prediction information of the first vehicle is obtained through the model.

2. The method according to claim 1, characterized in that, The method includes executing K rounds through the model, where K is an integer greater than or equal to 1, and each of the K rounds includes: obtaining prediction information for the second vehicle based on the prediction information of the first vehicle, and obtaining prediction information for the first vehicle based on the prediction information of the second vehicle.

3. The method according to claim 2, characterized in that, The model includes a first module and a second module. The step of obtaining prediction information for the first vehicle based on the first information through the model includes: The first feature is obtained by extracting features from the first information using the model. Based on the first feature, the predicted information of the first vehicle and the updated first feature are obtained through the first module; The step of obtaining updated prediction information for the first vehicle based on the prediction information of the second vehicle through the model includes: obtaining updated prediction information for the first vehicle through the first module based on the updated first feature and the prediction information of the second vehicle.

4. The method according to claim 3, characterized in that, The step of extracting features from the first information using the model to obtain the first feature includes: extracting features from the first information using the model to obtain the first feature and the second feature; The step of obtaining the prediction information of the second vehicle based on the prediction information of the first vehicle through the model includes: obtaining the prediction information of the second vehicle through the second module based on the second feature and the prediction information of the first vehicle.

5. The method according to any one of claims 2-4, characterized in that, The step of obtaining the prediction information of the second vehicle through the second module based on the updated second feature and the prediction information of the first vehicle includes: in the k-th round, based on the second feature of the (k-1)-th round and the prediction information of the first vehicle of the (k-1)-th round, obtaining the prediction information of the second vehicle of the k-th round through the second module, where k is any value from 1 to K; The step of obtaining updated prediction information of the first vehicle based on the prediction information of the second vehicle through the model includes: in the kth round, based on the first feature of the (k-1)th round and the prediction information of the second vehicle in the kth round, obtaining the prediction information of the first vehicle in the kth round through the first module.

6. The method according to any one of claims 2-4, characterized in that, The step of obtaining the prediction information of the second vehicle through the second module based on the updated second feature and the prediction information of the first vehicle includes: in the k-th round, based on the second feature of the (k-1)-th round and the prediction information of the first vehicle of the (k-1)-th round, obtaining the prediction information of the second vehicle of the k-th round through the second module, where k is any value from 1 to K; The method further includes: in the k-th round, based on the first feature of the (k-1)-th round and the prediction information of the second vehicle in the (k-1)-th round, obtaining the first feature of the k-th round through the second module; The step of obtaining updated prediction information of the first vehicle based on the prediction information of the second vehicle through the model includes: in the (k+1)th round, based on the first feature of the kth round and the prediction information of the second vehicle in the kth round, obtaining the prediction information of the first vehicle in the (k+1)th round through the first module.

7. The method according to any one of claims 2-4, characterized in that, The value of K is obtained based on the first information.

8. The method according to any one of claims 2-4, characterized in that, The information about objects in the traffic environment includes the location information of dynamic objects in the traffic environment and the topological information of roads in the traffic environment.

9. A method for training a model, characterized in that, The method includes: Obtain first information, which includes information about objects in the traffic environment; Based on the first information, the prediction information of the first vehicle is obtained through the model, wherein the first vehicle is a self-driving vehicle, and the prediction information of the vehicle indicates the trajectory of the vehicle and / or the driving strategy of the vehicle. Based on the prediction information of the first vehicle, the prediction information of the second vehicle is obtained through the model, and the second vehicle is different from the first vehicle; Based on the prediction information of the second vehicle, the prediction information of the first vehicle is obtained through the model; The model is trained using a loss function, which includes a first loss term and a second loss term. The first loss term indicates the similarity between the predicted information of the first vehicle and the ground truth of the first vehicle, and the second loss term indicates the similarity between the predicted information of the second vehicle and the ground truth of the second vehicle. The ground truth of the vehicle indicates the correct trajectory of the vehicle and / or the correct driving strategy of the vehicle.

10. The method according to claim 9, characterized in that, The loss function further includes a third loss term, which indicates at least one of the following obtained based on the prediction information of the first vehicle and / or the prediction information of the second vehicle: the safety of the first vehicle, the traffic efficiency of the first vehicle, the safety of the second vehicle, or the traffic efficiency of the second vehicle.

11. A device for acquiring predictive information, characterized in that, The device includes: An acquisition unit is configured to acquire first information, the first information including information about objects in the traffic environment; The processing unit is configured to obtain prediction information of the first vehicle through a model based on the first information, wherein the first vehicle is a vehicle, and the prediction information of the vehicle indicates the trajectory of the vehicle and / or the driving strategy of the vehicle. The processing unit is further configured to obtain prediction information for a second vehicle based on the prediction information of the first vehicle through the model, wherein the second vehicle is different from the first vehicle; The processing unit is further configured to obtain updated prediction information of the first vehicle based on the prediction information of the second vehicle through the model.

12. The apparatus according to claim 11, characterized in that, The device includes executing K rounds through the model, where K is an integer greater than or equal to 1, and each of the K rounds includes: obtaining prediction information for the second vehicle based on prediction information for the first vehicle, and obtaining prediction information for the first vehicle based on prediction information for the second vehicle.

13. The apparatus according to claim 12, characterized in that, The model includes a first module and a second module. The processing unit is specifically used to extract features from the first information through the model to obtain a first feature, and based on the first feature, to obtain the prediction information of the first vehicle and the updated first feature through the first module. The processing unit is specifically used to obtain the updated prediction information of the first vehicle through the first module based on the updated first feature and the prediction information of the second vehicle.

14. The apparatus according to claim 13, characterized in that, The processing unit is specifically used to extract features from the first information using the model to obtain the first feature and the second feature. The processing unit is specifically used to obtain the prediction information of the second vehicle through the second module based on the second feature and the prediction information of the first vehicle.

15. The apparatus according to any one of claims 12-14, characterized in that, The processing unit is specifically used to obtain the prediction information of the second vehicle in the kth round through the second module based on the second feature in the (k-1)th round and the prediction information of the first vehicle in the (k-1)th round, where k is any value from 1 to K. The processing unit is specifically used to obtain the prediction information of the first vehicle in the kth round through the first module, based on the first feature in the (k-1)th round and the prediction information of the second vehicle in the kth round.

16. The apparatus according to any one of claims 12-14, characterized in that, The processing unit is specifically used to obtain the prediction information of the second vehicle in the kth round through the second module based on the second feature in the (k-1)th round and the prediction information of the first vehicle in the (k-1)th round, where k is any value from 1 to K. The processing unit is further configured to, in the k-th round, obtain the first feature of the k-th round through the second module based on the first feature of the (k-1)-th round and the prediction information of the second vehicle of the (k-1)-th round; The processing unit is specifically used to obtain the prediction information of the first vehicle in the (k+1)th round through the first module, based on the first feature of the kth round and the prediction information of the second vehicle in the kth round.

17. The apparatus according to any one of claims 12-14, characterized in that, The value of K is obtained based on the first information.

18. The apparatus according to any one of claims 12-14, characterized in that, The information about objects in the traffic environment includes the location information of dynamic objects in the traffic environment and the topological information of roads in the traffic environment.

19. A training device for a model, characterized in that, The device includes: An acquisition unit is configured to acquire first information, the first information including information about objects in the traffic environment; The processing unit is configured to obtain prediction information of the first vehicle through a model based on the first information, wherein the first vehicle is a vehicle, and the prediction information of the vehicle indicates the trajectory of the vehicle and / or the driving strategy of the vehicle. The processing unit is further configured to obtain prediction information for a second vehicle based on the prediction information of the first vehicle through the model, wherein the second vehicle is different from the first vehicle; The processing unit is further configured to obtain the prediction information of the first vehicle based on the prediction information of the second vehicle through the model; A training unit is used to train the model using a loss function, wherein the loss function includes a first loss term and a second loss term, the first loss term indicating the similarity between the predicted information of the first vehicle and the ground truth of the first vehicle, the second loss term indicating the similarity between the predicted information of the second vehicle and the ground truth of the second vehicle, and the ground truth of the vehicle indicating the correct trajectory of the vehicle and / or the correct driving strategy of the vehicle.

20. The apparatus according to claim 19, characterized in that, The loss function further includes a third loss term, which indicates at least one of the following obtained based on the prediction information of the first vehicle and / or the prediction information of the second vehicle: the safety of the first vehicle, the traffic efficiency of the first vehicle, the safety of the second vehicle, or the traffic efficiency of the second vehicle.

21. A device, characterized in that, It includes a processor and a memory, wherein the processor is coupled to the memory. The memory is used to store programs; The processor is configured to execute a program in the memory, causing the device to perform the method as described in any one of claims 1 to 10.

22. A vehicle, characterized in that, The method includes a processor coupled to a memory storing program instructions, which, when executed by the processor, implement the method of any one of claims 1 to 8.

23. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 10.

24. A computer program product, characterized in that, The computer program product includes a program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 10.

25. A chip system, characterized in that, The chip system includes a processor that performs the method as described in any one of claims 1 to 10.

Images