Data processing method and apparatus, and model training method and apparatus

Abstract

Provided in the present application are a data processing method and apparatus, and a model training method and apparatus, which are used for implementing the whole process of end-to-end perception, and planning and control path output by means of integrating data of multiple modalities, navigation map data, etc. The data processing method comprises: acquiring map data and sensor data, wherein the sensor data comprises data collected by a plurality of sensors; performing feature extraction on the map data and the sensor data, so as to obtain perception features; then, on the basis of the perception features, acquiring a perception result, wherein the perception result comprises information of at least one instance in an environment of a vehicle, and the at least one instance may comprise a moving object and a stationary object in the environment of the vehicle; and then on the basis of the perception result, determining a traveling path of the vehicle.

Description

A data processing method and apparatus, a model training method and apparatus Technical Field

[0001] This application relates to the field of intelligent driving, and in particular to a data processing method and apparatus, and a model training method and apparatus. Background Technology

[0002] With the development of intelligent driving functions in vehicles, the number of supported intelligent driving functions is also increasing. For example, intelligent driving functions in vehicles can be divided into perception, prediction, decision-making, planning, and control.

[0003] For example, an existing intelligent driving solution can perform multiple tasks, such as tracking, map generation, path prediction, and occupancy grid prediction and planning. By connecting the various modules of the architecture through a unified query vector, it fully models the complex interactions between the agent and the environment, continuously extracts the agent's motion information, and applies the results of upstream module tasks to driving planning, ultimately achieving safe planning of the intelligent driving path. However, due to the decoupling of different tasks, each module suffers from information loss relative to the final driving planning goal, and the optimization objectives are inconsistent among multiple modules, leading to error propagation between modules. This results in a relatively large overall functional error, and there is room for further improvement in output accuracy. Summary of the Invention

[0004] This application provides a data processing method and apparatus, and a model training method and apparatus, for realizing the entire process of end-to-end perception and control path output by integrating data from multiple modalities and navigation map data.

[0005] In view of this, in a first aspect, this application provides a data processing method applied to a vehicle, comprising: acquiring map data and sensor data, the sensor data including data collected by multiple sensors; extracting features from the map data and sensor data to obtain perception features; subsequently acquiring perception results based on the perception features, the perception results including information on at least one instance in the vehicle's environment, the at least one instance including moving targets and stationary targets in the vehicle's environment; and subsequently determining the vehicle's driving path based on the perception results.

[0006] In this embodiment, an overall end-to-end architecture is implemented, that is, all modules are integrated into one architecture, reducing the transmission error between modules, thereby making the overall architecture error of the vehicle smaller and realizing higher precision vehicle intelligent driving functions.

[0007] In one possible implementation, the aforementioned perception results include at least one of the following: road structure information in the vehicle's environment, moving target information, sparse features, or dense features. In this application's implementation, the desired feature types can be extracted from multiple dimensions to adapt to various tasks of environmental perception, thereby enhancing the generalizability of the solution.

[0008] In one possible implementation, the aforementioned acquisition of perception results based on perception features may include: performing multiple perception tasks on the perception features through multiple modules and outputting perception results. The multiple modules include a whitelist detection module, an obstacle detection module, a road information detection module, a forward perspective (PV) detection module, or a traffic light detection module. The whitelist detection module is used to detect information of instances of a preset category, the obstacle detection module is used to detect information of the drivable area of ​​the vehicle, the road information detection module is used to detect road information, the PV detection module is used to output information of instances from the PV perspective, and the traffic light detection information is used to output information of traffic lights.

[0009] In this embodiment, multiple sensing modules can be set up to achieve multiple sensing tasks, enabling full perception of the environment. This allows downstream tasks to output more accurate results based on more comprehensive environmental perception, thereby improving the accuracy of downstream tasks.

[0010] In one possible implementation, the aforementioned determination of the vehicle's driving path based on perception results includes: acquiring navigation field information, which includes recommended scores corresponding to the vehicle's driving in different lanes according to the navigation path; and determining the driving path based on the navigation field information and perception results. In this application's implementation, combining the various dimensions of perception results provided in the embodiments of this application during navigation can improve navigation accuracy.

[0011] In one possible implementation, the aforementioned method further includes: determining a vehicle control signal based on the driving path, the vehicle control signal being used to control the vehicle to travel according to the driving path. Therefore, in this embodiment, the vehicle can be controlled to travel according to a planned driving path, achieving more intelligent vehicle driving functions, enabling higher-precision intelligent driving, and improving the user experience.

[0012] In one possible implementation, the aforementioned determination of vehicle control signals based on the driving path may include: determining the vehicle's driving decision by solving a multi-objective optimization algorithm based on the driving path. The optimization objective of this algorithm can be set according to the actual application scenario, such as the distance between the vehicle and obstacles, the distance between the vehicle and lane lines, or the turning angle of the vehicle; subsequently, vehicle control signals are generated based on the driving decision. In this embodiment, a better driving decision can be obtained using a multi-objective optimization algorithm.

[0013] In one possible implementation, the aforementioned solution based on the current information of the vehicle's current travel path or the travel paths of other vehicles is obtained using a multi-objective optimization algorithm. This includes: determining at least one driving decision for the vehicle using a multi-modal tree based on the current travel path and the travel paths of other vehicles; and determining the vehicle's driving decision from the at least one driving decision using the multi-objective optimization algorithm. In this embodiment, a multi-modal tree, combined with multi-modal travel paths, can be used to make a more suitable driving decision for the vehicle, thereby improving vehicle driving safety.

[0014] Secondly, this application provides a model training method, including:

[0015] First, a training set is obtained, which includes samples of various types. Then, the training set is input into the end-to-end model, and the end-to-end model is iteratively updated based on the output of the end-to-end model to obtain the updated end-to-end model. The end-to-end model is used to perform multiple tasks. The end-to-end model includes a feature extraction network, a perception network, and a regulation network. The feature extraction network is used to extract features from the input samples of various types. The perception network is used to output perception results based on the features output by the feature extraction network. The regulation network is used to output vehicle driving decisions based on the perception results output by the perception network. For example, the end-to-end model can be used to perform the steps of the first aspect or any optional implementation of the first aspect.

[0016] In this embodiment, the deployed end-to-end model is trained using a training set to achieve an overall end-to-end architecture, that is, to integrate various modules into one architecture, reduce the transmission error between modules, thereby making the overall architecture error of the vehicle smaller and achieving higher precision vehicle intelligent driving functions.

[0017] In one possible implementation, before iteratively updating the end-to-end model based on the output of the end-to-end model, the method further includes: configuring a corresponding training batch size for each of the multiple tasks, wherein the training batch size for each task is related to the duration of the step in each task; correspondingly, the aforementioned inputting the training set into the end-to-end model and iteratively updating the end-to-end model based on the output of the end-to-end model to obtain the updated end-to-end model includes: inputting the training set into the end-to-end model according to the training batch size, and iteratively updating the end-to-end model based on the output of the end-to-end model to obtain the updated end-to-end model.

[0018] In the embodiments of this application, during the training process of the end-to-end model, the training batch size can be set to keep the duration of a step as close as possible between different tasks, so as to synchronize the parameters corresponding to different tasks and improve training efficiency.

[0019] In one possible implementation, before iteratively updating the end-to-end model based on its output, the method further includes: determining a grouping strategy and a synchronization strategy. The grouping strategy is used to divide the parameters of the end-to-end model into at least one group, and the synchronization strategy is used to synchronize the model parameters of each group at N-step intervals when updating the end-to-end model, where N is a positive integer. Correspondingly, the aforementioned process of inputting the training set into the end-to-end model according to the training batch size and iteratively updating the end-to-end model based on its output to obtain the updated end-to-end model includes: inputting the training set into the end-to-end model according to the grouping strategy and the training batch size, and synchronizing the model parameters of each group at N-step intervals according to the synchronization strategy to obtain the updated end-to-end model.

[0020] In this embodiment, before training, a grouping strategy and a synchronization strategy can be determined, thereby improving training efficiency through group training and increasing the overall output accuracy of the model through synchronization.

[0021] In one possible implementation, the aforementioned iterative update of the end-to-end model based on the output of the end-to-end model to obtain the updated end-to-end model further includes: in one iteration, using a loss function to calculate the difference between the output of the end-to-end model and the corresponding true value, updating the end-to-end model based on the difference to obtain the updated end-to-end model, wherein the loss function includes a loss term for each task, and the weight parameter corresponding to the loss term for each task is determined based on the user input data.

[0022] In this embodiment, the user can adjust the weights of each task to improve the convergence efficiency of each task. For example, if the convergence speed of a task is slow, the weight of that task can be increased to improve the convergence speed of the model corresponding to that task.

[0023] Thirdly, this application provides an end-to-end intelligent driving system applied to a vehicle, comprising:

[0024] The input module is used to acquire map data and sensor data, including data collected by multiple sensors.

[0025] The feature extraction module is used to extract features from map data and sensor data to obtain perceptual features;

[0026] The perception module is used to acquire perception results based on perception features, and the perception results include information about at least one instance in the vehicle's environment.

[0027] The planning and control module is used to determine the vehicle's driving path based on the perception results.

[0028] The effects achieved by the third aspect and any optional implementation of the third aspect can be referred to the description of the first aspect or any optional implementation of the first aspect, and will not be repeated below.

[0029] In one possible implementation, the aforementioned perception results include at least one of the following: information on road structure in the vehicle's environment, information on moving targets, sparse features, or dense features.

[0030] In one possible implementation, the aforementioned perception module is specifically used to: perform multiple perception tasks on the perceived features through multiple modules, and output perception results. The multiple modules include a whitelist detection module, an obstacle detection module, a road information detection module, a forward perspective (PV) detection module, or a traffic light detection module. The whitelist detection module is used to detect information of instances of a preset category, the obstacle detection module is used to detect information of the drivable area of ​​the vehicle, the road information detection module is used to detect road information, the PV detection module is used to output information of instances from the PV perspective, and the traffic light detection information is used to output information of traffic lights.

[0031] In one possible implementation, the aforementioned planning and control module is specifically used to: acquire navigation field information, which includes recommended scores for vehicles traveling in different lanes according to the navigation path; and determine the driving path based on the navigation field information and perception results.

[0032] In one possible implementation, the aforementioned system further includes a control module for determining a vehicle control signal based on the driving path, the vehicle control signal being used to control the vehicle's movement.

[0033] In one possible implementation, the aforementioned system further includes:

[0034] The multi-objective optimization module is used to solve the problem based on the driving path using a multi-objective optimization algorithm to determine the vehicle's driving decision.

[0035] The control module is also used to generate vehicle control signals based on driving decisions.

[0036] In one possible implementation, the aforementioned system further includes: a multimodal tree module for determining at least one driving decision for the vehicle based on the driving path and the driving paths of other vehicles through the multimodal tree;

[0037] The multi-objective optimization module is specifically used to solve the problem according to the multi-objective optimization algorithm and determine the vehicle's driving decision from at least one driving decision.

[0038] Fourthly, this application provides a model training apparatus, comprising:

[0039] The acquisition module is used to acquire the training set, which includes samples of various types.

[0040] The training module is used to input the training set into the end-to-end model and iteratively update the end-to-end model based on its output to obtain the updated end-to-end model. The end-to-end model is used to perform multiple tasks and includes a feature extraction network, a perception network, and a regulation network. The feature extraction network is used to extract features from various types of input samples. The perception network is used to output perception results based on the features output by the feature extraction network. The regulation network is used to output vehicle driving decisions based on the perception results output by the perception network.

[0041] The effects achieved by the fourth aspect and any optional implementation of the fourth aspect can be referred to the description of the second aspect or any optional implementation of the second aspect, and will not be repeated below.

[0042] In one possible implementation, the aforementioned training module is further configured to: configure a corresponding training batch size for each of the multiple tasks before iteratively updating the end-to-end model based on the output of the end-to-end model, wherein the training batch size for each task is related to the duration of each task's step; input the training set into the end-to-end model according to the training batch size, and iteratively update the end-to-end model based on the output of the end-to-end model to obtain the updated end-to-end model.

[0043] In one possible implementation, the aforementioned training module is further configured to: determine a grouping strategy and a synchronization strategy before iteratively updating the end-to-end model based on the output of the end-to-end model; the grouping strategy is used to divide the parameters of the end-to-end model into at least one group; and the synchronization strategy is used to synchronize the model parameters of each group at N-step intervals when updating the end-to-end model, where N is a positive integer; input the training set into the end-to-end model according to the grouping strategy and the training batch size; and synchronize the model parameters of each group at N-step intervals according to the synchronization strategy to obtain the updated end-to-end model.

[0044] In one possible implementation, the aforementioned training module is further configured to: in one iteration, use a loss function to calculate the difference between the output of the end-to-end model and the corresponding ground truth, update the end-to-end model based on the difference, and obtain the updated end-to-end model, wherein the loss function includes a loss term for each task, and the weight parameters corresponding to the loss term for each task are determined based on the user input data.

[0045] Fifthly, this application provides a computing device, including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to perform processing-related functions in the data processing method shown in any of the first aspects above. Optionally, the computing device may be a chip.

[0046] Sixthly, this application provides a model training apparatus, comprising: a processor and a memory, wherein the processor and the memory are interconnected via circuits, and the processor calls program code in the memory to execute processing-related functions in the model training method shown in any of the second aspects above. Optionally, the model training apparatus may be a chip.

[0047] In a seventh aspect, embodiments of this application provide a vehicle including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to perform processing-related functions in the method shown in any of the first aspects above.

[0048] Eighthly, embodiments of this application provide a digital processing chip or chip, the chip including a processing unit and a communication interface, the processing unit obtains program instructions through the communication interface, the program instructions are executed by the processing unit, and the processing unit is used to perform processing-related functions as described in any optional embodiment of the first or second aspect above.

[0049] In a ninth aspect, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method in any of the optional embodiments of the first or second aspect described above.

[0050] In a tenth aspect, embodiments of this application provide a computer program product comprising a computer program / instructions, which, when executed by a processor, causes the processor to perform the method in any of the optional embodiments of the first or second aspect described above. Attached Figure Description

[0051] Figure 1 is a schematic diagram of the architecture of a vehicle provided in an embodiment of this application;

[0052] Figure 2 is a flowchart illustrating a model training method provided in an embodiment of this application;

[0053] Figure 3 is a schematic diagram of an end-to-end model provided in an embodiment of this application;

[0054] Figure 4 is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0055] Figure 5 is a schematic diagram of another end-to-end model provided in an embodiment of this application;

[0056] Figure 6 is a schematic diagram of another end-to-end model provided in an embodiment of this application;

[0057] Figure 7 is a schematic diagram of another end-to-end model provided in an embodiment of this application;

[0058] Figure 8 is a schematic diagram of another end-to-end model provided in an embodiment of this application;

[0059] Figure 9 is a schematic diagram of another end-to-end model provided in an embodiment of this application;

[0060] Figure 10 is a schematic diagram of an end-to-end intelligent driving system provided in an embodiment of this application;

[0061] Figure 11 is a schematic diagram of a model training device provided in an embodiment of this application;

[0062] Figure 12 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0063] Figure 13 is a schematic diagram of another model training device provided in an embodiment of this application. Detailed Implementation

[0064] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0065] The methods provided in this application involve neural networks. For ease of understanding, some terms or concepts involved in the embodiments of this application are explained below.

[0066] (1) Deep Neural Networks

[0067] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple intermediate layers. Based on the position of these layers, the internal neural network of a DNN can be divided into three categories: input layer, intermediate layers, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are considered intermediate layers, or hidden layers.

[0068] Although DNNs appear complex, each layer can be represented as a linear relational expression: in, It is the input vector. It is the output vector. is the offset vector, also known as the bias parameter; w is the weight matrix (also called coefficients); and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also quite large. The definitions of these parameters in DNNs are as follows: Taking the coefficient w as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0069] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0070] It's important to note that the input layer does not have a W parameter. In deep neural networks, more intermediate layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0071] (2) Convolutional Neural Network

[0072] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN includes a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting. For example, in the embodiments of this application, the feature extraction module in the end-to-end model can use a CNN or a neural network containing convolutional layers.

[0073] (3) transformer

[0074] A transformer structure is a feature extraction network that includes both an encoder and a decoder (classified as a convolutional neural network). Of course, in some cases, a transformer structure may not include an encoder but may include a decoder.

[0075] Encoder: Learns features, such as pixel features, in the global receptive field through self-attention.

[0076] Decoder: Learns the features of the desired module, such as the features of the output box, through self-attention and cross-attention.

[0077] For example, a Transformer layer structure may include an attention network and a feedforward network module. Taking natural language processing as an example, the attention network obtains corresponding weight values ​​by calculating the relevance between words based on the attention mechanism, thus obtaining context-related word representations, which is the core part of the Transformer structure. The feedforward network further transforms the obtained representations to obtain the final output of the Transformer layer. In addition to these two important components, residual layers (ADD) and linear normalization (Norm) are also stacked on these two components to optimize the output of the Transformer layer.

[0078] The method provided in this application can be applied to various smart devices, such as vehicles, robots, or other in-vehicle devices.

[0079] For example, for intelligent vehicles, the vehicle's perception of the environment and road structure recognition are crucial. Intelligent driving functions can include modules such as perception, prediction, decision-making, planning, and control. The decision-making, planning, and control modules can also be called planning and control functions. For instance, intelligent driving functions typically include lane departure warning (LDW) or lane keeping assist (LKA) functions from advanced driving assistance systems (ADAS). Lane departure warning and lane keeping assist require real-time detection of lane lines and determination of whether the vehicle is deviating from its current lane. If the vehicle deviates from its lane without a turn signal being given, it will issue a warning signal, vibrate the steering wheel, or even actively pull the steering wheel back to remind the driver to return to the lane. The method provided in this application embodiment can achieve a more accurate end-to-end model, realizing intelligent driving functions with lower errors.

[0080] For example, in robot applications, sensor data collected by the robot can be fused with initial map data to obtain a more accurate and information-rich fused map. Based on this, accurate perception of the robot's environment and high-precision positioning of the robot can be achieved, thus planning a more suitable route for the robot based on high-precision positioning and the fused map.

[0081] The following describes the device structure used in the method provided in this application embodiment, taking deployment in a vehicle as an example.

[0082] For example, taking the method provided in this application embodiment as an example of deployment in a vehicle, the structure of the vehicle can be as shown in Figure 1.

[0083] Referring to Figure 1, which is a schematic diagram of a vehicle structure provided in an embodiment of this application, the vehicle 100 can be configured in an intelligent driving mode. For example, the vehicle 100 can control itself while in intelligent driving mode, and can determine the current state of the vehicle and its surrounding environment through human operation, determine whether there are obstacles in the surrounding environment, and control the vehicle 100 based on the obstacle information. When the vehicle 100 is in intelligent driving mode, it can also be set to operate without human interaction.

[0084] Please refer to Figure 1 first. Figure 1 is a structural schematic diagram of a vehicle provided in an embodiment of this application. Figure 1 is a functional block diagram of a vehicle 100 provided in an embodiment of this application. The vehicle 100 can be configured to a full or partial intelligent driving mode. For example, the vehicle 100 can obtain environmental information about its surroundings through the perception system 120, and obtain an intelligent driving strategy based on the analysis of the surrounding environmental information to achieve full intelligent driving, or present the analysis results to the user to achieve partial intelligent driving.

[0085] Vehicle 100 may include various subsystems, such as an infotainment system 110, a perception system 120, a decision control system 130, a drive system 140, and a computing platform 150. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0086] In some embodiments, the infotainment system 110 may include a communication system 111, an entertainment system 112, and a navigation system 113.

[0087] Communication system 111 may include wireless communication system 111, which can communicate wirelessly with one or more devices directly or via a communication network. For example, wireless communication system 111 may use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. Wireless communication system 111 may communicate using WiFi and wireless local area network (WLAN). In some embodiments, wireless communication system 146 may communicate directly with devices using an infrared link, Bluetooth, or ZigBee. Wireless communication system 111 may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0088] The entertainment system 112 may include a central control screen, a microphone, and speakers. Users can listen to the radio and play music within the vehicle using the entertainment system 112; or connect their mobile phones to the vehicle and project their screens onto the central control screen, which may be touch-sensitive, allowing users to operate the system. In some cases, the microphone can capture the user's voice signal, and analysis of this signal can enable the user to control certain aspects of the vehicle 100, such as adjusting the interior temperature. In other cases, music can be played to the user through the speakers.

[0089] The navigation system 113 may include map services to provide navigation for the vehicle 100, and the navigation system 113 may be used in conjunction with the vehicle's global positioning system 121 and inertial measurement unit 122. The map may be a two-dimensional map, a high-precision map, or a map constructed based on data collected during the vehicle's operation.

[0090] The perception system 120 may include several sensors for sensing information about the environment surrounding the vehicle 100. For example, the perception system 120 may include a global positioning system 121 (which may include a global navigation satellite system (GNSS), specifically GPS, BeiDou, or other positioning systems), an inertial measurement unit (IMU) 122, a lidar (LiDAR) 123, a millimeter-wave radar 124, an ultrasonic radar 125, and a camera device 126. The perception system 120 may also include sensors from the internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of the vehicle 100. The data collected by sensors in the vehicle mentioned below may include data collected by various units in the perception system 120, such as point cloud data collected by lidar or environmental images collected by the camera device.

[0091] The Global Positioning System 121 can be used to determine the geographical location of vehicle 100.

[0092] The inertial measurement unit 122 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In some embodiments, the inertial measurement unit 122 may be a combination of an accelerometer and a gyroscope.

[0093] The lidar 123 can use lasers to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the lidar 123 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components.

[0094] The millimeter-wave radar 124 can use radio signals to sense objects in the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing objects, the millimeter-wave radar 124 can also be used to sense the speed and / or direction of travel of objects.

[0095] The ultrasonic radar 125 can use ultrasonic signals to sense objects around the vehicle 100.

[0096] The camera device 126 can be used to capture image information of the surrounding environment of the vehicle 100. The camera device 126 may include a monocular camera, a binocular camera, a structured light camera, and a panoramic camera, etc. The image information acquired by the camera device 126 may include still image information or video stream information.

[0097] The decision control system 130 includes a computing system 131 that analyzes and makes decisions based on information acquired by the sensing system 120. The decision control system 130 also includes a vehicle controller 132 that controls the power system of the vehicle 100, and a steering system 133, a throttle 134, and a braking system 135 for controlling the vehicle 100.

[0098] The computing system 131 can process and analyze various information acquired by the perception system 120 to identify targets, objects, and / or features in the environment surrounding the vehicle 100. The targets may include pedestrians or animals, and the objects and / or features may include traffic signals, road boundaries, and obstacles. The computing system 131 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other techniques. In some embodiments, the computing system 131 may be used to map the environment, track objects, estimate object speeds, etc. The computing system 131 can analyze the acquired information and derive a control strategy for the vehicle.

[0099] The vehicle controller 132 can be used to coordinate the control of the vehicle's power battery and engine 141 to improve the power performance of the vehicle 100.

[0100] The steering system 133 can be used to adjust the forward direction of the vehicle 100. For example, in one embodiment, it can be a steering wheel system.

[0101] The throttle 134 is used to control the operating speed of the engine 141 and thus the speed of the vehicle 100.

[0102] Braking system 135 is used to control the deceleration of vehicle 100. Braking system 135 can use friction to slow down the rotational speed of wheel 144. In some embodiments, braking system 135 can convert the kinetic energy of wheel 144 into electric current. Braking system 135 may also take other forms to slow down the rotational speed of wheel 144 to control the speed of vehicle 100.

[0103] The drive system 140 includes components that provide powered motion to the vehicle 100. In one embodiment, the drive system 140 may include an engine 141, an energy source 142, a transmission system 143, and wheels 144. The engine 141 may be an internal combustion engine, an electric motor, an air-compressed engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. The engine 141 converts the energy source 142 into mechanical energy.

[0104] Examples of energy sources 142 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 142 may also provide energy to other systems of vehicle 100.

[0105] The drivetrain 143 transmits mechanical power from the engine 141 to the wheels 144. The drivetrain 143 may include a gearbox, a differential, and a drive shaft. In one embodiment, the drivetrain 143 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 144.

[0106] Some or all of the functions of vehicle 100 are controlled by computing platform 150. Computing platform 150 may include at least one processor 151, which can execute instructions 153 stored in a non-transitory computer-readable medium such as memory 152. In some embodiments, computing platform 150 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner.

[0107] Processor 151 can be any conventional processor, such as a commercially available CPU. Alternatively, processor 151 may also include a graphics processing unit (GPU), a field-programmable gate array (FPGA), a system-on-chip (SoC), an application-specific integrated circuit (ASIC), or a combination thereof. Processor 151 can be located on a device remote from the vehicle and can communicate wirelessly with the vehicle.

[0108] In some embodiments, memory 152 may contain instructions 153 (e.g., program logic) that can be executed by processor 151 to perform various functions of vehicle 100. Memory 152 may also contain additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the infotainment system 110, perception system 120, decision control system 130, and drive system 140.

[0109] The method provided in this application embodiment can be executed by a computing platform 150. For example, the processor 151 can read the program stored in the memory 152, fuse the data collected by the perception system 120 with the initial map, output a higher precision map, and determine the vehicle's driving decision based on the higher precision map. By sending instructions to the decision control system 130, the vehicle can be controlled by the decision control system 130 to realize the intelligent driving function.

[0110] Of course, the method provided in this application embodiment can also be directly deployed in the perception system 120 to output the high-precision positioning of the vehicle, or it can be deployed in the decision control system 130. The decision control system 130 fuses the data from the perception system with the initial map to obtain a higher-precision map. Based on the higher-precision map, it realizes the perception of the vehicle environment and the positioning of the vehicle, and controls the vehicle based on the perceived target to realize the intelligent driving function of the vehicle.

[0111] In addition to instruction 153, memory 152 may also store data such as road maps, route information, vehicle position, direction, speed, and other similar vehicle data, as well as other information. It may also store higher-precision maps obtained by the methods provided in this application embodiment. This information can be used by vehicle 100 and computing platform 150 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0112] The computing platform 150 can control the functions of the vehicle 100 based on inputs received from various subsystems, such as the drive system 140, the perception system 120, and the decision control system 130. For example, the computing platform 150 can utilize inputs from the decision control system 130 to control the steering system 133 to avoid obstacles detected by the perception system 120. In some embodiments, the computing platform 150 is operable to provide control over many aspects of the vehicle 100 and its subsystems.

[0113] Alternatively, one or more of these components may be installed separately from or associated with vehicle 100. For example, memory 152 may exist partially or completely separately from vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0114] Optionally, the above components are just an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 1 should not be construed as a limitation on the embodiments of this application.

[0115] The aforementioned vehicle 100 can be any vehicle or vehicle-mounted terminal capable of intelligent driving, such as a car, truck, motorcycle, bus, ship, airplane, helicopter, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, or train. This application embodiment does not impose any special limitations on this type of vehicle.

[0116] Vehicle intelligent driving functions can be divided into modules such as perception, prediction, decision-making, planning, and control. Existing solutions, such as a full-stack autonomous driving planning scheme, can execute multiple tasks with high precision, including tracking, map generation, path prediction, grid occupancy prediction, and planning. By connecting the various modules of the architecture through a unified query vector, the complex interactions between the agent and the environment are fully modeled, continuously extracting the agent's motion information, and applying the results of upstream module tasks to driving planning, ultimately achieving safe planning of autonomous driving paths. However, during the training phase, the optimization objectives of each module are inconsistent, and error propagation may occur between modules, leading to significant errors in the final output. Furthermore, some existing multi-task models are difficult to train and are complex, making them difficult to apply in real-world scenarios.

[0117] Therefore, this application provides a model training method and data processing method based on an end-to-end model. In the training phase, by using group training, an efficient and easily convergent scheme can be achieved. Furthermore, the coupled training of various tasks yields a complete end-to-end model coupled with multiple tasks, thereby reducing the error transmitted between tasks and improving the higher output accuracy of the end-to-end model in the inference phase.

[0118] The flow of the method provided in the embodiments of this application will be described below.

[0119] The method provided in this application can be divided into a training phase and an inference phase. In the training phase, the end-to-end model provided in this application is trained. In the inference phase, downstream tasks are executed using the end-to-end model provided in this application. The training and inference phases can be deployed on the same device or on different devices. For example, the training phase can be deployed in the cloud or on a server, while the inference phase can be deployed on a smart terminal, such as a mobile phone, tablet, smart car, robot, or wearable device.

[0120] The training phase and inference phase of the method provided in the embodiments of this application will be described separately below.

[0121] I. Training Phase

[0122] Referring to Figure 2, a flowchart of a model training method provided in an embodiment of this application is shown below.

[0123] 201. Obtain the training set, which includes samples of various types.

[0124] The training set may include samples of types such as map data or sensor data.

[0125] This can be understood as follows: the samples in the training set can be divided into map data and sensor data. Map data includes map vector data, while sensor data includes data collected by sensors, such as samples collected by image sensors, millimeter-wave radar, lidar, or infrared sensors. Correspondingly, these can be samples such as images or point clouds.

[0126] For example, in some scenarios, map data and various types of sensor data can be collected, labeled, and used as a training set. The labels can include driving decisions or drivable areas corresponding to the samples, and can be done manually or using pre-trained neural networks.

[0127] 202. Input the training set into the end-to-end model, and iteratively update the end-to-end model based on the output of the end-to-end model to obtain the updated end-to-end model.

[0128] During training, samples from the training set are input into the initial end-to-end model, and the end-to-end model is updated in reverse based on the output of the end-to-end model. A converged end-to-end model can be obtained through multiple iterations.

[0129] Specifically, the end-to-end model is used to perform multiple tasks. The end-to-end model includes a feature extraction network, a perception network, and a regulation network. The feature extraction network is used to extract features from various types of input samples. The perception network is used to output perception results based on the features output by the feature extraction network. The regulation network is used to output vehicle driving decisions based on the perception results output by the perception network.

[0130] During training, a joint optimization objective can be set for the multiple tasks. For example, a corresponding loss term can be set for each task in the loss function, so that the tasks are coupled in the end-to-end model, thereby improving the overall output accuracy of the model and obtaining an end-to-end model that performs better in each task.

[0131] Specifically, the structure of this end-to-end model can be found in the following description of the inference phase, which will not be repeated here.

[0132] In one possible implementation, a corresponding training batch size can be configured for each of the multiple tasks. The training batch size for each task is related to the duration of each step in the task. During model training, the training set can be input into the end-to-end model according to the training batch size, and the end-to-end model can be iteratively updated according to the output of the end-to-end model to obtain the updated end-to-end model.

[0133] Typically, batch size affects the duration of each step. For example, the larger the batch size, the longer the duration of a step. Different tasks may have different durations for the next step with the same batch size. Therefore, in this application, by setting the batch size, the step durations of each task are made as close as possible or the same, so that the training progress of each task is more synchronized during the model training process.

[0134] In one possible implementation, a grouping strategy and a synchronization strategy can be determined. The grouping strategy is used to divide the parameters of the end-to-end model into at least one group, such as grouping modules corresponding to each task into one group, or grouping modules corresponding to multiple tasks into one group. The synchronization strategy is used to synchronize the model parameters of each group at N-step intervals when updating the end-to-end model, where N is a positive integer. During training, the training set is input into the end-to-end model according to the grouping strategy and the training batch size, and the model parameters of each group are synchronized at N-step intervals according to the synchronization strategy to obtain the updated end-to-end model. Therefore, in this embodiment, the model training process can be divided into smaller-granularity, synchronous processes through grouping and synchronization, thereby improving the convergence speed of the model and reducing the computational complexity of model training. Furthermore, a grouping or synchronization strategy can be set for each task, so that the trained model performs better on each task and reduces the error propagation between modules of different tasks.

[0135] Optionally, in one iteration, the difference between the output of the end-to-end model and the corresponding ground truth can be calculated using a loss function. The end-to-end model is then updated based on this difference, resulting in an updated end-to-end model. The loss function includes a loss term for each task, and the weight parameters corresponding to the loss term for each task are determined based on the user's input data. For example, during model training, the results of each task can be evaluated. The user can adjust the weights of each task in real time based on the training progress of each task, thereby adjusting the convergence speed of the tasks. If a task performs poorly in convergence, the weights of each task can be adjusted in real time during the step without terminating the task.

[0136] For example, the process of the model training phase in the method provided in this application embodiment can be represented as follows:

[0137] First, configure different task configuration files, such as yaml files;

[0138] Then, the corresponding batch size is set for different tasks so that the step time of each task is similar, which means that the training speed of each step of each task is similar.

[0139] Adjustable parameters can be set for different tasks. For example, weights for each task can be set in the overall loss function. These weights can be adjusted to adjust the convergence speed of each task during training.

[0140] Set up grouping and synchronization strategies. The synchronization strategy, also known as the synchronous batch normalization (syncbn) strategy, groups the model parameters. The grouping method can be based on training nodes. A training node can be a GPU or a CPU, or a server or multiple servers, etc. This divides the model parameters into multiple groups and synchronizes the parameters at intervals of N steps to achieve very high training efficiency.

[0141] During model training, the currently trained model can be evaluated, and the evaluation results can be used to represent the convergence performance of each task. For example, if the convergence performance of a certain task is poor, the aforementioned adjustable parameters can be modified, such as changing the weight of each task in the loss function, so that the task can achieve the expected convergence speed. Moreover, without terminating the task, the new adjustable parameters can be loaded into the step for training, enabling online adjustment during model training.

[0142] The method provided in this application, with its modular end-to-end model, is easy to integrate and extend, enabling it to adapt to a wide range of tasks in the field of intelligent driving. The ability to independently configure and adjust parameters for each task provides significant flexibility, allowing it to adapt to different datasets and requirements.

[0143] II. Reasoning Stage

[0144] After the aforementioned training phase, the model can be trained to execute the inference process using the trained end-to-end model.

[0145] First, the structure of the end-to-end model provided in this application embodiment can be as shown in Figure 3. This end-to-end model may include a feature extraction network 301, a perception network 302, and a planning and control network 303, etc.

[0146] The feature extraction network 301 can be used to extract features from multiple types of input samples. This allows for the extraction of dimension-aligned features available to downstream modules from various sample types.

[0147] The perception network 302 is used to output perception results based on the features extracted by the feature extraction network. This perception network can specifically perform one or more perception tasks. For example, it may include, but is not limited to, a whitelist detection module, an obstacle detection module, a road information detection module, a forward perspective (PV) detection module, or a traffic light detection module. The whitelist detection module is used to detect information about instances of a preset category; the obstacle detection module is used to detect information about the drivable area of ​​a vehicle; the road information detection module is used to detect road information; the PV detection module is used to output information about instances from a PV perspective; and the traffic light detection module is used to output information about traffic lights, etc.

[0148] The planning and control network 303 is used to output the vehicle's driving decisions based on the perception results output by the perception network. For example, the planning and control network can determine the vehicle's driving decisions based on the environmental conditions output by the perception network, such as surrounding obstacles or the driving paths of other vehicles.

[0149] The following section describes the structure of the aforementioned end-to-end model and the inference phase of the method provided in the embodiments of this application.

[0150] Referring to Figure 4, a flowchart of a data processing method provided in an embodiment of this application is shown below.

[0151] 401. Obtain map data and sensor data.

[0152] The map data can include high-precision maps, standard-precision maps, low-precision maps, or other vector maps, which can be determined according to the actual application scenario.

[0153] Sensor data may include data collected by one or more sensors in the vehicle as shown in Figure 1 above. For example, if the vehicle includes an image sensor, the sensor data may include images collected by the image sensor; if the vehicle includes radar, such as lidar, millimeter-wave radar, or ultrasonic radar, the sensor data may include point cloud data collected by the radar. The specific data included in the sensor data can be determined according to the actual application scenario. Different vehicles may have different types of sensors, and this application embodiment does not limit this.

[0154] 402. Extract features from map data and sensor data to obtain perception features.

[0155] Combining the aforementioned end-to-end model, map data and sensor data are input into a feature extraction network, which then performs feature extraction. This feature extraction network can be a CNN, a network containing convolutional layers, or an encoder, and can output features of a uniform dimension.

[0156] 403. Obtain the perception results based on the perception characteristics.

[0157] The perceptual features output by the feature extraction network can be processed to perceive the environment in which the vehicle is located, and the perception results can be obtained based on the perceptual features.

[0158] Specifically, the sensing network may include, but is not limited to, a whitelist detection module, an obstacle detection module, a road information detection module, a forward-looking PV detection module, or a traffic light detection module.

[0159] Specifically, the type of perception result output by the perception network can be determined based on the actual application scenario, and the type of features to be output by the feature extraction network can be determined based on the actual functional requirements. For example, the perception result output by the perception network may include, but is not limited to, information on road structure in the vehicle's environment, information on moving targets, sparse features, or dense features. Road structure refers to information representing the road structure in the vehicle's environment, such as the shape, location, size, lane lines, or lane center lines. Moving targets can include targets in the vehicle's environment with a speed greater than 0 or greater than a set speed, such as pedestrians, other vehicles, or other moving obstacles. The difference between sparse and dense features usually lies in the extraction method. Dense features have fewer missing values ​​or fewer zero values ​​than sparse features, making downstream tasks more computationally efficient for sparse features, while dense features can contain richer features.

[0160] 404. Determine the vehicle's driving path based on the perception results.

[0161] After obtaining the perception results, the specific conditions of the vehicle's environment can be identified, such as the road structure, traffic lights, obstacles, or the driving paths of other vehicles. Based on the perception results, the vehicle's driving decision can be determined. For example, based on the perception results, obstacle avoidance, efficient passage, faulty vehicle detection, parked vehicle detection, queued vehicle detection, or traffic jam detection can be implemented to plan a more suitable driving path for the vehicle.

[0162] For example, in obstacle avoidance scenarios, the method provided in this application embodiment can determine the location of obstacles or driving paths more efficiently and accurately in an end-to-end manner, thereby improving the driving safety of the vehicle.

[0163] For example, in scenarios requiring efficient traffic flow, the method provided in this application embodiment can identify drivable paths more efficiently and accurately through an end-to-end approach, thereby improving the driving efficiency of the vehicle.

[0164] For example, in scenarios such as parking vehicle detection, queuing vehicle detection, or traffic jam detection, the method provided in this application embodiment can more accurately detect parked vehicles, stationary or moving obstacles in the environment by using multimodal data in an end-to-end manner, thereby achieving more accurate target detection.

[0165] In this embodiment, an end-to-end model is used to implement multiple tasks in the intelligent driving function of the vehicle. The modules executing each task are coupled. For example, a feature extraction network can extract features that meet the dimensional requirements for downstream modules, and the perception network can perceive the vehicle environment in various dimensions to accurately identify the situation of the vehicle's environment. Finally, the planning and control network can plan more suitable driving decisions for the vehicle. Therefore, the coupling between the modules realizes an end-to-end model that can execute multiple tasks, reduces error propagation between modules, and obtains an end-to-end model with higher output accuracy.

[0166] Optionally, in determining a driving decision, navigation field information can be obtained, including recommended scores for different lanes when the vehicle travels along the navigation path, or recommended probabilities for each lane, etc.; at least one driving decision is determined based on the navigation field information and perception results. Therefore, in this embodiment of the application, when navigating the vehicle, the navigation field can also be referenced when determining the vehicle's driving decision, thereby further determining a more accurate driving decision.

[0167] Optionally, after determining the vehicle's driving path, a vehicle control signal can be determined based on at least one determined driving decision, thereby controlling the vehicle's driving through the vehicle control signal to achieve intelligent driving of the vehicle.

[0168] In one possible implementation, a multi-objective optimization algorithm can be used to solve for the driving path and determine the vehicle's driving decision. The optimization objective of this multi-objective optimization algorithm can be set according to the actual application scenario, such as the distance between the vehicle and obstacles, the distance between the vehicle and lane lines, or the turning angle of the vehicle. Subsequently, vehicle control signals are generated based on the driving decision. It can be understood that the end-to-end model provided in this application embodiment may also include a multi-objective optimization module, which can use a multi-objective optimization algorithm to obtain a better driving decision from multiple optimization dimensions.

[0169] In one possible implementation, at least one driving decision for the vehicle can be determined using a multimodal tree based on the driving path and the driving paths of other vehicles; a multi-objective optimization algorithm is then used to solve the problem, and the final driving decision for the vehicle is determined from the at least one driving decision. It can be understood that the end-to-end model provided in this application embodiment can also incorporate a multimodal tree, utilizing multimodal paths to obtain available driving decisions for the vehicle. In this application embodiment, a multimodal tree, combined with multimodal driving paths, can be used to make a more suitable driving decision for the vehicle, improving driving safety.

[0170] The following section provides a further description of the specific structure of the end-to-end model provided in the embodiments of this application, in conjunction with the aforementioned method and process.

[0171] Referring to Figure 5, this application provides a schematic diagram of another end-to-end model.

[0172] It is divided into input data, feature extraction network, perception network, planning and control network, and control module.

[0173] 1. Input data

[0174] In the input section, various types of data are input. This embodiment uses map data and sensor data as examples, with the sensor data including images and point cloud data. Point cloud data can come from LiDAR, millimeter-wave radar, or ultrasonic radar, etc.

[0175] 2. Feature Extraction Network

[0176] Feature extraction networks can be used to extract desired features from map data and sensor data. A feature extraction network is essentially the backbone network in an end-to-end model, and can specifically include CNNs or other networks containing convolutional layers, as well as encoders or other networks that can be used for feature encoding.

[0177] Therefore, in this embodiment, information from various sensors such as lidar, millimeter-wave radar, and cameras can be fully utilized and converted into features with uniform dimensions through a feature encoder. For example, each type of feature can be mapped to the BEV space or PV space, so that navigation map information and sensor data can be combined to provide a more comprehensive understanding of the environment for perception and planning.

[0178] 3. Perceptual Networks

[0179] A perception network may include one or more perception modules, each of which can perform a perception task. For example, this perception network may include a whitelist detection module (i.e., SparseBEV in Figure 5), a general obstacle detection network (GOD) module, a road information monitoring (COG) module, a PV detection module, and a traffic light detection module.

[0180] (1) SparseBEV, also known as the aforementioned whitelist detection module, can be used for instance detection. The output of SparseBEV can include information from the perspective of BEVs or PVs, specifically including but not limited to:

[0181] 3D instance detection results and the corresponding attributes of the instances. For example, the 3D instance detection results may include, but are not limited to, the position coordinates (x, y, z), confidence (conf), classification (cls), distance (ry), size (lx, ly, lz) or velocity (vx, vy) of the 3D instance; attributes such as turn signals, brake lights, long and short stops, traffic signs (trackid) and other information.

[0182] Instance features, and the hidden features corresponding to each obj.

[0183] (2) The GOD module can be used to output information from the BEV perspective, such as including but not limited to one or more of the following:

[0184] BEV characteristics of obstacles;

[0185] Information such as the occupancy, semantics, and visibility of obstacles;

[0186] Information on drivable areas, etc.

[0187] (3) The COG module can be used for road detection and output information about the detected roads. Specifically, it can output, but is not limited to:

[0188] Road element detection results: such as lane, lane attributes, intersection, crosswalk, curb polyline, waiting area, etc.

[0189] BEV features or instance features corresponding to static road elements, etc.

[0190] (4) The PV detection module can be used to output information from the PV perspective, including information about each instance from the PV perspective. For example, it can output hidden layer features, 2D segmentation maps (such as semantic segmentation, slippery road surface or drivable space (freespace)) from the PV perspective.

[0191] (5) The traffic light detection module is used to output the 2D detection frame of traffic lights (i.e., traffic signals) or the specific category of traffic lights.

[0192] The output interface of the perception network connects to the input interface of the control network, transmitting the perception results to the control network. In this embodiment, the perception network can extract features of the desired dimension from the output of the feature extraction network, thereby enabling the control network to process perception results obtained from various types of input data.

[0193] 4. Regulatory network

[0194] The perceptual road structure information and hidden layer features output by the perception network are transmitted to the planning and control network. The planning and control network can output the vehicle's driving path, and further, it can also output the vehicle's driving path and the multimodal driving paths of other vehicles.

[0195] Specifically, the perception network can also include a navigation field module, trafficCNN, and a pre-decision module. The navigation field module takes the navigation output as input and outputs the navigation field, which is the recommended score for the vehicle in each lane when following the navigation path. The input to trafficCNN includes the navigation field and the perception results output by the perception network, and outputs the probability of passage for each lane. The input to the pre-decision network includes the communication probability output by trafficCNN and the perception results output by the perception network, and outputs the vehicle's driving path and the driving paths of other vehicles in the environment.

[0196] Of course, in scenarios where navigation is not required, the navigation field module may not need to be started, or even trafficCNN may not need to be started. The specific modules to be deployed or started can be determined according to the actual application scenario. The embodiments in this application are only illustrative examples and are not intended to limit the scope.

[0197] Furthermore, through the modular processing of each task in the end-to-end model provided in this application, the accuracy of the regulatory control output path can be guaranteed while ensuring that the accuracy of the output of each perception module remains unaffected, and the active safety-related functions are not affected. In addition, compared with existing solutions, the access of the perception network to the perception network output results can avoid problems such as error accumulation between modules, information loss, inconsistency, and excessive frame compression. Moreover, some typical scenarios can be further optimized, such as CURRENT intrusion, pedestrian crossing intent judgment, multi-objective interactive game at intersections, large vehicle cutin, and illegal parking escape.

[0198] 5. Control Module

[0199] Based on the vehicle path output by the control network and the paths of other vehicles, a control module can generate control signals to control the vehicle to travel along the set path.

[0200] Therefore, in this embodiment, the autonomous vehicle's driving path output by the planning and control network and the driving paths of other vehicles can be used to control the autonomous vehicle to drive along an accurate path.

[0201] Furthermore, the modules in the aforementioned end-to-end model can be deployed in various structures or forms. Some possible deployment structures are described below.

[0202] Deployment Structure 1

[0203] Referring to Figure 6, this application provides a schematic diagram of the structure of an end-to-end model. The input to the regulatory network is the output of the perception model.

[0204] The output of a perception network can include various types, specifically those shown in Figure 5. For example, as shown in Figure 6, it can be divided into feature sequences and element detection results. Feature sequences can include instance features, BEV features, or PV features, etc.; detection results can specifically include moving target detection results, traffic light detection results, static element detection results, GOD detection results, or PV detection results, etc.

[0205] The planning and control network outputs the driving path of the vehicle and the driving paths of other vehicles. The output of the planning and control network is connected to a multimodal tree, which then passes through the multimodal tree, a multi-objective programming (MOP) model, and a control module. The multimodal tree can be used to combine different types of data to make decisions and output more accurate driving decisions; for example, it can determine multiple driving decisions based on the input driving path of the vehicle and the driving paths of other vehicles. The MOP can further search for the optimal solution among the driving decisions output by the multimodal tree, outputting a driving decision more suitable for the vehicle. The control module then generates control signals for the vehicle based on the driving decisions output by the MOP to control the vehicle to drive according to the driving decisions.

[0206] Deployment Structure 2

[0207] Referring to Figure 7, this application provides a schematic diagram of the structure of an end-to-end model.

[0208] The input to the planning and control network is the output of the perception model. The planning and control network outputs the driving path of the vehicle and the driving paths of other vehicles. The output of the planning and control network is connected to the Model Optimizer (MOP), and the output of the MOP is connected to the control module. The MOP can be used to search for the optimal solution from the driving path output by the planning and control network, and output a driving decision that is more suitable for the vehicle. The control module can then generate control signals for the vehicle based on the driving decision output by the MOP, so as to control the vehicle to drive according to the driving decision.

[0209] Deployment Structure 3

[0210] Referring to Figure 8, this application provides a schematic diagram of the structure of an end-to-end model.

[0211] The end-to-end model also includes a post-processing module. The input to the planning and control network includes the output of the perception network and the output of the post-processing module. The output of the planning and control network is connected to the multimodal tree, and after passing through the MOP and the control module, it outputs the vehicle's control signal.

[0212] The post-processing module can be used to post-process the perception results, such as performing frame stacking or filtering on the target, and outputting the post-processed moving target detection results or static element detection results.

[0213] The control network determines the driving path of the vehicle and the driving paths of other vehicles based on the feature sequence output by the perception network, the detection results of elements, and the detection results output by the post-processing module.

[0214] The vehicle paths output by the control network and other vehicle paths are input into the multimodal tree. The steps executed by the multimodal tree, MOP, and control module can be found in the description of the aforementioned deployment structure one, and will not be repeated here.

[0215] Deployment Structure Four

[0216] Referring to Figure 9, this application provides a schematic diagram of the structure of an end-to-end model.

[0217] The end-to-end model includes a perception network, a post-processing module, a planning and control network, a vehicle operating system (MOP), and a control module. The inputs to the planning and control network include the outputs of the perception network and the post-processing module. The output of the planning and control network is connected to the MOP and then outputs the vehicle's control signals via the control module.

[0218] The post-processing module can be referred to in the description of deployment structure three above, and the steps executed by the MOP and control module can be referred to in the description of deployment structure two above, which will not be repeated here.

[0219] Therefore, the method provided in this application embodiment can be deployed in application scenarios in various ways, providing a more complete end-to-end model. By accessing sensing features in the regulatory network, it reduces error transmission between modules and improves the overall output accuracy of the end-to-end model.

[0220] The foregoing has described the method flow provided in the embodiments of this application. The following describes the structure of the apparatus for performing the foregoing method.

[0221] Referring to Figure 10, this application provides a schematic diagram of an end-to-end intelligent driving system, which can be used to perform the steps of the aforementioned reasoning stage. This device can be applied to a vehicle and includes:

[0222] Input module 1001 is used to acquire map data and sensor data, including data collected by multiple sensors;

[0223] Feature extraction module 1002 is used to extract features from map data and sensor data to obtain perception features;

[0224] The perception module 1003 is used to obtain perception results based on perception features, wherein the perception results include information on at least one instance in the vehicle's environment;

[0225] The planning and control module 1004 is used to determine the vehicle's driving path based on the perception results.

[0226] In one possible implementation, the aforementioned perception results include at least one of the following: information on road structure in the vehicle's environment, information on moving targets, sparse features, or dense features.

[0227] In one possible implementation, the aforementioned perception module 1003 may specifically be the perception network in the aforementioned method flow, specifically used to: perform multiple perception tasks on the perception features through multiple modules, and output perception results. The multiple modules include a whitelist detection module, an obstacle detection module, a road information detection module, a forward perspective PV detection module, or a traffic light detection module. The whitelist detection module is used to detect information of instances of a preset category, the obstacle detection module is used to detect information of the drivable area of ​​the vehicle, the road information detection module is used to detect road information, the PV detection module is used to output information of instances under the PV view, and the traffic light detection information is used to output information of traffic lights.

[0228] In one possible implementation, the aforementioned planning and control module 1004 may specifically be the planning and control network in the aforementioned method flow, specifically used for: acquiring navigation field information, which includes the recommended scores corresponding to the vehicle driving in different lanes when driving according to the navigation path; and determining the driving path based on the navigation field information and perception results.

[0229] In one possible implementation, the aforementioned device further includes a control module 1005, used to determine a vehicle control signal based on the driving path, the vehicle control signal being used to control the vehicle's movement.

[0230] In one possible implementation, the aforementioned system further includes:

[0231] The multi-objective optimization module 1006 is used to solve the problem based on the driving path using a multi-objective optimization algorithm to determine the vehicle's driving decision.

[0232] The control module is also used to generate vehicle control signals based on driving decisions.

[0233] In one possible implementation, the aforementioned system further includes:

[0234] The multimodal tree module 1007 is used to determine at least one driving decision for a vehicle based on the driving path and the driving paths of other vehicles through the multimodal tree.

[0235] The multi-objective optimization module is specifically used to solve the problem according to the multi-objective optimization algorithm and determine the vehicle's driving decision from at least one driving decision.

[0236] Referring to Figure 11, this application provides a schematic diagram of a model training device that can be used to perform the aforementioned training phase steps. The device includes:

[0237] The acquisition module 1101 is used to acquire the training set, which includes samples of various types.

[0238] The training module 1102 is used to input the training set into the end-to-end model and iteratively update the end-to-end model based on the output of the end-to-end model to obtain the updated end-to-end model. The end-to-end model is used to perform multiple tasks. The end-to-end model includes a feature extraction network, a perception network, and a regulation network. The feature extraction network is used to extract features from various types of input samples. The perception network is used to output perception results based on the features output by the feature extraction network. The regulation network is used to output vehicle driving decisions based on the perception results output by the perception network.

[0239] In one possible implementation, the aforementioned training module 1102 is further configured to: configure a corresponding training batch size for each of the multiple tasks before iteratively updating the end-to-end model based on the output of the end-to-end model, wherein the training batch size for each task is related to the duration of the step of each task; input the training set into the end-to-end model according to the training batch size, and iteratively update the end-to-end model based on the output of the end-to-end model to obtain the updated end-to-end model.

[0240] In one possible implementation, the aforementioned training module 1102 is further configured to: determine a grouping strategy and a synchronization strategy before iteratively updating the end-to-end model based on the output of the end-to-end model; the grouping strategy is used to divide the parameters of the end-to-end model into at least one group; the synchronization strategy is used to synchronize the model parameters of each group at intervals of N steps when updating the end-to-end model, where N is a positive integer; input the training set into the end-to-end model according to the grouping strategy and the training batch size; and synchronize the model parameters of each group at intervals of N steps according to the synchronization strategy to obtain the updated end-to-end model.

[0241] In one possible implementation, the aforementioned training module 1102 is further configured to: in one of the iterations, use a loss function to calculate the difference between the output of the end-to-end model and the corresponding true value, update the end-to-end model according to the difference, and obtain the updated end-to-end model, wherein the loss function includes a loss term for each task, and the weight parameter corresponding to the loss term for each task is determined according to the user input data.

[0242] Figure 12 shows a schematic diagram of the hardware structure of a model training device 120 provided in an embodiment of this application. This model training device 120 can be used to implement the steps of the aforementioned training phase method.

[0243] The model training device 120 shown in Figure 12 may include a processor 1201, a memory 1202, a communication interface 1203, and a bus 1204. The processor 1201, the memory 1202, and the communication interface 1203 can be connected to each other via the bus 1204.

[0244] The processor 1201 is the control center of the model training device 120. It can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adapted to the actual application scenario.

[0245] As an example, processor 1201 may include one or more CPUs, and may also include other processors, such as the CPU, NPU or GPU shown in Figure 12.

[0246] The memory 1202 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0247] In one possible implementation, the memory 1202 may exist independently of the processor 1201. The memory 1202 can be connected to the processor 1201 via a bus 1204 and is used to store data, instructions, or program code. When the processor 1201 calls and executes the instructions or program code stored in the memory 1202, it can implement the methods provided in the embodiments of this application, such as the method shown in the exercise stage.

[0248] In another possible implementation, the memory 1202 can also be integrated with the processor 1201.

[0249] The communication interface 1203 is used for the model training device 120 to connect with other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 1203 may include a receiving unit for receiving data and a transmitting unit for sending data.

[0250] Bus 1204 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 12, but this does not indicate that there is only one bus or one type of bus.

[0251] It should be noted that the structure shown in Figure 12 does not constitute a limitation on the model training device 120. In addition to the components shown in Figure 12, the model training device 120 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0252] Figure 13 shows a schematic diagram of the hardware structure of a computing device 130 provided in an embodiment of this application. This computing device 130 can be used to implement the steps of the methods shown in Figures 3 to 9.

[0253] The computing device 130 shown in Figure 13 may include a processor 1301, a memory 1302, a communication interface 1303, and a bus 1304. The processor 1301, the memory 1302, and the communication interface 1303 can be connected to each other via the bus 1304.

[0254] The processor 1301 is the control center of the computing device 130. It can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adapted to the actual application scenario.

[0255] As an example, processor 1301 may include one or more CPUs, and may also include other processors, such as the CPU, NPU or GPU shown in Figure 13.

[0256] The memory 1302 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0257] In one possible implementation, the memory 1302 may exist independently of the processor 1301. The memory 1302 can be connected to the processor 1301 via a bus 1304 and is used to store data, instructions, or program code. When the processor 1301 calls and executes the instructions or program code stored in the memory 1302, it can implement the methods provided in the embodiments of this application, such as the methods shown in Figures 3 to 9.

[0258] In another possible implementation, the memory 1302 can also be integrated with the processor 1301.

[0259] The communication interface 1303 is used for connecting the computing device 130 to other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 1303 may include a receiving unit for receiving data and a transmitting unit for sending data.

[0260] Bus 1304 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 13, but this does not indicate that there is only one bus or one type of bus.

[0261] It should be noted that the structure shown in FIG13 does not constitute a limitation on the computing device 130. In addition to the components shown in FIG13, the computing device 130 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0262] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0263] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0264] This application also provides a computer-readable storage medium storing a program for training a model or performing inference tasks, which, when run on a computer, causes the computer to perform all or part of the steps in the methods described in the embodiments shown in Figures 2 to 6 above.

[0265] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the aforementioned processor or processor functions, and one or more interfaces. When the digital processing chip integrates a memory, it can perform the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the method steps of any one or more of the foregoing embodiments based on the program code stored in the external memory.

[0266] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0267] The target sensing device or target sensing device provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the server to execute the method described in the embodiments shown in Figures 3-5 above. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, such as random access memory (RAM).

[0268] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0269] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0270] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0271] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0272] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0273] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps in this application does not imply that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved. The division of modules in this application is a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed may be through some ports, and the indirect coupling or communication connection between modules may be electrical or other similar forms, which are not limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules can be selected to achieve the purpose of the solution in this application according to actual needs.

Claims

1. A data processing method, characterized in that, Applied to vehicles, including: Acquire map data and sensor data, wherein the sensor data includes data collected by multiple sensors; Feature extraction is performed on the map data and the sensor data to obtain perception features; A perception result is obtained based on the perception features, and the perception result includes information about at least one instance in the environment of the vehicle; The vehicle's driving path is determined based on the perception results.

2. The method according to claim 1, characterized in that, The perception results include at least one of the following: information on road structure in the vehicle's environment, information on moving targets, sparse features, or dense features.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the perception result based on the perception features includes: The perceived features are processed through multiple modules to perform various perception tasks and output the perception results. These multiple modules include a whitelist detection module, an obstacle detection module, a road information detection module, a forward perspective (PV) detection module, or a traffic light detection module. The whitelist detection module is used to detect information of instances of a preset category. The obstacle detection module is used to detect information of the drivable area of ​​the vehicle. The road information detection module is used to detect road information. The PV detection module is used to output information of instances from the PV perspective. The traffic light detection module is used to output information of traffic lights.

4. The method according to any one of claims 1-3, characterized in that, Determining the vehicle's driving path based on the perception results includes: Obtain navigation field information, which includes recommended scores for vehicles traveling in different lanes according to the navigation path; The driving path is determined based on the navigation field information and the perception results.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: The vehicle control signal is determined based on the driving path, and the vehicle control signal is used to control the driving of the vehicle.

6. The method according to claim 5, characterized in that, Determining the vehicle control signal based on the driving path includes: The driving decision of the vehicle is determined by solving the driving path using a multi-objective optimization algorithm. The vehicle control signal is generated based on the driving decision.

7. The method according to claim 6, characterized in that, The step of determining the vehicle's driving decision based on the driving path using a multi-objective optimization algorithm includes: At least one driving decision for the vehicle is determined by using a multimodal tree based on the driving path and the driving paths of other vehicles. The driving decision of the vehicle is determined from the at least one driving decision by solving the multi-objective optimization algorithm.

8. A model training method, characterized in that, include: Obtain a training set, which includes samples of various types; The training set is input into the end-to-end model, and the end-to-end model is iteratively updated based on the output of the end-to-end model to obtain the updated end-to-end model. The end-to-end model is used to perform multiple tasks. The end-to-end model includes a feature extraction network, a perception network, and a regulation network. The feature extraction network is used to extract features from various types of input samples. The perception network is used to output perception results based on the features output by the feature extraction network. The regulation network is used to output vehicle driving decisions based on the perception results output by the perception network.

9. The method according to claim 8, characterized in that, Before iteratively updating the end-to-end model based on its output, the method further includes: Configure a corresponding training batch size for each of the multiple tasks, and the training batch size for each task is related to the duration of each task's step. The step of inputting the training set into the end-to-end model and iteratively updating the end-to-end model based on its output to obtain the updated end-to-end model includes: The training set is input into the end-to-end model according to the training batch size, and the end-to-end model is iteratively updated according to the output of the end-to-end model to obtain the updated end-to-end model.

10. The method according to claim 9, characterized in that, Before iteratively updating the end-to-end model based on its output, the method further includes: Determine a grouping strategy and a synchronization strategy. The grouping strategy is used to divide the parameters of the end-to-end model into at least one group. The synchronization strategy is used to synchronize the model parameters of each group at intervals of N steps when updating the end-to-end model, where N is a positive integer. The step of inputting the training set into the end-to-end model according to the training batch size, and iteratively updating the end-to-end model according to the output of the end-to-end model to obtain the updated end-to-end model includes: The training set is input into the end-to-end model according to the grouping strategy and the training batch size. The model parameters of each group are synchronized at intervals of N steps according to the synchronization strategy to obtain the updated end-to-end model.

11. The method according to any one of claims 8-10, characterized in that, The step of iteratively updating the end-to-end model based on the output of the end-to-end model to obtain the updated end-to-end model further includes: In one iteration, the difference between the output of the end-to-end model and the corresponding ground truth is calculated using a loss function. The end-to-end model is then updated based on the difference to obtain an updated end-to-end model. The loss function includes a loss term for each task, and the weight parameters corresponding to the loss term for each task are determined based on user input data.

12. An end-to-end intelligent driving system, characterized in that, Applied to vehicles, including: The input module is used to acquire map data and sensor data, wherein the sensor data includes data collected by multiple sensors; The feature extraction module is used to extract features from the map data and the sensor data to obtain perception features; The perception module is used to obtain a perception result based on the perception features, wherein the perception result includes information about at least one instance in the environment of the vehicle. The planning and control module is used to determine the driving path of the vehicle based on the perception results.

13. The system according to claim 12, characterized in that, The perception results include at least one of the following: information on road structure in the vehicle's environment, information on moving targets, sparse features, or dense features.

14. The system according to claim 12 or 13, characterized in that, The sensing module is specifically used for: The perceived features are processed through multiple modules to perform various perception tasks and output the perception results. These multiple modules include a whitelist detection module, an obstacle detection module, a road information detection module, a forward perspective (PV) detection module, or a traffic light detection module. The whitelist detection module is used to detect information of instances of a preset category. The obstacle detection module is used to detect information of the drivable area of ​​the vehicle. The road information detection module is used to detect road information. The PV detection module is used to output information of instances from the PV perspective. The traffic light detection module is used to output information of traffic lights.

15. The system according to any one of claims 12-14, characterized in that, The control module is specifically used for: Obtain navigation field information, which includes recommended scores for vehicles traveling in different lanes according to the navigation path; The driving path is determined based on the navigation field information and the perception results.

16. The system according to any one of claims 12-15, characterized in that, The system also includes: A control module is used to determine a vehicle control signal based on the driving path, and the vehicle control signal is used to control the driving of the vehicle.

17. The system according to claim 16, characterized in that, The system also includes: A multi-objective optimization module is used to solve the driving path using a multi-objective optimization algorithm to determine the driving decision of the vehicle. The control module is also used to generate the vehicle control signal based on the driving decision.

18. The system according to claim 16 or 17, characterized in that, The system also includes: A multimodal tree module is used to determine at least one driving decision for the vehicle based on the driving path and the driving paths of other vehicles using a multimodal tree. The multi-objective optimization module is specifically used to solve the problem according to the multi-objective optimization algorithm and determine the driving decision of the vehicle from the at least one driving decision.

19. A model training device, characterized in that, include: The acquisition module is used to acquire a training set, which includes samples of various types. The training module is used to input the training set into the end-to-end model, and iteratively update the end-to-end model according to the output of the end-to-end model to obtain the updated end-to-end model. The end-to-end model is used to perform multiple tasks. The end-to-end model includes a feature extraction network, a perception network, and a regulation network. The feature extraction network is used to extract features from multiple types of input samples. The perception network is used to output perception results based on the features output by the feature extraction network. The regulation network is used to output vehicle driving decisions based on the perception results output by the perception network.

20. The apparatus according to claim 19, characterized in that, The training module is further configured to: before iteratively updating the end-to-end model based on the output of the end-to-end model, Configure a corresponding training batch size for each of the multiple tasks, and the training batch size for each task is related to the duration of each task's step. The training set is input into the end-to-end model according to the training batch size, and the end-to-end model is iteratively updated according to the output of the end-to-end model to obtain the updated end-to-end model.

21. The apparatus according to claim 20, characterized in that, The training module is further configured to: before iteratively updating the end-to-end model based on the output of the end-to-end model, Determine a grouping strategy and a synchronization strategy. The grouping strategy is used to divide the parameters of the end-to-end model into at least one group. The synchronization strategy is used to synchronize the model parameters of each group at intervals of N steps when updating the end-to-end model, where N is a positive integer. The training set is input into the end-to-end model according to the grouping strategy and the training batch size. The model parameters of each group are synchronized at intervals of N steps according to the synchronization strategy to obtain the updated end-to-end model.

22. The apparatus according to any one of claims 19-21, characterized in that, The training module is further configured to: in one iteration, calculate the difference between the output of the end-to-end model and the corresponding ground truth using a loss function, update the end-to-end model based on the difference, and obtain an updated end-to-end model, wherein the loss function includes a loss term for each task, and the weight parameter corresponding to the loss term for each task is determined based on user input data.

23. A computing device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the device performs the method as described in any one of claims 1-7.

24. A model training device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the device performs the method as described in any one of claims 8-11.

25. A vehicle, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the device performs the method as described in any one of claims 1-7.

26. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 7 or 8 to 11.

27. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7 or 8 to 11.

Images