Decision-making method and related apparatus

Abstract

A decision-making method and a related apparatus, which are applied to the technical field of computers. The decision-making method comprises: using a first neural network to perform feature extraction on attribute data of an ego vehicle and surrounding environment information of the ego vehicle, and using an attention network to observe, on the basis of features extracted by the first neural network and features of the ego vehicle, which quasi-static obstacles around the ego vehicle should be attended to, that is, which quasi-static obstacles may affect the driving of the ego vehicle, in order to obtain a decision-making result for the ego vehicle. In this way, a trajectory prediction process of a prediction module can be omitted. A first neural network and an attention network are used, such that an attention result can be directly output for quasi-static obstacles so as to indicate a decision-making result for an ego vehicle, thereby improving the decision-making response speed and improving the accuracy of decision-making results. Moreover, the decision-making method is not limited by the length of a prediction horizon, and can reduce the probability of decision-making errors, reduce possible collision risks, and thus improve the safety of an intelligent driving system.

Description

A decision-making method and related apparatus Technical Field

[0001] This application relates to the field of computer technology, and in particular to a decision-making method and related apparatus. Background Technology

[0002] Currently, intelligent driving systems are gradually becoming more common in the automotive market. These systems rely on the collaborative efforts of artificial intelligence, computer vision, radar, monitoring devices, and positioning systems to enable computers to automatically and safely operate motor vehicles without any human intervention.

[0003] In the field of intelligent driving, modules typically include a perception module, a prediction module, and a control module. The prediction module receives environmental and obstacle information from the perception module, predicts the trajectory of obstacles, and then sends this prediction information to the control module. The control module receives the prediction information from the prediction module, combines it with the vehicle's path planning and the prediction information of other obstacles, determines the spatial relationship between the vehicle and other obstacles at each future moment, and makes corresponding strategic decisions, ultimately providing the vehicle's lateral and longitudinal control behavior. If, within the field of vision, an obstacle is stationary or moving at low speed, and in the near future encroaches on the vehicle's path, this prediction method may lead to incorrect decisions due to an overly short prediction line or the vehicle being too close to the obstacle, resulting in the vehicle overtaking and posing a significant collision risk, thus lowering the safety of the intelligent driving system. Summary of the Invention

[0004] This application provides a decision-making method and related apparatus that can omit the trajectory prediction process of the prediction module. Utilizing a first neural network and an attention network, for near-static obstacles (e.g., obstacles that are stationary or moving at low speeds), it can directly output attention results to indicate the vehicle's decision-making outcome, thereby improving the response speed and accuracy of the decision. Furthermore, this application is not limited by the length of the prediction line, reducing the possibility of decision errors, lowering potential collision risks, and improving the safety of the intelligent driving system.

[0005] Firstly, this application provides a decision-making method for outputting vehicle control behavior. This method can be executed by the vehicle. Optionally, when the method is executed by a processing device in the vehicle or a component (such as a chip or module) within the processing device, it can be considered as the method being executed by the vehicle. Exemplarily, the processing device includes a central processing unit (CPU), a microcontroller unit (MCU), etc. For example, the processing device is a domain controller (DC), an electronic control unit (ECU), a vehicle integration unit (VIU), etc., where the domain controller includes an intelligent driving domain controller (such as a mobile data center), etc. For ease of description, the following description uses the processing device as the execution subject of this method; in actual implementation, the execution subject of this method can have other names.

[0006] The decision-making method includes: a processing device using a first neural network to extract features based on the attribute data of a first vehicle and the surrounding environment information of the first vehicle, obtaining features of the first vehicle and features of at least one obstacle. The surrounding environment information of the first vehicle includes current attribute data and historical attribute data of at least one obstacle. The historical attribute data of the at least one obstacle is the attribute data of the at least one obstacle within a first time period prior to the current moment. Each of the at least one obstacle is a near-static obstacle, meaning it is stationary or its speed is less than a first threshold. Further, the processing device using an attention network, based on the features of the first vehicle and the features of the at least one obstacle, obtains an attention result for the at least one obstacle. This attention result is used to indicate the decision result for the first vehicle.

[0007] Optionally, when the speed of a near-static obstacle is less than a first threshold, the near-static obstacle can be said to be in a low-speed state.

[0008] Optionally, the decision outcome may include one of the following: yield, do not yield, brake, etc.

[0009] Based on the above scheme, the processing device uses a first neural network to extract features from the vehicle's attribute data and surrounding environmental information. Then, using an attention network, it uses these features to observe which near-static obstacles around the vehicle require attention—that is, which near-static obstacles might affect the vehicle's driving—and arrives at the vehicle's decision-making result. Thus, this application can omit the trajectory prediction process of the prediction module. Utilizing the first neural network and the attention network, it can directly output attention results to indicate the vehicle's decision-making result for near-static obstacles, improving the response speed and accuracy of the decision. Furthermore, this application is not limited by the length of the prediction line, reducing the possibility of decision errors, lowering the potential collision risk, and improving the safety of the intelligent driving system.

[0010] In one possible implementation of the first aspect, the first neural network and the attention network are trained based on training data. The training data includes vehicle attribute data for a second time period, information about the vehicle's surrounding environment, and ground truth results. The vehicle's surrounding environment information includes attribute data of surrounding obstacles for the second time period, and the ground truth results indicate whether the surrounding obstacles affect the vehicle's movement during the second time period; the surrounding obstacles are near-static obstacles. The first neural network is used to extract features based on the vehicle's attribute data for the second time period and the vehicle's surrounding environment information to obtain features of the vehicle and the surrounding obstacles. The attention network is used to obtain attention results for the surrounding obstacles based on the vehicle's features and the features of the surrounding obstacles. The ground truth results are used to adjust the model parameters of the first neural network and the attention network in conjunction with the attention results for the surrounding obstacles.

[0011] Optionally, the vehicle's surrounding environment information also includes static environment attribute data, which includes road lane lines and curbs.

[0012] The above implementation method can adjust the model parameters of the first neural network and the attention network based on the attention results of surrounding obstacles and the ground truth results in the training data. This can reduce the error between the output data of the first neural network and the attention network and the actual output data, thereby improving the accuracy of the first neural network and the attention network, and improving the accuracy of the decision results indicated by the attention results. Moreover, the training process involves judging whether there is a game relationship between the surrounding obstacles and the vehicle, that is, judging whether the surrounding obstacles affect the vehicle's driving, which is beneficial to improving the accuracy of the first neural network and the attention network, and improving the accuracy of the decision results indicated by the attention results.

[0013] In another possible implementation of the first aspect, the attribute data of the first vehicle includes the first vehicle's orientation, speed, position, length, and width. The attribute data of the first vehicle also includes at least one of the first vehicle's height, acceleration, angular velocity, angular acceleration, headlight status, wheel status, door status, and dynamic / static status, wherein the dynamic / static status of the first vehicle is used to indicate whether the first vehicle is in a dynamic or static state.

[0014] In the above embodiments, the attribute data of the first vehicle may include multiple data, which helps the first neural network and the attention network to better capture the complex relationship between the input data and the actual output data, and further helps to improve the accuracy of the first neural network and the attention network, and improve the accuracy of the decision results indicated by the attention results.

[0015] In another possible implementation of the first aspect, the current attribute data of at least one obstacle includes the orientation, velocity, position, length, and width of at least one obstacle at the current moment. The current attribute data of at least one obstacle also includes at least one of the height, acceleration, angular velocity, angular acceleration, and dynamic / static state of at least one obstacle at the current moment. The historical attribute data of at least one obstacle includes the orientation, velocity, position, length, and width of at least one obstacle during a first time period prior to the current moment. The historical attributes of at least one obstacle also include at least one of the height, acceleration, angular velocity, angular acceleration, and dynamic / static state of at least one obstacle during the first time period prior to the current moment, the dynamic / static state of at least one obstacle indicating whether at least one obstacle is dynamic or static.

[0016] In the above embodiments, the current attribute data of at least one obstacle and the historical attribute data of at least one obstacle may include multiple data, which helps the first neural network and the attention network to better capture the complex relationship between the input data and the actual output data, and further helps to improve the accuracy of the first neural network and the attention network, and improve the accuracy of the decision results indicated by the attention results.

[0017] In another possible implementation of the first aspect, the environmental information surrounding the first vehicle further includes attribute data of the static environment, which includes lane lines and curbs of the road.

[0018] In the above embodiments, the surrounding environment information of the first vehicle also includes road-related information, which can enhance the understanding of road topology during the processing of the first neural network and the attention network, thereby improving the accuracy of the first neural network and the attention network, and improving the accuracy of the decision results indicated by the attention results.

[0019] In another possible implementation of the first aspect, the attention network includes a first attention network and a second attention network. Using the attention network, based on the features of the first vehicle and the features of at least one obstacle, an attention result for at least one obstacle is obtained, including: using the first attention network, obtaining intermediate features of the first obstacle based on the current attribute data of the first obstacle and the features of the first obstacle; and using the second attention network, obtaining the attention result for at least one obstacle based on the intermediate features of at least one obstacle and the features of the first vehicle. Wherein, the first obstacle is any one of the at least one obstacles.

[0020] In the above embodiments, the attention results of obstacles obtained by using the attention network can be processed by the attention network twice, which can better utilize the characteristics of the vehicle to observe which near-static obstacles around the vehicle need attention, that is, which near-static obstacles may affect the vehicle's driving, which helps to improve the accuracy of the attention network and the accuracy of the decision results indicated by the attention results.

[0021] In another possible implementation of the first aspect, an attention network is used to obtain an attention result for at least one obstacle based on the features of a first vehicle and the features of at least one obstacle. This includes using the attention network to compress the features of the first vehicle through a compression operation to obtain a feature vector; using the attention network to process the feature vector through an activation function to obtain weight parameters for each set of hidden layer information; and using the attention network based on the weight parameters for each set of hidden layer information to obtain the attention result for at least one obstacle. The feature vector includes multiple sets of hidden layer information, each set of hidden layer information characterizing the features of one obstacle among the at least one obstacle, and the weight parameters indicating the importance of one obstacle among the at least one obstacle.

[0022] The above implementation illustrates a possible process for an attention network to obtain attention results, namely the processing of a squeeze-and-excitation layer (SELayer). The SELayer can dynamically adjust the importance of each obstacle among at least one obstacle through weight parameters, thereby enabling the attention network to focus more on and utilize important features. This improves the attention network's focus on important features, enhances its expressive power, allows it to better adapt to different input data, and improves its performance and generalization ability.

[0023] In another possible implementation of the first aspect, the attribute data of the first vehicle and the information about the surrounding environment of the first vehicle include sensor data.

[0024] In some scenarios, the input data for the first neural network can also be sensor data, such as image data collected by vision sensors, or point cloud data collected by radar sensors.

[0025] In yet another possible implementation of the first aspect, the first neural network includes a transformer network or a convolutional neural network (CNN).

[0026] In another possible implementation of the first aspect, the network of concern includes at least one of the following: a compressed excitation network SELayer, a multilayer perceptron (MLP), and a motion-aware layer normalization network (MLN).

[0027] In another possible implementation of the first aspect, when the second obstacle affects the driving of the first vehicle, the attention result of the second obstacle is used to instruct the replanning of the driving path of the first vehicle so that the first vehicle avoids the second obstacle, which is at least one obstacle.

[0028] In the above embodiments, when the second obstacle affects the driving of the first vehicle, the output attention result is used to instruct the replanning of the first vehicle's driving path; that is, the decision result of the first vehicle is to replan its driving path. Thus, for the near-static second obstacle, the above embodiments directly output the attention result to instruct the vehicle's decision result, which can improve the response speed of the decision, improve the accuracy of the decision result, reduce the potential collision risk, and improve the safety of the intelligent driving system.

[0029] In yet another possible implementation of the first aspect, the method further includes generating a first interface for indicating that a second obstacle affects the movement of the first vehicle, the second obstacle being at least one obstacle.

[0030] In the above embodiments, a visual interface indicates how the second obstacle affects the driving of the first vehicle, allowing the user to intuitively perceive the relationship between the first vehicle's decision and the second obstacle. This improves the interpretability of the intelligent driving system and enhances the user experience. For example, if the first vehicle's decision is to brake, and the first interface indicates how the second obstacle affects the first vehicle's driving, the user can promptly perceive that the first vehicle's braking behavior is related to the second obstacle's influence on its driving—that is, the second obstacle causes the first vehicle's braking behavior—thus improving the user experience.

[0031] Secondly, this application provides a processing apparatus, comprising a processing unit configured to perform feature extraction using a first neural network based on attribute data of a first vehicle and surrounding environmental information of the first vehicle, to obtain features of the first vehicle and features of at least one obstacle. The surrounding environmental information of the first vehicle includes current attribute data and historical attribute data of at least one obstacle. The historical attribute data of the at least one obstacle is attribute data of the at least one obstacle within a first time period prior to the current moment. Each of the at least one obstacle is a near-static obstacle, which is either stationary or has a speed less than a first threshold. Further, the processing apparatus is also configured to use an attention network to obtain an attention result for at least one obstacle based on the features of the first vehicle and the features of the at least one obstacle. The attention result is used to indicate the decision result of the first vehicle.

[0032] In one possible implementation of the second aspect, the first neural network and the attention network are trained based on training data. The training data includes vehicle attribute data for a second time period, information about the vehicle's surrounding environment, and ground truth results. The vehicle's surrounding environment information includes attribute data of surrounding obstacles for the second time period and attribute data of the static environment. The ground truth results indicate whether surrounding obstacles affect the vehicle's movement during the second time period; these surrounding obstacles are near-static. The first neural network is used to extract features based on the vehicle's attribute data for the second time period and the vehicle's surrounding environment information, obtaining features of the vehicle and surrounding obstacles. The attention network is used to obtain attention results for the surrounding obstacles based on the vehicle's features and the features of the surrounding obstacles. The ground truth results are used to adjust the model parameters of the first neural network and the attention network in conjunction with the attention results for the surrounding obstacles.

[0033] In another possible implementation of the second aspect, the attribute data of the first vehicle includes the first vehicle's orientation, speed, position, length, and width. The attribute data of the first vehicle also includes at least one of the first vehicle's height, acceleration, angular velocity, angular acceleration, headlight status, wheel status, door status, and dynamic / static status, wherein the dynamic / static status of the first vehicle is used to indicate whether the first vehicle is in a dynamic or static state.

[0034] In another possible implementation of the second aspect, the current attribute data of at least one obstacle includes the orientation, velocity, position, length, and width of at least one obstacle at the current moment. The current attribute data of at least one obstacle also includes at least one of the height, acceleration, angular velocity, angular acceleration, and dynamic / static state of at least one obstacle at the current moment. The historical attribute data of at least one obstacle includes the orientation, velocity, position, length, and width of at least one obstacle during a first time period prior to the current moment. The historical attributes of at least one obstacle also include at least one of the height, acceleration, angular velocity, angular acceleration, and dynamic / static state of at least one obstacle during the first time period prior to the current moment, wherein the dynamic / static state of at least one obstacle is used to indicate whether at least one obstacle is dynamic or static.

[0035] In another possible implementation of the second aspect, the environmental information surrounding the first vehicle also includes attribute data of the static environment, which includes lane lines and curbs of the road.

[0036] In another possible implementation of the second aspect, the attention network includes a first attention network and a second attention network. The processing unit is further configured to utilize the first attention network to obtain intermediate features of the first obstacle based on the current attribute data and features of the first obstacle. The processing unit is further configured to utilize the second attention network to obtain attention results for at least one obstacle based on the intermediate features of at least one obstacle and features of the first vehicle. Herein, the first obstacle is any one of the at least one obstacles.

[0037] In another possible implementation of the second aspect, the processing unit is further configured to compress the features containing the first vehicle through a compression operation to obtain a feature vector. The processing unit is further configured to process the feature vector using an activation function to obtain weight parameters for each set of hidden layer information, and based on the weight parameters for each set of hidden layer information, obtain a focus result for at least one obstacle. Here, the feature vector includes multiple sets of hidden layer information, each set of hidden layer information is used to characterize the features of one obstacle among multiple obstacles, and the weight parameters are used to indicate the importance of each obstacle among the at least one obstacle.

[0038] In another possible implementation of the second aspect, the attribute data of the first vehicle and the information about the surrounding environment of the first vehicle include sensor data.

[0039] In another possible implementation of the second aspect, the first neural network includes a transformer network or a CNN network.

[0040] In another possible implementation of the second aspect, the network of concern includes at least one of the compression excitation networks SELayer, MLP, and MLN.

[0041] In another possible implementation of the second aspect, when the second obstacle affects the driving of the first vehicle, the attention result of the second obstacle is used to instruct the replanning of the driving path of the first vehicle so that the first vehicle avoids the second obstacle, which is at least one obstacle.

[0042] In another possible implementation of the second aspect, the processing unit is further configured to generate a first interface for indicating that a second obstacle affects the driving of the first vehicle, the second obstacle being at least one obstacle.

[0043] Thirdly, embodiments of this application provide a computing device, the computing device including a processor and a memory, the memory storing a program, the processor executing the program stored in the memory to enable the computing device to implement the method described in any of the first aspects above.

[0044] Fourthly, this application provides a vehicle that includes the processing device described in the second aspect or the computing device described in the third aspect, the vehicle being used to implement the method described in any of the first aspects.

[0045] Fifthly, embodiments of this application provide a computer-readable storage medium for storing a computer program, the computer program including instructions for performing the method described in any of the first aspects above.

[0046] In a sixth aspect, this application provides a computer program product including computer instructions that, when executed by a control device, computing device, or processor, cause the method described in any of the first aspects to be implemented.

[0047] In a seventh aspect, this application provides a chip including a processor for executing computer execution instructions to cause a device on which the chip is mounted to perform the method described in any of the first aspects above.

[0048] The solutions provided in the second to seventh aspects above are used to implement or cooperate with the methods provided in the first aspect above, and therefore can achieve the same or corresponding beneficial effects as the first aspect, which will not be elaborated here. Attached Figure Description

[0049] The accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0050] Figure 1 is a schematic diagram of the architecture of a decision system provided in an embodiment of this application;

[0051] Figure 2 is a schematic diagram of a decision-making scenario provided in an embodiment of this application;

[0052] Figure 3 is a schematic diagram of an in-vehicle display device provided in an embodiment of this application;

[0053] Figure 4 is a flowchart illustrating a decision-making method provided in an embodiment of this application;

[0054] Figure 5 is a schematic diagram of a first interface provided in an embodiment of this application;

[0055] Figure 6 is a schematic diagram of a processing device provided in an embodiment of this application;

[0056] Figure 7 is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0057] For ease of understanding, the following examples illustrate some concepts related to the embodiments of this application for reference. As follows:

[0058] 1. Neural Networks

[0059] In the field of computer science, neural networks typically refer to artificial neural networks (ANNs), often simply referred to as neural networks or neural network-like systems in engineering and academia. A neural network is a mathematical model that mimics the behavioral characteristics of animal neural networks, performing distributed parallel information processing. From an information processing perspective, neural networks abstract the human brain's neuronal network, establishing a simple model and assembling different networks with different connection methods. This type of network, depending on its complexity, adjusts the relationships between a large number of internal nodes (or neurons) to achieve the purpose of processing information and possesses self-learning and adaptive capabilities. Each node represents a specific output function, called the activation function. The connection between any two nodes represents a weighted value for the signal passing through that connection, called a weight, which is equivalent to the memory of the artificial neural network. The network's output varies depending on the network's connection methods, weight values, and activation functions.

[0060] 2. Transformer network

[0061] The Transformer network, a deep learning model architecture, is essentially an encoder-decoder architecture composed of encoding and decoding components. It uses a self-attention mechanism to capture dependencies between different positions in the input sequence, regardless of the sequence order. This allows the model to process information from different positions simultaneously when processing sequential data, significantly improving computational efficiency and parallel processing capabilities, enabling it to learn more data features in a shorter time. Furthermore, the Transformer network can capture dependencies between all positions in the input sequence, directly calculating the relationship between two elements regardless of their distance within the sequence, thus providing a better understanding of the global context.

[0062] 3. Convolutional Neural Network (CNN)

[0063] A CNN (Convolutional Neural Network) is a feedforward neural network whose artificial neurons respond to a subset of surrounding units within their coverage area, making it excellent for processing large images. A CNN consists of one or more convolutional layers and a fully connected layer at the top (corresponding to a classic neural network), as well as associated weights and pooling layers. This structure allows CNNs to utilize the two-dimensional structure of the input data. Compared to other deep learning architectures, CNNs deliver better results in image and speech recognition.

[0064] 4. Squeeze-and-excitation Layer (SELayer)

[0065] SELayer is a neural network architecture used in deep learning. SELayer includes squeezing, excitation, and reweighting operations. SELayer can automatically learn the importance weights of each channel, allowing the model to dynamically adjust the feature responses of different channels based on the characteristics of the input data. This helps the model focus more on the features of channels with higher information content and suppress less important channel features, thereby improving the model's representational power and performance.

[0066] 5. Multilayer perceptron (MLP)

[0067] MLP is an artificial neural network model. By introducing activation functions, MLP can perform non-linear mappings, fit complex functions, handle high-dimensional data, and is not dependent on specific data types. It is suitable for various tasks such as classification and regression. Through the backpropagation algorithm, MLP can quickly adjust network parameters and gradually improve model performance.

[0068] 6. Motion-aware layer normalization (MLN)

[0069] MLN (Motion Normalization) is a normalization technique used to process motion information of objects. It primarily normalizes features related to object motion by adjusting the normalization process based on the object's motion information. Specifically, MLN can weight or adjust features according to the object's motion state, such as position, velocity, and direction, so that the normalized features better reflect the actual motion of the object.

[0070] The above descriptions of technical terms may be used in the embodiments below.

[0071] Currently, in the field of intelligent driving, modules typically include a perception module, a prediction module, and a control module. The perception module sends environmental and obstacle information to the prediction module. After receiving the environmental and obstacle information from the perception module, the prediction module predicts the trajectory of the obstacles and then sends the prediction information to the control module. This prediction information includes the predicted trajectory of the obstacles. After receiving the prediction information from the prediction module, the control module combines the vehicle's path planning with the prediction information of other obstacles to determine the spatial relationship between the vehicle and other obstacles at each future moment, as well as the corresponding game-theoretic decisions, ultimately providing the vehicle's lateral and longitudinal control behaviors. Therefore, the vehicle's future driving path and the predicted trajectories of other obstacles are completely coupled. Any tiny change in other obstacles that leads to a change in the predicted trajectory can significantly affect the vehicle's future driving path in the short term. If, within the field of vision, an obstacle is stationary or moving at low speed, and then encroaches on the vehicle's driving path in the near future (e.g., the obstacle suddenly starts moving), this prediction method may lead to incorrect decisions due to an excessively short prediction line or the vehicle being too close to the obstacle, resulting in the vehicle overtaking and posing a significant collision risk, thus lowering the safety of the intelligent driving system.

[0072] In view of this, this application provides a decision-making method and related apparatus that can omit the trajectory prediction process of the prediction module. Utilizing a first neural network and an attention network, for near-static obstacles (e.g., obstacles that are stationary or moving at low speeds), it can directly output attention results to indicate the vehicle's decision-making outcome, thereby improving the response speed and accuracy of the decision. Furthermore, this application is not limited by the length of the prediction line, reducing the possibility of decision errors, lowering potential collision risks, and improving the safety of the intelligent driving system.

[0073] The following section provides an exemplary description of the systems and scenarios in which this application may be applied. It should be noted that the system architecture and business scenarios described in this application are for the purpose of more clearly illustrating the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. Those skilled in the art will understand that, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in this application are equally applicable to similar technical problems.

[0074] Please refer to Figure 1. Figure 1 is a schematic diagram of the architecture of a decision system provided in an embodiment of this application. The decision system includes a vehicle 10 and at least one obstacle. Figure 1 uses two obstacles as an example for illustration. For ease of description, they are referred to as obstacle 20 and obstacle 30, respectively.

[0075] Vehicle 10 includes a processing unit 101. Of course, vehicle 10 also includes devices supporting vehicle movement and control, such as a driving system and a braking system, which are not described in detail here. It should be understood that the types of vehicles shown here are merely examples; in specific implementations, vehicle 10 can be different types of vehicles such as automobiles, trucks, trains, buses, vans, and electric vehicles.

[0076] At least one obstacle is located around vehicle 10, for example, within the field of vision of vehicle 10, or within the detectable (or identifiable) range of vehicle 10. Each of the at least one obstacle is a near-static obstacle, meaning it is stationary or its speed is less than a first threshold. Optionally, when the speed of a near-static obstacle is less than the first threshold, it may be described as being in a low-speed state. For example, obstacles 20 and 30 are near-static obstacles. It should be understood that the number, shape, size, and type of obstacles shown herein are merely examples; in a specific implementation, obstacles may also be other vehicles besides the vehicle itself, robots, pedestrians, animals, etc.

[0077] The following is an exemplary description of the devices included in the vehicle 10:

[0078] The processing device 101 has data processing capabilities. Exemplarily, the processing device 101 can utilize a first neural network to perform feature extraction based on the attribute data of vehicle 10 and the surrounding environment information of vehicle 10, obtaining the features of vehicle 10 and the features of at least one obstacle. The surrounding environment information of vehicle 10 includes the current attribute data of at least one obstacle and the historical attribute data of at least one obstacle, where the historical attribute data of at least one obstacle is the attribute data of at least one obstacle within a first time period prior to the current moment. For example, referring to Figure 2, if obstacle 40 needs to turn left onto the road where vehicle 10 is located, and vehicle 10 needs to go straight, obstacle 40 will affect the driving of vehicle 10. The processing device 101 can utilize the first neural network to perform feature extraction based on the attribute data of vehicle 10 and the surrounding environment information of vehicle 10, obtaining the features of vehicle 10 and the features of obstacle 40.

[0079] As another example, processing device 101 can utilize an attention network to obtain an attention result for at least one obstacle based on the features of vehicle 10 and the features of at least one obstacle. The attention result is used to instruct vehicle 10 on its decision. Optionally, the decision result may include one of yielding, not yielding, braking, etc. Referring to FIG2, processing device 101 can utilize an attention network to obtain an attention result for obstacle 40 based on the features of vehicle 10 and the features of obstacle 40. The attention result for obstacle 40 can be used to instruct vehicle 10 to brake so that vehicle 10 avoids obstacle 20. Here, obstacle 40 is at least one obstacle.

[0080] As one possible implementation, the processing device 101 can be a physical device, such as including one or more of the following modules: a central processing unit (CPU), a microprocessor unit (MPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), a coprocessor (assisting the CPU in completing corresponding processing and applications), a microcontroller unit (MCU), a mobile data center (MDC), and / or an electronic control unit (ECU), a cockpit domain controller (CDC), a vehicle integrated / integration unit (VIU), etc. Further, the processing device includes at least one processor integrated in the form of a system-on-chip (SOC), commonly referred to by those skilled in the art as an SOC. The SOC may include at least one processor, and when the SOC includes multiple processors, the types of processors may be different. In addition, a processor can also be a hardware circuit designed for artificial intelligence, which can be understood as a type of ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc.

[0081] Of course, the above description assumes that the processing device 101 is an in-vehicle device. In some solutions, the processing device 101 can be a physical device located outside the vehicle, such as a server, cloud, or host, which communicates and transmits information with the vehicle. As one possible implementation, the processing device 101 can be a software module, such as a virtual machine, software, program code, or container.

[0082] Optionally, the processing device 101 may also have data acquisition capabilities. For example, the processing device 101 may acquire attribute data of the first vehicle and information about the surrounding environment of the first vehicle. In one possible implementation, the processing device 101 is communicatively connected to other devices of the vehicle 10, and can receive attribute data of the first vehicle and information about the surrounding environment of the first vehicle sent by other devices according to the connection line between them.

[0083] In another possible implementation, vehicle 10 may further include sensor 102. Sensor 102 has information acquisition capabilities and is able to collect sensor data. The attribute data of the first vehicle and the surrounding environment information of the first vehicle may include sensor data. Exemplarily, processing device 101 is communicatively connected to sensor 102 and can receive sensor data sent by sensor 102 according to the connection line between them.

[0084] Optionally, the sensor data includes image data and / or point cloud data. For example, sensor 102 may include sensors such as vision sensors and / or radar sensors. Image data includes image data of the first vehicle or images of the environment surrounding the first vehicle. Point cloud data includes point cloud data of the first vehicle or point cloud data of targets surrounding the first vehicle.

[0085] Vision sensors include image sensors (often called cameras) used to acquire image data, including images and videos. An image sensor may contain multiple photosensitive units that sense light signals and form images. Examples of photosensitive units include complementary metal-oxide-semiconductor (CMOS), charge-coupled devices (CCD), and Live MOS. Image sensors are typically classified based on the type of photosensitive units they contain. For example, an image sensor with CMOS photosensitive units is called a CMOS image sensor (CIS). A CIS may include one or more of the following: a color sensor, a monosensor, etc. A color sensor may include a red-green-blue (RGB) sensor. Taking cameras as an example, vision sensors may include monocular cameras, binocular cameras, time-of-flight (TOF) cameras, cameras in driver monitoring systems (DMS), or cameras in cockpit monitoring systems (CMS).

[0086] Radar sensors are used to detect objects using electromagnetic waves and obtain point cloud data. Based on the different detection signals emitted, radar sensors can be classified into millimeter-wave radar, centimeter-wave radar, ultrasonic radar, lidar, and other categories.

[0087] Optionally, vehicle 10 may also include a display device 103. The display device 103 has interface display capabilities. The display device 103 includes one or more of the following: a physical screen (such as a central control screen), a projection system, a smart device, or a button panel. Projection systems include, for example, light field screens, head-up displays (HUDs), or other projection systems. As shown in Figure 3, a central control screen 1031 can be installed on the vehicle's central console. The central control screen 1031 is used to display the vehicle's driving route, display obstacles affecting the vehicle's driving, display configuration information of vehicle functions, or play audio and video, etc. In addition to the central control screen, other types of display devices 103 are also installed in the vehicle. As shown in Figure 3, passengers in the front passenger seat, rear passengers, etc., can interact with the vehicle through the display device 103. Of course, the display device 103 (including the central control screen 1031) here includes one or more of the following: a physical screen, a projection system, a smart device, or a button panel. Projection systems include, for example, light field screens, head-up displays, or other projection systems. This application does not strictly limit the number and location of the display devices 103 in the applicable scenario. Taking the application of this application in a vehicle as an example, the display devices 103 can be located in front of the passenger seat, on the headrest (or backrest) of the front seat, on the armrest of the seat, on the door, or on the top of the cabin.

[0088] For example, the display device 103 is connected to the processing device 101. The display device 103 can receive content sent (or output) by the processing device 101 through the connection line between the two and display the content through an interface. For example, the display device 103 receives a first interface sent (or output) by the processing device 101, the first interface being used to indicate that a second obstacle affects the driving of the first vehicle.

[0089] In one possible implementation, the processing device 101 utilizes a first neural network to extract features based on the attribute data of the vehicle 10 and the surrounding environment information of the vehicle 10, thereby obtaining features of the vehicle 10 and features of at least one obstacle. Further, the processing device 101 utilizes an attention network to obtain attention results for at least one obstacle based on the features of the first vehicle and the features of at least one obstacle.

[0090] In this embodiment, the processing device can utilize a first neural network to extract features from the vehicle's attribute data and surrounding environmental information. Based on the features extracted by the first neural network, an attention network is used to observe which near-static obstacles around the vehicle require attention—that is, which near-static obstacles might affect the vehicle's driving—and thus determine the vehicle's decision. In this way, this application can omit the trajectory prediction process of the prediction module. Using the first neural network and the attention network, it can directly output attention results to indicate the vehicle's decision for near-static obstacles, improving the response speed and accuracy of the decision. Furthermore, this application is not limited by the length of the prediction line, reducing the possibility of decision errors, lowering the potential collision risk, and improving the safety of the intelligent driving system.

[0091] The methods of the embodiments of this application will be described in detail below.

[0092] Please refer to Figure 4, which is a flowchart illustrating a decision-making method provided in an embodiment of this application. Optionally, this method can be applied to a vehicle, such as the vehicle 10 shown in Figure 1, and can optionally be executed by the processing device 101 in the vehicle 10.

[0093] The decision-making method shown in Figure 4 may include steps S401 and S402. Steps S401 and S402 are detailed below:

[0094] Step S401: The processing device uses a first neural network to extract features based on the attribute data of the first vehicle and the surrounding environment information of the first vehicle, to obtain the features of the first vehicle and the features of at least one obstacle.

[0095] The processing device is a device with data processing capabilities. For example, the processing device is the processing device 101 shown in FIG1 above.

[0096] The first neural network is an artificial neural network, such as a transformer network or a CNN.

[0097] The attribute data of the first vehicle includes its orientation, speed, position, length, and width. The attribute data also includes at least one of the following: height, acceleration, angular velocity, angular acceleration, headlight status, wheel status, door status, and dynamic / static status. For example, headlight status may include one of left turn signal, right turn signal, hazard lights, etc. Wheel status may include one of clockwise rotation angle and counterclockwise rotation angle. Door status includes whether the door is open or closed. The dynamic / static status of the first vehicle indicates whether the first vehicle is in a dynamic (i.e., moving) state or a static (i.e., stationary) state.

[0098] The attribute data of the first vehicle includes its current attribute data. Specifically, the attribute data of the first vehicle includes its current orientation, speed, position, length, and width. The attribute data of the first vehicle also includes at least one of the following: current height, acceleration, angular velocity, angular acceleration, headlight status, wheel status, door status, and dynamic / static status.

[0099] Optionally, the attribute data of the first vehicle may also include historical attribute data of the first vehicle. That is, the attribute data of the first vehicle includes the orientation, speed, position, length, and width of the first vehicle in a first time period before the current moment. The attribute data of the first vehicle also includes at least one of the following: height, acceleration, angular velocity, angular acceleration, headlight status, wheel status, door status, and dynamic / static status of the first vehicle in the first time period before the current moment.

[0100] The environmental information surrounding the first vehicle includes current attribute data and historical attribute data of at least one obstacle. Each of the at least one obstacle is a near-static obstacle, meaning it is either stationary or its speed is less than a first threshold. Optionally, when the speed of a near-static obstacle is less than the first threshold, it can be said to be in a low-speed state. The first threshold can be a pre-set speed threshold, such as 5 km / h, 6 km / h, etc.

[0101] For example, the current attribute data of at least one obstacle includes the orientation, velocity, position, length, and width of the at least one obstacle at the current moment. The current attribute data of at least one obstacle may also include at least one of the following: height, acceleration, angular velocity, angular acceleration, and dynamic / static state of the at least one obstacle at the current moment. Optionally, when the at least one obstacle is a vehicle, the current attribute data of the at least one obstacle may also include at least one of the following: headlight state, wheel state, and door state of the at least one obstacle at the current moment.

[0102] The historical attribute data of at least one obstacle refers to the attribute data of at least one obstacle within a first time period prior to the current moment. For example, the historical attribute data of at least one obstacle includes its orientation, velocity, position, length, and width within the first time period prior to the current moment. The historical attribute data of at least one obstacle may also include at least one of its height, acceleration, angular velocity, angular acceleration, and dynamic / static state within the first time period prior to the current moment, where the dynamic / static state indicates whether the at least one obstacle is dynamic or static. The first time period is a preset time range, such as 3 minutes, 2.5 minutes, etc. Optionally, the first time period may be related to the training data of the first neural network and the attention network. For example, the training data of the first neural network and the attention network may be data within a second time period, and the first time period may have the same length as the second time period, i.e., the time period of the training data is consistent with the length of the time series input to the first neural network. Optionally, when at least one obstacle is a vehicle, the historical attribute data of at least one obstacle may also include at least one of its headlight state, wheel state, and door state within the first time period prior to the current moment.

[0103] The features of the first vehicle may be the hidden layer features of the first vehicle output by the first neural network, and the features of at least one obstacle may include the hidden layer features of each obstacle among the at least one obstacles output by the first neural network.

[0104] In one possible implementation, the attribute data of the first vehicle and the information about its surrounding environment may include sensor data. That is, the attribute data of the first vehicle and the information about its surrounding environment can be collected by sensors in the first vehicle. The sensors are connected to a processing device, and the processing device can receive the sensor data sent by the sensors through the connection between the two.

[0105] As one possible implementation, the processing device may utilize a first neural network to extract features based on sensor data to obtain features of the first vehicle and features of at least one obstacle.

[0106] For example, the sensor data includes image data, which includes image data of the first vehicle and image data of the environment surrounding the first vehicle. For instance, the first vehicle includes a vision sensor for acquiring the image data.

[0107] As another example, the sensor data includes point cloud data, which includes image data of the first vehicle and point cloud data of the environment surrounding the first vehicle. For example, the first vehicle includes radar-based sensors that can use electromagnetic waves to detect and obtain point cloud data.

[0108] Optionally, the surrounding environment information of the first vehicle also includes attribute data of the static environment, which includes lane lines and curbs of the road. Lane lines may include dashed lines or straight lines.

[0109] Optionally, the static environment attribute data may also include at least one of the following: road connectivity, traffic signs, stop lines, pedestrian crossings, and road surface condition information. Road connectivity includes different types of connections such as intersections, T-junctions, and roundabouts, as well as interchanges on highways. Traffic signs include directional signs (such as straight, turn, and U-turn signs), warning signs (such as sharp turn, steep slope, and pedestrian warning signs), and prohibitory signs (such as no entry, no parking, and speed limit signs). Stop lines include, for example, the stop line at a red light at an intersection. Road surface condition information includes, for example, the smoothness of the road surface or the coefficient of friction of the road surface.

[0110] For example, the processing device inputs the attribute data of the first vehicle and the surrounding environment information of the first vehicle into the first neural network, so that the current attribute data of at least one obstacle, the historical attribute data of at least one obstacle, and the attribute data of the static environment are fused and interacted in time and space to extract the features of the first vehicle and the features of at least one obstacle.

[0111] For example, referring to Figure 2, obstacle 40 requires a left turn onto the road where vehicle 10 is located, while vehicle 10 needs to go straight. Obstacle 40 will affect the driving of vehicle 10. The processing device can use a first neural network to extract features based on the attribute data of vehicle 10 and the surrounding environment information of vehicle 10, thereby obtaining the features of vehicle 10 and obstacle 40.

[0112] Step S402: The processing device uses an attention network to obtain an attention result for at least one obstacle based on the features of the first vehicle and the features of at least one obstacle.

[0113] Among them, the networks of interest include at least one of SELayer, MLP, and MLN.

[0114] The attention result is used to indicate the decision outcome of the first vehicle. Optionally, the decision outcome may include one of yielding, not yielding, braking, etc.

[0115] For example, the attention network can be an SELayer. The processing device can use the SELayer to compress the features of the first vehicle through a compression operation to obtain a feature vector. The feature vector includes multiple sets of hidden layer information, each set of hidden layer information representing the features of one of the at least one obstacles. Further, the processing device can use the SELayer to process the feature vector through an activation function to obtain weight parameters for each set of hidden layer information. These weight parameters indicate the importance of one of the at least one obstacles. Still further, the processing device can use the SELayer based on the weight parameters of each set of hidden layer information to obtain the attention result for at least one obstacle.

[0116] For example, referring to FIG2, the processing device can utilize an attention network to obtain an attention result for obstacle 40 based on the features of vehicle 10 and obstacle 40. The attention result for obstacle 40 can be used to instruct vehicle 10 to brake so that vehicle 10 avoids obstacle 20, that is, the decision result of vehicle 10 is braking.

[0117] As one possible implementation, the attention network may include a first attention network and a second attention network. The processing device may utilize the first attention network to obtain intermediate features of the first obstacle based on its current attribute data and characteristics. Further, the processing device may utilize the second attention network to obtain attention results for at least one obstacle based on the intermediate features of at least one obstacle and the characteristics of the first vehicle. Here, the first obstacle is any one of the at least one obstacles.

[0118] For example, the first attention network is an MLP (Multi-Level Processing) network. The processing device inputs the current attribute data of each obstacle (e.g., the orientation, position, and velocity of each obstacle at the current moment) and the features of that obstacle into the MLP, thereby fusing the current attribute data and the features of each obstacle to obtain intermediate features of each obstacle. For example, the processing device inputs the current attribute data of the first obstacle (e.g., the orientation, position, and velocity of the first obstacle at the current moment) and the features of the first obstacle (e.g., the hidden layer features of the first obstacle output by the first neural network) into the MLP, thereby fusing the current attribute data and the features of the first obstacle to obtain intermediate features of the first obstacle, which can more accurately determine the current state of the first obstacle. Here, the first obstacle is any one of at least one obstacle.

[0119] For another example, the second attention network is SELayer. The processing device inputs the features of the first vehicle (e.g., the features of the first vehicle output by the first neural network) and the intermediate features of each obstacle into SELayer, compresses the features of the first vehicle and inspires the feature representation of at least one obstacle, and uses the features of the vehicle to observe which near-static obstacles among at least one obstacle around the vehicle need attention, i.e. which near-static obstacles may affect the driving of the vehicle.

[0120] For example, when a second obstacle affects the driving of the first vehicle, the processing device obtains the attention result of the second obstacle according to the above decision-making method and uses it to instruct the replanning of the driving path of the first vehicle so that the first vehicle avoids the second obstacle; that is, the decision result of the first vehicle is to replan the driving path of the first vehicle. Here, the second obstacle is at least one obstacle. Further, the processing device can generate a first interface for indicating that the second obstacle affects the driving of the first vehicle. Referring to Figure 5, the first interface includes controls 501, 502, 503, and 504. Control 501 is used to display the drivable space of vehicle 10 (i.e., the first vehicle) and its surrounding map, wherein the location of the drivable space of vehicle 10 and the map can be provided to vehicle 10 from the cloud. Control 502 is used to display the specific location of vehicle 10 on the map and the driving direction of vehicle 10; control 503 is used to display the specific location of obstacle 40 (i.e., the second obstacle) on the map and the driving direction of obstacle 40; and control 504 is used to indicate that obstacle 40 affects the driving of the first vehicle. Optionally, Figure 5 shows one possible interface. In actual use, control 504 can also be in other display forms, such as highlighted markings, blinking markings, etc., which will not be exemplified here.

[0121] Optionally, the first interface shown in Figure 5 is displayed through the display device 103 shown in Figure 3. For example, the first interface shown in Figure 5 is displayed through the central control screen 1031 shown in Figure 3. If the processing device of vehicle 10 obtains the attention result of obstacle 40 according to the above decision method and uses it to instruct vehicle 10 to brake so that vehicle 10 avoids obstacle 40, after the user of vehicle 10 perceives the first interface, he / she can know that obstacle 40 affects the driving of vehicle 10. The braking of vehicle 10 is related to the impact of obstacle 40 on the driving of vehicle 10, which can improve the interpretability of intelligent driving system and enhance user experience.

[0122] The training logic for the first neural network and the attention network is described below.

[0123] The first neural network and the attention network are trained based on training data. The training data includes vehicle attribute data for the second time period, information about the vehicle's surrounding environment, and ground truth results. Similarly, the vehicle attribute data for the second time period includes the vehicle's orientation, speed, position, length, and width. The vehicle attribute data for the second time period also includes at least one of the following: vehicle height, acceleration, angular velocity, angular acceleration, headlight status, wheel status, door status, and dynamic / static status. The vehicle's dynamic / static status indicates whether the vehicle is in a dynamic or static state.

[0124] The vehicle's surrounding environment information includes attribute data of surrounding obstacles during the second time period. Similarly, the attribute data of surrounding obstacles during the second time period includes the orientation, speed, position, length, and width of the surrounding obstacles during the second time period. The current attribute data of surrounding obstacles may also include at least one of the following: height, acceleration, angular velocity, angular acceleration, and dynamic / static state of the surrounding obstacles during the second time period. The dynamic / static state of the surrounding obstacles is used to indicate whether the vehicle is dynamic or static. The surrounding obstacles are near-static obstacles. Optionally, when the surrounding obstacle is a vehicle, the historical attribute data of the surrounding obstacle may also include at least one of the following: headlight state, wheel state, and door state of the surrounding obstacle during the second time period.

[0125] Optionally, the vehicle's surrounding environment information also includes static environment attribute data, which includes road lane lines and curbs. Similarly, the static environment attribute data may also include at least one of the following: road shape, road slope, number of lanes, lane width, road connectivity, traffic signs, stop lines, pedestrian crossings, and road surface condition information.

[0126] The ground truth results are used to indicate whether surrounding obstacles affect the vehicle's movement during the second time period. For example, the ground truth results include a "true" or "false" label for each obstacle. A "true" label indicates that the obstacle will affect the vehicle's movement during the second time period, while a "false" label indicates that the obstacle will not affect the vehicle's movement during the second time period. Optionally, the labels can be manually labeled and / or pre-labeled by a deep learning model.

[0127] In one possible implementation, the processing device acquires the vehicle's attribute data and surrounding environment information during a second time period. Through manual annotation and / or a deep learning model, it determines whether the obstacle undergoes a static-dynamic transition during the second time period, such as changing from static to dynamic. Further, if the obstacle undergoes a static-dynamic transition during the second time period, and it is determined that the obstacle may affect the vehicle's driving in the near future, the processing device can mark the obstacle as "yes". If the obstacle does not undergo a static-dynamic transition during the second time period, and it is determined that the obstacle will not affect the vehicle's driving in the near future, the processing device can mark the obstacle as "no".

[0128] Optionally, the vehicle's attribute data and surrounding environment information in the training data during the second time period can be image data. For example, there may be 24 frames of images within 1 second, and the processing device can combine multiple frames to annotate each frame during annotation. For instance, obstacle A starts moving from frame 8 until it stops moving in frame 18; obstacle A is stationary in the first 7 frames. The processing device can label the ground truth result of obstacle A in the first 7 frames as "No". From frame 8 to frame 18, obstacle A is moving, and the processing device can label the ground truth result of obstacle A in frames 8 to 18 as "Yes".

[0129] Optionally, after all obstacles are labeled, the processing device can review all obstacle labels and reset outlier labels. For example, if the truth value of the label corresponding to obstacle A is "yes" for 10 consecutive frames, the truth value of the label corresponding to obstacle A is "no" for the next 3 consecutive frames, and the truth value of the label corresponding to obstacle A is "yes" for the last 10 consecutive frames, then the truth value of the label corresponding to obstacle A in the middle 3 frames will be flipped to "yes", which is a consistency filtering.

[0130] For example, the processing device inputs the vehicle's attribute data and the vehicle's surrounding environment information during a second time period into a first neural network, and uses the first neural network to extract features based on the vehicle's attribute data and the surrounding environment information during the second time period to obtain the features of the vehicle and the features of the surrounding obstacles. Further, the processing device inputs the vehicle's features and the features of the surrounding obstacles into an attention network, and uses the attention network to obtain attention results for the surrounding obstacles based on the vehicle's features and the features of the surrounding obstacles.

[0131] Furthermore, the processing device adjusts the model parameters of the first neural network and the attention network based on the ground truth results and the attention results from surrounding obstacles. It should be noted that each time the first neural network and the attention network are trained, they fine-tune their model parameters based on the labeled ground truth results to make the attention results obtained by the attention network more accurate. This process improves the accuracy of the attention results output by the attention network, thereby continuously optimizing the first neural network and the attention network.

[0132] In the embodiment shown in Figure 4, the processing device can use a first neural network to extract features from the vehicle's attribute data and surrounding environment information. Based on the features extracted by the first neural network, an attention network is used to observe which near-static obstacles around the vehicle require attention—that is, which near-static obstacles might affect the vehicle's driving—and thus determine the vehicle's decision. In this way, the trajectory prediction process of the prediction module can be omitted. Using the first neural network and the attention network, for near-static obstacles, the attention results can be directly output to indicate the vehicle's decision, improving the response speed and accuracy of the decision. Furthermore, this application is not limited by the length of the prediction line, reducing the possibility of decision errors, lowering the potential collision risk, and improving the safety of the intelligent driving system.

[0133] The methods of the embodiments of this application have been described in detail above. Below, some apparatuses for implementing the foregoing methods are described. It should be understood that the division of units in the apparatuses provided in the embodiments of this application is only a logical functional division; in actual implementation, they can be fully or partially integrated onto a single physical entity, or they can be physically separated.

[0134] Furthermore, the units or modules in the device can be implemented in the form of processor calling software. For example, the device includes a processor connected to a memory, which stores instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each unit of the device. The processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is either internal or external to the device.

[0135] Alternatively, the units or modules in the device can be implemented in the form of hardware circuits. The functionality of some or all units can be achieved through the design of these hardware circuits, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functionality of some or all of the above units is achieved through the design of the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a programmable logic device (PLD), such as a field-programmable gate array (FPGA). This PLD can include a large number of logic gates, and the connection relationships between these logic gates can be configured through configuration files to achieve the functionality of some or all of the above units. All units of the above device can be implemented entirely through processor-invoked software, entirely through hardware circuits, or partially through processor-invoked software with the remaining parts implemented through hardware circuits.

[0136] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU) or a digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor is a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. Therefore, each unit in the device can be one or more processors (or processing circuits) configured to implement the above methods, such as a CPU, GPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor types.

[0137] Furthermore, the units or modules in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units or modules are integrated together as a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA.

[0138] Several possible devices are listed below.

[0139] Please refer to Figure 6, which is a schematic diagram of a processing device provided in an embodiment of this application, namely processing device 60. Optionally, the processing device 60 can be a standalone device, such as the processing device 101 shown in Figure 1. Alternatively, the processing device 60 can also be a component in a standalone device (such as a node), such as a chip or integrated circuit. The processing device 60 is used to execute the steps performed by the processing device in the decision-making method shown in Figure 4 above.

[0140] As shown in Figure 6, the processing device 60 includes a processing unit 601. The processing unit 601 is used to perform one or more operations such as processing, calculation, determination, generation, and updating. For example, it uses a first neural network to extract features based on the attribute data of a first vehicle and the surrounding environment information of the first vehicle to obtain the features of the first vehicle and the features of at least one obstacle. The surrounding environment information of the first vehicle includes the current attribute data and historical attribute data of at least one obstacle. The historical attribute data of the at least one obstacle is the attribute data of the at least one obstacle within a first time period before the current moment. Each of the at least one obstacle is a near-static obstacle, meaning it is stationary or its speed is less than a first threshold. Furthermore, the processing unit 601 is also used to use an attention network to obtain the attention result of at least one obstacle based on the features of the first vehicle and the features of the at least one obstacle. The attention result is used to indicate the decision result of the first vehicle.

[0141] In one possible implementation, the first neural network and the attention network are trained based on training data. The training data includes vehicle attribute data for a second time period, information about the vehicle's surrounding environment, and ground truth results. The vehicle's surrounding environment information includes attribute data of surrounding obstacles for the second time period and attribute data of the static environment. The ground truth results indicate whether surrounding obstacles affect the vehicle's movement during the second time period; these obstacles are considered near-static. The first neural network is used to extract features based on the vehicle's attribute data for the second time period and the vehicle's surrounding environment information, obtaining features of the vehicle and surrounding obstacles. The attention network is used to obtain attention results for the surrounding obstacles based on the vehicle's features and the features of the surrounding obstacles. The ground truth results are used to adjust the model parameters of the first neural network and the attention network in conjunction with the attention results for the surrounding obstacles.

[0142] In one possible implementation, the attribute data of the first vehicle includes the first vehicle's orientation, speed, position, length, and width. The attribute data of the first vehicle also includes at least one of the following: height, acceleration, angular velocity, angular acceleration, headlight status, wheel status, door status, and dynamic / static status, wherein the dynamic / static status of the first vehicle indicates whether the first vehicle is in a dynamic or static state.

[0143] In one possible implementation, the current attribute data of at least one obstacle includes the obstacle's orientation, velocity, position, length, and width at the current moment. The current attribute data of at least one obstacle also includes at least one of the obstacle's height, acceleration, angular velocity, angular acceleration, and dynamic / static state at the current moment. The historical attribute data of at least one obstacle includes the obstacle's orientation, velocity, position, length, and width during a first time period prior to the current moment. The historical attributes of at least one obstacle also include at least one of the obstacle's height, acceleration, angular velocity, angular acceleration, and dynamic / static state during the first time period prior to the current moment, where the dynamic / static state of at least one obstacle indicates whether the obstacle is dynamic or static.

[0144] In one possible implementation, the environmental information surrounding the first vehicle also includes attribute data of the static environment, which includes lane lines and curbs of the road.

[0145] In one possible implementation, the attention network includes a first attention network and a second attention network. Processing unit 601 is further configured to utilize the first attention network to obtain intermediate features of the first obstacle based on its current attribute data and features. Processing unit 601 is also configured to utilize the second attention network to obtain attention results for at least one obstacle based on the intermediate features of at least one obstacle and the features of the first vehicle. Here, the first obstacle is any one of the at least one obstacles.

[0146] In one possible implementation, processing unit 601 is further configured to compress the features containing the first vehicle through a compression operation to obtain a feature vector. Processing unit 601 is further configured to process the feature vector using an activation function to obtain weight parameters for each set of hidden layer information, and based on the weight parameters for each set of hidden layer information, obtain a focus result for at least one obstacle. The feature vector includes multiple sets of hidden layer information, each set of hidden layer information is used to characterize the features of one obstacle among multiple obstacles, and the weight parameters are used to indicate the importance of each obstacle among the at least one obstacle.

[0147] In one possible implementation, the attribute data of the first vehicle and the information about the surrounding environment of the first vehicle include sensor data.

[0148] In one possible implementation, the first neural network includes a transformer network or a CNN network.

[0149] In one possible implementation, the network of interest includes at least one of the compressed excitation network SELayer, MLP, and MLN.

[0150] In one possible implementation, if a second obstacle affects the movement of a first vehicle, the attention result of the second obstacle is used to instruct the replanning of the first vehicle's driving path so that the first vehicle avoids the second obstacle, which is at least one obstacle.

[0151] In one possible implementation, the processing unit 601 is further configured to generate a first interface, the first interface being configured to indicate that a second obstacle affects the driving of the first vehicle, the second obstacle being at least one obstacle.

[0152] It further includes other operations for implementing the decision-making method, and related descriptions can be found in the description of the embodiment shown in Figure 4, which will not be described in detail here.

[0153] Please refer to Figure 7, which is a schematic diagram of the structure of a computing device provided in an embodiment of this application. A computing device is a device with processing capabilities. The device here can be a physical device, such as a host, or a virtual device, such as a virtual machine or a container.

[0154] As shown in Figure 7, the computing device 70 includes a processor 701, a memory 702, and one or more programs, and may include a communication interface 703. It should be understood that this application does not limit the number of processors and memories in the computing device 70.

[0155] Processor 701 is a module for performing calculations and may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), a micro controller unit (MCU), or one or more integrated circuits for controlling the execution of programs in the above schemes.

[0156] Memory 702 provides storage space, in which application data, user data, operating system, and computer programs can be optionally stored. Memory 702 may include read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0157] The memory 702 can exist independently and be connected to the processor 701 via a bus. Alternatively, the memory 702 can be integrated with the processor 701.

[0158] The communication interface 703 is used to provide information input or output to the at least one processor. And / or, the communication interface 703 can be used to receive data transmitted externally and / or transmit data externally. The communication interface 703 can be a wired link interface, such as an Ethernet cable, or a wireless link interface (Bluetooth, general wireless transmission, and other wireless communication technologies, etc.). Optionally, the communication interface 703 may also include a transmitter (such as a radio frequency transmitter, antenna, etc.) or a receiver coupled to the interface.

[0159] In this embodiment, one or more programs are stored in the memory 702 in the form of program code and configured to be executed by the processor 701. The programs include instructions for implementing the steps performed by the processing device in the decision-making method shown in FIG. 4. That is, the memory 702 stores executable instructions, and the processor 701 executes these executable instructions to implement the method implemented by the processing device in the decision-making method shown in FIG. 4. In other words, the memory 702 stores instructions for executing the steps performed by the processing device in the decision-making method shown in FIG. 4.

[0160] This application embodiment also provides a vehicle, which includes the aforementioned processing device 60 or the aforementioned computing device 70, and the vehicle is used to implement the decision-making method shown in FIG4.

[0161] This application also provides a chip, which includes a processor for executing computer execution instructions to enable a device equipped with the chip to implement the decision-making method shown in FIG4 above.

[0162] This application also provides a computer program product containing computer instructions. The computer program product may be a software or program product containing computer instructions, capable of running on a computing device or stored on any usable medium. When the computer instructions are executed by a processor, the aforementioned decision-making method is implemented, such as the decision-making method shown in FIG4.

[0163] This application also provides a computer-readable storage medium. This computer-readable storage medium is used to store a computer program, the computer program including instructions for implementing the aforementioned decision-making method, such as the decision-making method shown in FIG4.

[0164] The computer-readable storage medium can be any available medium that can be stored in any device of a processing apparatus or computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid-state drive).

[0165] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0166] In this application, "at least one" in the embodiments refers to one or more items, and "more than one" refers to two or more items. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, and c can be single or multiple. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0167] Furthermore, unless otherwise stated, the use of ordinal numbers such as "first" and "second" in the embodiments of this application is for distinguishing multiple objects and is not for limiting the order, sequence, priority or importance of multiple objects.

[0168] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A decision-making method, characterized in that, Applied to a first vehicle, the method includes: Using a first neural network, feature extraction is performed based on the attribute data of the first vehicle and the surrounding environment information of the first vehicle to obtain the features of the first vehicle and the features of at least one obstacle. The surrounding environment information of the first vehicle includes the current attribute data of the at least one obstacle and the historical attribute data of the at least one obstacle. The historical attribute data of the at least one obstacle is the attribute data of the at least one obstacle in a first time period before the current moment. Each of the at least one obstacle is a near-static obstacle, and the near-static obstacle is in a stationary state or the speed of the near-static obstacle is less than a first threshold. Using an attention network, based on the features of the first vehicle and the features of the at least one obstacle, an attention result for the at least one obstacle is obtained, and the attention result is used to indicate the decision result of the first vehicle.

2. The method according to claim 1, characterized in that, The first neural network and the attention network are trained based on training data; The training data includes vehicle attribute data during a second time period, the vehicle's surrounding environment information, and ground truth results. The vehicle's surrounding environment information includes attribute data of surrounding obstacles during the second time period. The ground truth results are used to indicate whether the surrounding obstacles affect the vehicle's driving during the second time period. The surrounding obstacles are the near-static obstacles. The first neural network is used to extract features based on the vehicle's attribute data during a second time period and the vehicle's surrounding environment information to obtain the features of the vehicle and the features of the surrounding obstacles. The attention network is used to obtain attention results for the surrounding obstacles based on the features of the vehicle and the features of the surrounding obstacles; The ground truth results are used to adjust the model parameters of the first neural network and the model parameters of the attention network in combination with the attention results of the surrounding obstacles.

3. The method according to claim 1 or 2, characterized in that, The attribute data of the first vehicle includes the orientation, speed, position, length and width of the first vehicle. The attribute data of the first vehicle also includes at least one of the following: height, acceleration, angular velocity, angular acceleration, headlight status, wheel status, door status and dynamic / static status of the first vehicle. The dynamic / static status of the first vehicle is used to indicate whether the first vehicle is in a dynamic state or in a static state.

4. The method according to any one of claims 1-3, characterized in that, The current attribute data of the at least one obstacle includes the orientation, speed, position, length, and width of the at least one obstacle at the current moment; The current attribute data of the at least one obstacle also includes at least one of the following: height, acceleration, angular velocity, angular acceleration, and dynamic / static state of the at least one obstacle at the current moment; The historical attribute data of the at least one obstacle includes the orientation, speed, position, length, and width of the at least one obstacle during a first time period prior to the current moment; The historical attributes of the at least one obstacle also include at least one of the following: height, acceleration, angular velocity, angular acceleration, and dynamic / static state of the at least one obstacle during a first time period prior to the current moment, wherein the dynamic / static state of the at least one obstacle is used to indicate whether the at least one obstacle is in a dynamic or static state.

5. The method according to any one of claims 1-4, characterized in that, The surrounding environment information of the first vehicle also includes static environment attribute data, which includes road lane lines and curbs.

6. The method according to any one of claims 1-5, characterized in that, The attention network includes a first attention network and a second attention network; the step of using the attention network to obtain the attention result of the at least one obstacle based on the features of the first vehicle and the features of the at least one obstacle includes: Using the first attention network, based on the current attribute data of the first obstacle and the features of the first obstacle, the intermediate features of the first obstacle are obtained, wherein the first obstacle is any one of the at least one obstacles; Using the second attention network, attention results for the at least one obstacle are obtained based on the intermediate features of the at least one obstacle and the features of the first vehicle.

7. The method according to any one of claims 1-6, characterized in that, The attribute data of the first vehicle and the information about the surrounding environment of the first vehicle include sensor data.

8. The method according to any one of claims 1-7, characterized in that, The first neural network includes a transformer network or a convolutional neural network (CNN).

9. The method according to any one of claims 1-8, characterized in that, The attention network includes at least one of the following: Compressed Excitation Network (SELayer), Multilayer Perceptron (MLP), and Motion-Aware Normalized Network (MLN).

10. The method according to any one of claims 1-9, characterized in that, The method further includes: A first interface is generated, which is used to indicate that a second obstacle affects the driving of the first vehicle, and the second obstacle belongs to the at least one obstacle.

11. A processing apparatus, characterized in that, The processing device includes: Processing unit, used for: Using a first neural network, feature extraction is performed based on the attribute data of the first vehicle and the surrounding environment information of the first vehicle to obtain the features of the first vehicle and the features of at least one obstacle. The surrounding environment information of the first vehicle includes the current attribute data of the at least one obstacle and the historical attribute data of the at least one obstacle. The historical attribute data of the at least one obstacle is the attribute data of the at least one obstacle in a first time period before the current moment. Each of the at least one obstacle is a near-static obstacle, and the near-static obstacle is in a stationary state or the speed of the near-static obstacle is less than a first threshold. Using an attention network, based on the features of the first vehicle and the features of the at least one obstacle, an attention result for the at least one obstacle is obtained, and the attention result is used to indicate the decision result of the first vehicle.

12. A computing device, characterized in that, The computing device includes a processor and a memory, the memory storing a program, and the processor executing the program to cause the computing device to perform the method as described in any one of claims 1-10.

13. A vehicle, characterized in that, The vehicle includes the processing device as described in claim 11 or the computing device as described in claim 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, the computer program including instructions for performing the method as described in any one of claims 1-10.

15. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a processor, cause the method as described in any one of claims 1-10 to be implemented.

16. A chip, characterized in that, Includes a processor for executing computer execution instructions to cause a device having the chip mounted to perform the method as described in any one of claims 1-10.

Images