Trajectory prediction method based on spatio-temporal interaction feature fusion

Abstract

A trajectory prediction method based on spatio-temporal interaction feature fusion. The method comprises the following steps: collecting information of traffic participants on a road in real time; on the basis of the information of the traffic participants, constructing a traffic scene representation model; constructing a multi-scale spatio-temporal graph convolution model to enable the multi-scale spatio-temporal graph convolution model to extract spatio-temporal interaction features of the traffic participants on the basis of the traffic scene representation model; and constructing a trajectory prediction model to enable the trajectory prediction model to perform trajectory prediction on the basis of the spatio-temporal interaction features, so as to acquire a predicted trajectory. In the method, a traffic scene representation is constructed, and spatio-temporal interaction features of traffic participants are extracted, such that the resulting predicted vehicle trajectory can capture the spatial correlation of traffic participant behaviors on different time scales, and multi-scale spatio-temporal fused interaction features are extracted, thereby effectively improving the accuracy of trajectory prediction.

Description

A trajectory prediction method based on spatiotemporal interaction feature fusion

[0001] Cross-reference of related applications

[0002] This application claims priority to Chinese Patent Application No. 202510056758.5, filed on January 14, 2025, entitled "A Trajectory Prediction Method Based on Spatiotemporal Interaction Feature Fusion", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to the field of intelligent driving technology, and in particular to a trajectory prediction method based on spatiotemporal interaction feature fusion. Background Technology

[0004] Accurate prediction of the future trajectories of surrounding road users is crucial for intelligent driving systems, serving as a prerequisite for rational and efficient behavioral interaction and collision detection between autonomous vehicles and other participants. In complex traffic scenarios, the future behavior of surrounding road users is highly uncertain, making it difficult for intelligent driving systems to characterize their highly uncertain future behaviors with a deterministic predicted trajectory. Therefore, in each decision-making cycle, the intelligent driving system not only needs to detect collision risks based on the most probable behavior of surrounding road users but also needs to comprehensively consider the probability distribution of all their possible future behaviors.

[0005] To comprehensively and realistically describe the highly uncertain future behavior of surrounding traffic participants, some recent technological studies have achieved multimodal trajectory prediction of surrounding traffic participants by considering road topology. However, due to the limited number of predicted trajectory modes, these methods still cannot comprehensively describe the highly uncertain future behavior of surrounding traffic participants. In addition, many methods lack consideration of vehicle kinematics, and the generated trajectories sometimes do not conform to vehicle kinematic constraints, making it difficult to provide accurate information on the movement of surrounding obstacles for subsequent decision-making and planning stages.

[0006] Many prediction methods have begun to improve trajectory prediction accuracy by modeling the behavioral interactions between traffic participants. However, existing trajectory prediction methods do not adequately model the spatial and temporal correlations of these behavioral interactions, resulting in low accuracy. Most of these methods only extract the behavioral interactions from spatial correlations, paying little attention to the temporal correlations. While other works model both spatial and temporal correlations, they do so separately, neglecting to capture more spatial correlations at different time scales. Summary of the Invention

[0007] This invention aims to provide a trajectory prediction method based on spatiotemporal interaction feature fusion to solve the above-mentioned technical problems. By constructing a traffic scene representation and extracting the spatiotemporal interaction features of traffic participants, the spatial correlation of traffic participants' behavior at different time scales can be captured, effectively improving the accuracy of trajectory prediction.

[0008] To address the aforementioned technical problems, this invention provides a trajectory prediction method based on spatiotemporal interactive feature fusion, comprising the following steps:

[0009] Real-time collection of information on traffic participants on the road;

[0010] Construct a traffic scenario representation model based on traffic participant information;

[0011] Construct a multi-scale spatiotemporal graph convolutional model to extract the spatiotemporal interaction features of traffic participants based on the traffic scene representation model;

[0012] A trajectory prediction model is constructed to predict trajectories based on spatiotemporal interaction features and obtain the predicted trajectories.

[0013] The above scheme constructs a traffic scene representation and extracts the spatiotemporal interaction features of traffic participants, so that the final vehicle prediction trajectory can capture the spatial correlation of traffic participants' behavior at different time scales and extract multi-scale spatiotemporal fusion interaction features, which can effectively improve the accuracy of trajectory prediction.

[0014] Furthermore, the traffic scene representation model includes an Euclidean distance adjacency matrix, a potential collision adjacency matrix, and an adaptive adjacency matrix; the construction of the traffic scene representation model based on traffic participant information includes:

[0015] Based on traffic participant information, a Euclidean distance adjacency matrix is ​​constructed, specifically represented as follows:

[0016] In the formula, It is an Euclidean distance adjacency matrix An element in the equation represents the adjacency relationship between traffic participant i and traffic participant j at time t; D represents the distance between traffic participant i and traffic participant j at time t; close The distance threshold between neighboring participants is represented; 1 indicates that traffic participant i and traffic participant j are neighboring participants at time t; 0 indicates that traffic participant i and traffic participant j are not neighboring participants at time t.

[0017] It should be noted that once a traffic participant and a vehicle are identified as neighboring participants, the traffic participant's behavior will be considered to influence the vehicle's trajectory in subsequent prediction stages. Furthermore, in real traffic conditions, the distance threshold D... close The behavior of external traffic participants still affects vehicles. Therefore, to address the issue that the fixed distance threshold of the Euclidean distance adjacency matrix cannot be dynamically adjusted, this solution obtains the current speed and direction of traffic participants based on their information, generates multiple directed line segments, and constructs a future trajectory model of the traffic participants based on these directed line segments. Specifically, this is represented as follows:

[0018] In the formula, L pred Indicates the length of a directed line segment; u cur The current speed of the traffic participant; t pred For the prediction time domain; θ represents the direction of the future trajectory; θ cur Current direction for traffic participants;

[0019] The potential collision adjacency matrix is ​​constructed based on the future trajectory model of traffic participants, and is specifically represented as follows:

[0020] In the formula, It is a potential collision adjacency matrix An element in the equation represents the potential collision relationship between traffic participant i and traffic participant j at time t. In this calculation, traffic participant i is the vehicle itself; AB is the future trajectory of the vehicle calculated based on the future trajectory model of the traffic participants; CD is the future trajectory of the current traffic participant calculated based on the future trajectory model of the traffic participants; if there is an intersection point (x, y), it means that the future trajectory of the vehicle and the future trajectory of the current traffic participant overlap, and there is a potential collision relationship. The value is assigned to 1.

[0021] It should be noted that the potential collision adjacency matrix This approach uses a simplified collision risk assessment algorithm to identify traffic participants with mutual influence. Because the future trajectory is unknown during the trajectory prediction phase, this scheme uses a directed line segment generated based on the participant's current speed and direction as the future trajectory for collision detection.

[0022] Building upon the Euclidean distance adjacency matrix and potential collision adjacency matrix constructed above, to further uncover hidden behavioral interaction relationships, an adaptive adjacency matrix can be constructed based on traffic participant information, specifically represented as follows:

[0023] In the formula, It is a potential adaptive adjacency matrix An element in E represents the adaptive relationship between traffic participant i and traffic participant j at time t; i This indicates that traffic participant i is the source node, and E j This indicates that traffic participant j is the target node; ReLU is the activation function, which is used to eliminate weak connections and increase the nonlinearity of the network layers; SoftMax is the normalization function used to normalize the adaptive adjacency matrix; the adaptive adjacency matrix... Used to capture the dependency between the source node and the target node embedding, that is, to represent the interaction between traffic participant i and traffic participant j.

[0024] It should be noted that node embedding maps a node to a multidimensional vector and uses this multidimensional vector to express, or more simply, replaces the node as input to subsequent neural network layers.

[0025] Furthermore, the construction of a multi-scale spatiotemporal graph convolutional model, which enables the multi-scale spatiotemporal graph convolutional model to extract spatiotemporal interaction features of traffic participants based on the traffic scene representation model, includes:

[0026] A multi-scale spatiotemporal graph convolutional model consisting of multiple stacked dilated TCN-GCN spatiotemporal layers is constructed. In this model, the GCN in each spatiotemporal layer is used to capture the interaction relationships between traffic participants at different time scales, and the dilated TCN in each spatiotemporal layer is used to capture the movement trends of traffic participants over time.

[0027] The multi-scale spatiotemporal graph convolutional model extracts the spatiotemporal interaction features of traffic participants based on the traffic scene representation model;

[0028] Wherein: Dilated TCN represents a dilated temporal convolutional network; GCN represents a graph convolutional network.

[0029] The dilated TCN constructed by the above scheme achieves the best balance between prediction performance and computational efficiency, providing feasibility for the algorithm to be deployed on low-computing-power domain controller chips; and it constructs a way to represent the traffic scene expression model by three adjacency matrices: Euclidean distance adjacency matrix, potential collision adjacency matrix, and adaptive adjacency matrix, which is conducive to fully modeling prior knowledge and thus optimizing the performance of GCN graph convolutional layers.

[0030] Furthermore, in the multi-scale spatiotemporal graph convolution model, for any dilated TCN, its dilation convolution operation F(t) at time t is specifically expressed as:

[0031] In the formula: Q represents a one-dimensional input sequence; f represents a filter; a is the element index in the filter; k1 is the size of the filter; d represents the dilation factor, which is used to control the distance between convolution kernel elements; in the multi-scale spatiotemporal graph convolution model, the d value of the odd-dilated TCN-GCN spatiotemporal layer is 1, and the d value of the even-dilated TCN-GCN spatiotemporal layer is 2.

[0032] Furthermore, in the multi-scale spatiotemporal graph convolution model, for any GCN, its output spatiotemporal interaction feature Z is specifically expressed as:

[0033] In the formula, k2 represents the diffusion step; This represents the power series of the Euclidean distance adjacency matrix; Represents the power series of the potential collision adjacency matrix; H represents the power series of the adaptive adjacency matrix; H is the output of the dilated TCN. This is the learnable parameter matrix in the convolution operation; Indicates the last observation timestamp t obs The Euclidean distance adjacency matrix; Indicates the last observation timestamp t obs The potential collision adjacency matrix; Indicates the last observation timestamp t obs The adaptive adjacency matrix; the Euclidean distance adjacency matrix, the potential collision adjacency matrix, and the adaptive adjacency matrix constitute the traffic scene representation model.

[0034] Furthermore, the construction of the trajectory prediction model, which enables the trajectory prediction model to predict trajectories based on spatiotemporal interaction features and obtain predicted trajectories, includes:

[0035] Construct a trajectory prediction model with an encoder-decoder structure; wherein both the encoder and the decoder contain LSTM neural networks;

[0036] The spatiotemporal interaction features are input into the encoder, and after being learned and processed by the LSTM neural network, the contextual feature information is obtained.

[0037] The contextual feature information is input into the decoder, processed and learned by the LSTM neural network, and combined with the vehicle's current position to obtain multiple sequence points;

[0038] Trajectory prediction is performed based on multiple sequence points to obtain the predicted trajectory.

[0039] Furthermore, the construction of the trajectory prediction model, which enables the trajectory prediction model to predict trajectories based on spatiotemporal interaction features and obtain predicted trajectories, includes:

[0040] A trajectory prediction model is constructed, comprising a target lane prediction sub-model, a trajectory cluster prediction sub-model, and a trajectory evaluation output sub-model; wherein:

[0041] The target lane prediction sub-model is used to obtain the vehicle's target lane based on pre-acquired vehicle historical trajectory data and lane data;

[0042] The trajectory cluster prediction sub-model is used to obtain the initial predicted trajectory cluster of the vehicle based on the pre-acquired historical trajectory data of the vehicle and the vehicle's target lane.

[0043] The trajectory evaluation output sub-model is used to construct a vehicle trajectory cluster based on the initial predicted trajectory cluster of the vehicle and evaluate the vehicle trajectory cluster based on spatiotemporal interaction features, calculate the probability distribution of the vehicle trajectory cluster, and obtain the predicted trajectory.

[0044] Furthermore, the target lane prediction sub-model is used to obtain the vehicle's target lane based on pre-acquired historical vehicle trajectory data and lane data, including:

[0045] Based on the pre-acquired historical vehicle trajectory data and lane data, the relative position of the vehicle and the candidate lane is obtained;

[0046] Based on the cross-attention mechanism and the relative position of the vehicle and the candidate lane, the correlation between the candidate lane centerline and the vehicle's historical trajectory is calculated.

[0047] The target lane of the vehicle is obtained based on the correlation between the candidate lane centerline and the vehicle's historical trajectory.

[0048] In the above approach, directly sampling all lanes around the vehicle and generating corresponding vehicle trajectory clusters without consideration would result in an excessively large number of trajectory clusters, increasing the complexity of the prediction model. Therefore, to reduce the sampling space of the predicted trajectory clusters, this approach utilizes a cross-attention mechanism to first select the target lane where the vehicle is most likely to travel, and then generates vehicle trajectory clusters based on this lane.

[0049] Furthermore, obtaining the target lane for the vehicle primarily relies on the relative positional relationship between the vehicle's historical trajectory and the centerlines of surrounding lanes, i.e., the relative position of the vehicle and the candidate lane. To characterize this relative positional relationship, the Frenet coordinates s and d of the vehicle's historical trajectory on the centerline of each candidate lane are first calculated. Here, s represents the longitudinal distance traveled along the lane centerline, and d represents the lateral distance perpendicular to the lane centerline. For curved lanes, the Frenet coordinate system, compared to Cartesian coordinates, can more clearly and directly describe the relative positional relationship between the vehicle and the lane. After describing the relative positional relationship between the vehicle and the candidate lane using Frenet coordinates, this scheme can utilize a cross-attention mechanism to handle the relationship between the sequence of vehicle historical trajectories and the sequence of path points on the candidate lane centerlines. Both cross-attention and self-attention mechanisms belong to attention networks, which model and capture the dependencies of the input sequence through (Q, K, V) triples. Based on the different input sources of the three vectors Q, K, and V, in this scheme, the candidate lane centerline can be used as the query vector Q for the cross-attention mechanism, and the vehicle historical trajectory can be used as the key vector K and the value vector V, thereby constructing the self-attention mechanism and the cross-attention mechanism.

[0050] Furthermore, in the self-attention mechanism, Q, K, and V are obtained by multiplying the same feature input vector X by three different weight matrices W. Q W K W V The generated vector has three identical dimensions, i.e., Q = XW. Q K = XW K V = XW V The weight matrix can be obtained through training a neural network. The essence of the self-attention mechanism is to calculate the mutual influence between elements within the input vector X, which can be implemented using a Transformer network. The query vector Q in the cross-attention mechanism is determined by the lane centerline X. A K and V are derived from the vehicle's historical trajectory X. B The transformed sequence is primarily used to model the relationship between two different sequences, i.e., Q = X. A W Q K = X B W K V = X B W V .

[0051] Therefore, in the target lane prediction sub-model, the lane centerline X A The query vector Q for the cross-attention mechanism, and the vehicle's historical trajectory X BAs the key vector K and value vector V, they are used together for cross-attention operation to obtain the correlation between the lane centerline and the historical trajectory. The specific steps are as follows:

[0052] 1) Feature mapping: Q = X A W Q K = X B W K V = X B W V X A X is the center line of the lane. B For vehicle historical trajectory;

[0053] 2) Similarity Calculation: The similarity score between the query vector Q and the key vector K is calculated using the dot product of Q and K. These similarity scores represent the similarity between the lane centerline X and the key vector K. A Each element in the X-ray is related to the vehicle's historical trajectory. B The relationship between all elements in the table can be represented as:

[0054] Where S is the similarity score matrix, It is a scaling factor used to prevent the dot product value from becoming too large;

[0055] 3) Attention weight calculation: The similarity score is normalized using the Softmax function to obtain the attention weight, which reflects the lane centerline X. A Each element in the X-ray is related to the vehicle's historical trajectory. B The importance of all elements in W can be represented as: a A→B =Softmax(S A→B )

[0056] Among them, W a It is attention weight;

[0057] 4) Weighted Summation: The attention weights are weighted and summed with the value vector V to obtain the output representation after cross-attention adjustment, which can be expressed as: O A→B =W a A→B V B

[0058] In the formula, O A→B The dimension of O is consistent with the dimension of the query vector Q. Through the aforementioned attention mechanism, O A→B Each element in the model that corresponds one-to-one with the query vector Q contains the association information between the key vector K and the value vector V, thus modeling the lane centerline X. A and vehicle historical trajectory X B The interrelationship between them.

[0059] Furthermore, after encoding the lane centerline information and vehicle historical trajectories to establish their interrelationship, all candidate lanes can be constructed as an [n,c]-dimensional vector: each candidate lane is represented by a c-dimensional vector, and this is sequentially increased to n lanes to form an [n,c]-dimensional lane array. Here, n is the number of candidate lanes, and c is the feature dimension of the lane. Next, a Transformer self-attention network is used to learn the connections and differences between candidate lanes, and a Multilayer Perceptron (MLP) and Softmax are used to output the probability distribution of the candidate lanes. Finally, considering possible prediction errors, this scheme achieves a balance between sampling efficiency and prediction accuracy, selecting the top 3 lanes with the highest probabilities as the predicted target lanes for the vehicle.

[0060] Furthermore, the trajectory cluster prediction sub-model is used to obtain the initial predicted trajectory cluster of the vehicle based on the pre-acquired historical trajectory data of the vehicle and the vehicle's target lane, including:

[0061] Construct the Frenet coordinate system for the lane centerline of the target lane of the vehicle;

[0062] In the Frenet coordinate system of the lane centerline, spatial sampling of the vehicle target lane is performed based on the local path planning method to obtain the first path polynomial curve;

[0063] Transform the first path polynomial curve into Cartesian coordinates to obtain the second path polynomial curve.

[0064] Curvature detection is performed on the second path polynomial curve, and path polynomial curves that meet the preset requirements are selected to obtain the initial predicted trajectory cluster of the vehicle.

[0065] It should be noted that, in order to generate a predicted trajectory cluster that conforms to the road topology and vehicle kinematics, the concept of local path planning is applied to spatially sample the target lane, and the initial predicted trajectory cluster of the vehicle is expressed using a polynomial curve, which may include the following steps:

[0066] 1) Sampling is performed in the Frenet coordinate system along the lane centerline, where s represents the longitudinal distance traveled along the centerline and d represents the lateral distance perpendicular to the centerline. The target lane centerline is fitted as a function of parameter s using a cubic spline curve:

[0067] In the formula, a x ,b x ,c x , a y ,b y ,c y , Here, x and y are the spline curve coefficients, and x and y are the values ​​of the centerline point in Cartesian coordinates. Parameterizing the centerline facilitates subsequent reconstruction of the trajectory family to Cartesian coordinates via s-interpolation.

[0068] 2) Construct the longitudinal distance s and lateral distance d of the initial predicted trajectory cluster of the vehicle in Frenet coordinates as fourth-order and fifth-order polynomials of the prediction duration t, respectively:

[0069] In the formula, for a fourth-degree polynomial, the initial states can be solved simultaneously. and terminal state Five equations are established for solving, with the states corresponding to the initial longitudinal distance, initial longitudinal velocity, initial longitudinal acceleration, end-effector longitudinal velocity, and end-effector longitudinal acceleration, respectively. Similarly, the coefficients of the fifth-order polynomial for the lateral distance d are obtained by simultaneously solving the equations for the initial states. and terminal state Six equations are established for solving.

[0070] 3) Sample different terminal states S t and terminal state D t By combining these parameters, a series of first path polynomial curves in Frenet coordinates can be generated. Then, by interpolating using the lane centerline parameterized in step 1), the curves are restored to Cartesian coordinates to obtain the second path polynomial curves. In this scheme, the end-vehicle velocity sampling range is set as follows:

[0071] In the formula, It is the current speed, which is obtained by the difference between historical trajectory points; t pred For prediction in the time domain; a max,brake and a max,acc These represent the maximum deceleration and maximum acceleration, respectively. Then, the end velocity is sampled at a resolution of 1 km / h, and the end lateral distance is sampled at a resolution of 0.5 m. By combining longitudinal and lateral measurements, the driving area of ​​the vehicle along the target lane is sampled, and a second path polynomial curve is generated.

[0072] 4) Perform curvature detection on the second path polynomial curve and eliminate unreasonable candidate trajectories with excessive curvature. The curvature of the trajectory cluster can be calculated using the circumcircle curvature method of triangles. Finally, based on the calculation results, select path polynomial curves that meet the preset requirements to obtain the initial predicted trajectory cluster of the vehicle.

[0073] Furthermore, the trajectory evaluation output sub-model is used to construct a vehicle trajectory cluster based on the initial predicted vehicle trajectory cluster and evaluate the vehicle trajectory cluster based on spatiotemporal interaction features, calculate the probability distribution of the vehicle trajectory cluster, and obtain the predicted trajectory, including:

[0074] Vehicle trajectory clusters are constructed by unifying heterogeneous information from vehicle trajectory clusters and vehicle historical trajectories.

[0075] The vehicle trajectory clusters and spatiotemporal interaction features are integrated and encoded through a cross-attention mechanism, and the temporal motion features of the vehicle trajectory clusters are captured by a preset Transformer self-attention model. Finally, the probability distribution of the vehicle trajectory clusters is output through an MLP multilayer perceptron and a Softmax layer.

[0076] Predicted trajectories are obtained based on the probability distribution of vehicle trajectory clusters.

[0077] It's important to note that on structured roads, human drivers first determine which lane surrounding vehicles are in, and then, based on this, assess their relative positions to their own vehicle. Therefore, lane guidance information is crucial for trajectory prediction. However, encoding multi-source heterogeneous information, such as lane markings and the historical trajectories of adjacent vehicles, and using neural networks to learn their interrelationships presents a significant challenge. This is because lane markings are a series of waypoints, and unlike the historical trajectory information of adjacent vehicles, the waypoint sequence does not implicitly contain speed or acceleration features. Furthermore, in general, there is more than one lane surrounding the target vehicle, which greatly hinders the network model from learning the relationship between the vehicle's historical trajectory and the correct lane during training.

[0078] To address the aforementioned technical issues, this solution concatenates the historical trajectory of the target vehicle (i.e., the vehicle itself) with each trajectory from its initial predicted trajectory cluster, constructing a complete vehicle trajectory cluster from the starting point of the historical trajectory to the ending point of the predicted trajectory. This vehicle trajectory cluster itself contains motion trajectory information including velocity and acceleration. Furthermore, when generating the initial predicted trajectory cluster, the road topology is also considered, reflecting the changing trends of lane direction and position. Therefore, by concatenating the historical trajectory with the initial predicted trajectory cluster, a unified representation of the heterogeneous information of static map information and dynamic vehicle motion trajectory is achieved.

[0079] Furthermore, the self-attention mechanism of the Transformer can be used to learn the continuous motion features from the vehicle's historical trajectory to the predicted vehicle trajectory cluster. The complete vehicle trajectory cluster constructed above is represented by x and y coordinates in a Cartesian coordinate system. In scenarios guided by the lane centerline, transforming the x and y coordinates to Frenet coordinates s and d based on the lane centerline yields a more direct and essential relative positional relationship, thereby accelerating the training of the neural network. However, due to special scenarios such as curves, the distance s projected by the x and y trajectories along the lane centerline is not linearly related to the vehicle speed. Therefore, the trajectory feature vector designed in this scheme uses the more essential vehicle speed v and the distance d perpendicular to the lane centerline to characterize the vehicle's forward and backward motion trend.

[0080] Furthermore, by unifying the representation of road topology information and vehicle trajectory information, the temporally complete predicted vehicle trajectory cluster includes both the vehicle's historical trajectory and the initial predicted trajectory cluster. Therefore, by inputting each temporally complete trajectory, the motion features of the trajectory points before and after the trajectory sequence can be encoded through a self-attention mechanism. This allows for the identification of the best candidate trajectory with the most matching motion trends, thus enabling the construction of a trajectory evaluation output sub-model to evaluate the predicted vehicle trajectory cluster and obtain the probability distribution and predicted trajectory of the final predicted vehicle trajectory cluster.

[0081] The aforementioned scheme can predict the future trajectories of surrounding vehicles. It considers road topology and vehicle kinematic constraints, ultimately obtaining a multimodal predicted trajectory cluster and its probability distribution that fully characterizes the uncertain behavior of the predicted vehicle. Furthermore, the generated vehicle trajectory cluster revolves around the vehicle's most probable predicted trajectory, covering almost all possible vehicle movement intentions, and generating a corresponding probability for each trajectory in the predicted trajectory cluster. This offers the advantage of accurate long-term multimodal trajectory prediction, providing more accurate information on the movement of surrounding obstacles for subsequent decision-making and planning stages, enabling more reliable collision risk assessment, and thus generating safe and intelligent driving decisions, ultimately improving the safety and ride comfort of intelligent driving. Attached Figure Description

[0082] Figure 1 is a schematic flowchart of a trajectory prediction method based on spatiotemporal interactive feature fusion according to an embodiment of the present invention;

[0083] Figure 2 is an example of a spatiotemporal diagram provided in an embodiment of the present invention;

[0084] Figure 3 is a flowchart of a multimodal trajectory prediction method that considers road topology and vehicle kinematic constraints according to an embodiment of the present invention.

[0085] Figure 4 is a schematic diagram of an encoder-decoder structure of an LSTM neural network provided in an embodiment of the present invention;

[0086] Figure 5 is a schematic diagram of a target lane prediction sub-model provided in an embodiment of the present invention;

[0087] Figure 6 is a schematic diagram of a vehicle initial prediction trajectory cluster provided in an embodiment of the present invention;

[0088] Figure 7 is a schematic diagram of a triangle circumcircle curvature method according to an embodiment of the present invention;

[0089] Figure 8 is a schematic diagram of a method for splicing historical trajectory and initial vehicle trajectory according to an embodiment of the present invention;

[0090] Figure 9 is a schematic diagram of a trajectory evaluation output sub-model architecture provided by an embodiment of the present invention. Detailed Implementation

[0091] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0092] In embodiments of the present invention, the term "multiple" refers to two or more.

[0093] Please refer to Figure 1. This embodiment provides a trajectory prediction method based on spatiotemporal interaction feature fusion, including the following steps:

[0094] S1: Real-time collection of information on traffic participants on the road;

[0095] In this embodiment, traffic participant information includes category information such as vehicles, bicycles, and pedestrians, as well as other basic information, where category information is one dimension of the state variable. When subsequently solving the adjacency matrix, different adjacency matrices are not explicitly and specifically generated for different categories. Instead, the impact of potential collision risks is indirectly considered or learned in a general and unified manner within the potential collision risk and adaptive adjacency matrix.

[0096] S2: Construct a traffic scenario representation model based on traffic participant information;

[0097] S3: Construct a multi-scale spatiotemporal graph convolutional model so that the multi-scale spatiotemporal graph convolutional model can extract the spatiotemporal interaction features of traffic participants based on the traffic scene representation model;

[0098] S4: Construct a trajectory prediction model to predict trajectories based on spatiotemporal interaction features and obtain the predicted trajectory.

[0099] In practice, the trajectory prediction method based on spatiotemporal interactive feature fusion in this embodiment can be executed by the vehicle controller, such as the vehicle's domain controller.

[0100] In one embodiment, after step S4, the method may further include:

[0101] Based on the predicted trajectory, a collision risk assessment is performed to obtain the collision risk assessment result.

[0102] Based on the collision risk assessment results, a driving decision is generated;

[0103] Based on the driving decisions, the vehicle's movement is controlled.

[0104] This embodiment constructs a traffic scene representation and extracts the spatiotemporal interaction features of traffic participants, enabling the final vehicle prediction trajectory to capture the spatial correlation of traffic participant behavior at different time scales. The extraction of multi-scale spatiotemporal fusion interaction features effectively improves the accuracy of trajectory prediction. Based on accurate trajectory prediction of the future behavior of surrounding traffic participants, intelligent driving vehicles can perform more accurate collision risk assessments, thereby generating safer and more intelligent driving decisions, ultimately enhancing the safety and comfort of intelligent driving.

[0105] It should be noted that the traffic scene representation model represents the traffic scene as a spatiotemporal graph in graph theory. Nodes in the graph represent traffic participants, and edges represent the interaction relationships between two traffic participants. This spatiotemporal graph can then be mathematically represented as an adjacency matrix. The following examples consider Euclidean distance, potential collisions, and adaptation, constructing three adjacency matrices: Euclidean distance adjacency matrix, potential collision adjacency matrix, and adaptive adjacency matrix. Combining these three adjacency matrices can collectively express the true relationships between traffic participants in a traffic scene.

[0106] In one embodiment, the traffic scene representation model includes an Euclidean distance adjacency matrix, a potential collision adjacency matrix, and an adaptive adjacency matrix. Referring to Figure 2, to model the dynamic interaction relationships among traffic participants, the traffic scene in Figure 2a is represented as the spatiotemporal graph G shown in Figures 2b and 2c. t ={V t E t}. Among them, G t This is a spatiotemporal graph representing the influence of traffic behavior at time t, with traffic participants as nodes. Nodes in the graph represent traffic participants, and edges represent the interactions between two traffic participants. The node set is V. t Each node in the network corresponds to a traffic participant, E t It is a set of edges representing the relationship between two traffic participants at time t. The spatiotemporal graph structure can then be mathematically represented using an adjacency matrix, specifically:

[0107] The traffic scene representation model constructed based on traffic participant information includes: constructing an Euclidean distance adjacency matrix based on traffic participant information, as shown in Figure 2b. Figure 2b depicts the Euclidean distance adjacency matrix scene representation method, which uses distance to represent and abstract traffic scenes. Each solid circle represents a traffic participant node. The Euclidean distance adjacency matrix can be represented as:

[0108] In the formula, It is an Euclidean distance adjacency matrix An element in the equation represents the adjacency relationship between traffic participant i and traffic participant j at time t; D represents the distance between traffic participant i and traffic participant j at time t; close The distance threshold between neighboring participants is represented; 1 indicates that traffic participant i and traffic participant j are neighboring participants at time t; 0 indicates that traffic participant i and traffic participant j are not neighboring participants at time t.

[0109] It should be noted that once a traffic participant and a vehicle are identified as neighboring participants, the traffic participant's behavior will be considered to influence the vehicle's trajectory in subsequent prediction phases. As shown in Figure 2b, from the perspective of the red target vehicle (i.e., the vehicle in the very center of the circle), the radius is D. close All traffic participants within the blue circle are considered neighbors of the red target vehicle. However, in real-world traffic, the behavior of traffic participants outside the blue circle still affects the red target vehicle. For example, in the diagram, two pedestrians are crossing the road in front of the red target vehicle. For a human driver, the red target vehicle would typically assess the risk of colliding with both pedestrians and eventually slow down to avoid the closer pedestrian. However, because both pedestrians are outside the blue circle, this significantly influential pedestrian is not considered in the Euclidean distance adjacency matrix.

[0110] Therefore, to address the issue that the fixed distance threshold of the Euclidean distance adjacency matrix cannot be dynamically adjusted, this embodiment proposes an adjacency matrix that considers potential collision risks. Adjacency matrix is ​​used to account for the probability of future collisions. A simplified collision risk assessment algorithm is used to identify traffic participants with mutual influence. Figure 2c continues to illustrate the core idea of ​​this collision assessment algorithm using the example of pedestrians crossing the road. Because the future trajectory is unknown during the trajectory prediction stage, this embodiment uses a directed line segment generated based on the current speed and direction of the traffic participants as the future trajectory for collision detection. Therefore, directed line segments AB, CD, and EF represent the future trajectories of the vehicle and the two pedestrians in Figures 2a and 2b, respectively, and are calculated using a constant velocity kinematics model, specifically as follows:

[0111] In the formula, L pred Indicates the length of a directed line segment; u cur The current speed of the traffic participant; t pred For the prediction time domain; θ represents the direction of the future trajectory; θ curLet be the current direction of the traffic participant; it is simplified to always remain consistent with the current direction. This model provides a simple and fast way to assess collision risk; that is, if line segments AB and CD intersect at a point, a collision risk is considered to exist between the vehicle and the pedestrian. Clearly, when a collision is possible between two participants, it is necessary to model their behavioral interactions, as the probability of a collision will force them to change their original movement patterns. Therefore, a potential collision adjacency matrix is ​​constructed based on the future trajectory model of the traffic participants, specifically represented as:

[0112] In the formula, It is a potential collision adjacency matrix An element in the equation represents the potential collision relationship between traffic participant i and traffic participant j at time t. In this calculation, traffic participant i is the vehicle itself; AB is the future trajectory of the vehicle calculated based on the future trajectory model of the traffic participants; CD is the future trajectory of the current traffic participant calculated based on the future trajectory model of the traffic participants; if there is an intersection point (x, y), it means that the future trajectory of the vehicle and the future trajectory of the current traffic participant overlap, and there is a potential collision relationship. The value is assigned to 1.

[0113] Building upon the Euclidean distance adjacency matrix and potential collision adjacency matrix constructed above, this embodiment constructs an adaptive adjacency matrix to further uncover hidden behavioral interaction relationships. To model the spatial dependencies and interactions among traffic participants that are not easily perceived. Adaptive adjacency matrix. The main idea is to embed the source node into E i and target node embedding E j Multiplication is used to construct a dynamic, learnable adjacency matrix. Specifically, an adaptive adjacency matrix can be constructed based on traffic participant information, as follows:

[0114] In the formula, It is a potential adaptive adjacency matrix An element in E represents the adaptive relationship between traffic participant i and traffic participant j at time t; i This indicates that traffic participant i is the source node, and E j This indicates that traffic participant j is the target node; ReLU is the activation function, which is used to eliminate weak connections and increase the nonlinearity of the network layers; SoftMax is the normalization function used to normalize the adaptive adjacency matrix; the adaptive adjacency matrix... Used to capture the dependency between the source node and the target node embedding, that is, to represent the interaction between traffic participant i and traffic participant j.

[0115] It should be noted that node embedding maps a node to a multidimensional vector and uses this multidimensional vector to express, or more simply, to replace the node as input to subsequent neural network layers. Both of these node embeddings are randomly initialized with a set of learnable parameters.

[0116] It should be noted that, based on the traffic scene representation model, multiple dilated TCN-GCN spatiotemporal layers can be stacked to capture the spatial correlation of traffic participant behavior at different time scales, achieving multi-scale spatiotemporal fusion interaction feature extraction. Thanks to the receptive fields at different scales, each spatiotemporal layer of the GCN can capture the interaction relationships between traffic participants at different time scales. More specifically, at the bottom layer, the GCN encodes short-term behavioral interaction information, while at the top layer, the GCN understands the interaction relationships from a longer-term time perspective. However, as the number of stacked layers increases, the number of model parameters and the computational burden increase significantly. To address this issue, this embodiment employs a dilated TCN temporal convolutional network when stacking multiple spatiotemporal layers, and improves upon it to achieve a balance between prediction accuracy and computational efficiency. Specifically:

[0117] The construction of a multi-scale spatiotemporal graph convolutional model, which extracts spatiotemporal interaction features of traffic participants based on a traffic scene representation model, includes:

[0118] A multi-scale spatiotemporal graph convolutional model consisting of multiple stacked dilated TCN-GCN spatiotemporal layers is constructed. In this model, the GCN in each spatiotemporal layer is used to capture the interaction relationships between traffic participants at different time scales, and the dilated TCN in each spatiotemporal layer is used to capture the movement trends of traffic participants over time.

[0119] The multi-scale spatiotemporal graph convolutional model extracts the spatiotemporal interaction features of traffic participants based on the traffic scene representation model;

[0120] Wherein: Dilated TCN represents a dilated temporal convolutional network; GCN represents a graph convolutional network.

[0121] The dilated TCN constructed in this embodiment achieves the best balance between prediction performance and computational efficiency, providing feasibility for the algorithm to be deployed on low-computing-power domain controller chips. Furthermore, it constructs a method to represent the traffic scene expression model using three adjacency matrices: Euclidean distance adjacency matrix, potential collision adjacency matrix, and adaptive adjacency matrix. This is beneficial for fully modeling prior knowledge and thus optimizing the performance of the GCN graph convolutional layer.

[0122] In one embodiment, in the multi-scale spatiotemporal graph convolution model, for any dilated TCN, its dilation convolution operation F(t) at time t is specifically expressed as:

[0123] In the formula: Q represents a one-dimensional input sequence; f represents a filter; a is the element index in the filter; k1 is the size of the filter; d represents the dilation factor, which is used to control the distance between convolution kernel elements; in the multi-scale spatiotemporal graph convolution model, the d value of the odd-dilated TCN-GCN spatiotemporal layer is 1, and the d value of the even-dilated TCN-GCN spatiotemporal layer is 2.

[0124] In this embodiment, to extract the temporal correlation of behavioral interactions among traffic participants, the present invention employs a temporal convolutional (TCN) network to process time-series information. Compared to recurrent neural networks (RNNs), TCNs, based on convolutional neural networks, offer advantages such as parallel computation and gradient stability, enabling the extraction of feature information over longer time scales. This invention uses stacked dilated temporal convolutional layers with different receptive fields as the TCN model to capture the temporal movement trends of traffic participants.

[0125] Furthermore, as shown in Figure 3, this embodiment demonstrates three different implementations of dilated convolution. The stacked dilated TCN method shown in Figure 3a involves designing a dilation factor sequence of 1,2; 1,2; 1,2; 1,2, and then applying it to the dilation parameter of the Conv1d convolution function in each layer (referred to as G-WaveNet TCN in this embodiment). If this method is directly applied to trajectory prediction, the time consumption remains high. Conversely, the exponentially dilated temporal convolution shown in Figure 3b significantly accelerates the computation speed but excessively increases the loss of local information, leading to a decrease in prediction accuracy. To address this dilemma, this embodiment explores a new implementation method for stacked dilated TCN layers in Figure 3c, which assigns 1 or 2 to the stride parameter of the Conv1d function sequentially according to the odd and even number of layers to implement the dilated TCN layers.

[0126] In one embodiment, to extract the mutual influence of the behaviors of spatially adjacent traffic participants, this embodiment further employs a graph convolutional network to extract the dependencies between nodes in the adjacency matrix, based on three adjacency matrices—Euclidean distance adjacency matrix, potential collision adjacency matrix, and adaptive adjacency matrix—to represent the traffic scene. Graph convolution has a significant advantage in extracting information about surrounding traffic participants due to its ability to aggregate and transform the adjacency information of nodes. By designing the adjacency matrix, prior knowledge can be integrated into the prediction, thereby improving prediction accuracy. After Chebyshev spectral filtering can be simplified using a first-order approximation, standard convolution can be used to process graph-structured data. Specifically, the spatiotemporal interaction feature Z output by any GCN in the multi-scale spatiotemporal graph convolutional model is expressed as:

[0127] In the formula, k2 represents the diffusion step; This represents the power series of the Euclidean distance adjacency matrix; Represents the power series of the potential collision adjacency matrix; H represents the power series of the adaptive adjacency matrix; H is the output of the dilated TCN. This is the learnable parameter matrix in the convolution operation; Indicates the last observation timestamp t obs The Euclidean distance adjacency matrix; Indicates the last observation timestamp t obs The potential collision adjacency matrix; Indicates the last observation timestamp t obs The adaptive adjacency matrix; the Euclidean distance adjacency matrix, the potential collision adjacency matrix, and the adaptive adjacency matrix constitute the traffic scene representation model; rowsum is the row summation function, that is, summing each row of the matrix input to the function.

[0128] It's important to note that, compared to standard convolution operations, graph convolution is essentially still a convolution operation implemented through a two-dimensional Conv2D convolutional layer. The key difference lies in the processing of the input signal, which involves multiplying by... The method effectively absorbs the structural information of the graph, allowing expert prior knowledge to be integrated into the prediction model through adjacency matrix design. Thus, a GCN layer and an inflated TCN layer together constitute a temporal-spatial layer. Then, by stacking multiple TCN-GCN spatiotemporal layers, the spatial correlation of traffic participant behavior at different time scales is captured, achieving multi-scale spatiotemporal fusion interaction feature extraction. Based on this, this embodiment can simultaneously capture long-term and short-term behavioral interaction patterns. For example, in a scenario where a pedestrian is crossing the lane ahead, the target vehicle will generally make a decision on a more macroscopic time scale and will not abandon yielding to the pedestrian due to a temporary pause while crossing the road. The GCN can be combined with the top TCN layer to capture this long-term interaction. In a following scenario, the following vehicle may maintain a constant relative speed and distance to the vehicle in front for a short period. This invention can also learn this short-term behavioral interaction through the cooperation of the bottom TCN layer and the corresponding GCN layer.

[0129] In one embodiment, constructing a trajectory prediction model to predict trajectories based on spatiotemporal interaction features and obtaining predicted trajectories includes:

[0130] Construct a trajectory prediction model with an encoder-decoder structure; wherein both the encoder and the decoder contain LSTM neural networks;

[0131] The spatiotemporal interaction features are input into the encoder, and after being learned and processed by the LSTM neural network, the contextual feature information is obtained.

[0132] The contextual feature information is input into the decoder, processed and learned by the LSTM neural network, and combined with the vehicle's current position to obtain multiple sequence points;

[0133] Trajectory prediction is performed based on multiple sequence points to obtain the predicted trajectory.

[0134] In this embodiment, an encoder-decoder structure with an LSTM neural network can output a sequence of trajectory points, thereby obtaining a predicted trajectory. As shown in Figure 4, the encoder-decoder structure consists of two main parts: an encoder and a decoder. The core network layers of both the encoder and decoder are two-layer LSTM neural networks. First, the spatiotemporal interaction features are input into the encoder, processed by the encoder, and transformed into contextual information, which is then saved as h. t , there is: h t =LSTM(Z) t )

[0135] In the formula, Z t Representing spatiotemporal interaction features; the encoding process inside the encoder can be described by the following formula: i t =sigmoid(W i Z t +U i h t-1 +b i f t =sigmoid(W f Z t +U f h t-1 +b f o t =sigmoid(W o Z t +U o h t-1 +b o ) a t =tanh(W a Z t +U a h t-1 +b a ) c t =f t c t-1 +i t a t h t =o t tanh(c t )

[0136] Among them, i t ,f t ,o tThese are the input gate, forget gate, and output gate of an LSTM network, respectively. i W f W o and W a For the corresponding linear transformation weights of the network layer, sigmoid is the non-linear activation function, and c t The current state information, or memory, extracted for the LSTM network; C t-1 This indicates the status information of the previous step; a t Represents the candidate cell state, i.e., the new memory information generated from the current input and the previous hidden state; U i U f U o U a b is the corresponding hidden state weight matrix; i b f b o b a These are the bias vectors for the forget gate, input gate, output gate, and candidate cell state, respectively; h t-1 This refers to the context information from the previous step, i.e., the output of the LSTM unit from the previous step.

[0137] Next, the context information is fed into the decoder. After processing and learning by the LSTM network in the decoder, and combined with the current position [x0, y0] of traffic participant (e.g., target vehicle) i, a set of point sequences for the predicted trajectory can be iteratively generated according to the following formula.

[0138] ...

[0139] Among them, h 0 h represents the context information at the initial moment. 1 h represents the context information at t=1. t-1 h represents the context information at time t-1. t This represents the context information at time t.

[0140] Finally, the predicted trajectory is obtained based on multiple sequence points.

[0141] In one embodiment, constructing a trajectory prediction model to predict trajectories based on spatiotemporal interaction features and obtaining predicted trajectories includes:

[0142] A trajectory prediction model is constructed, comprising a target lane prediction sub-model, a trajectory cluster prediction sub-model, and a trajectory evaluation output sub-model; wherein:

[0143] The target lane prediction sub-model is used to obtain the vehicle's target lane based on pre-acquired vehicle historical trajectory data and lane data;

[0144] The trajectory cluster prediction sub-model is used to obtain the initial predicted trajectory cluster of the vehicle based on the pre-acquired historical trajectory data of the vehicle and the vehicle's target lane.

[0145] The trajectory evaluation output sub-model is used to construct a vehicle trajectory cluster based on the initial predicted trajectory cluster of the vehicle and evaluate the vehicle trajectory cluster based on spatiotemporal interaction features, calculate the probability distribution of the vehicle trajectory cluster, and obtain the predicted trajectory.

[0146] This embodiment uses the traffic participants surrounding the vehicle as examples to illustrate the concept. Accordingly, the predicted trajectory is the predicted trajectory of the vehicles surrounding the vehicle (i.e., the predicted trajectory of the vehicle being predicted). The trajectory prediction model constructed in this embodiment can generate a cluster of initial predicted vehicle trajectories that satisfy road topology and vehicle kinematic constraints. First, the relative relationship between the historical trajectory of the vehicle being predicted (i.e., the target vehicle) and the target lane of the target vehicle is utilized. Then, spatial sampling is performed on the target lane to generate a cluster of initial predicted vehicle trajectories that satisfies the road topology. Finally, the probability of each trajectory in the initial predicted trajectory cluster being a predicted trajectory is evaluated through the trajectory evaluation output sub-model. The most likely future trajectory can be obtained through its probability distribution. Simultaneously, the probability distribution also expresses the uncertainty of the behavior of other vehicles, which is a prerequisite for decision-making and planning to address uncertain behaviors of other vehicles.

[0147] In one embodiment, the target lane prediction sub-model is used to obtain the vehicle's target lane based on pre-acquired historical vehicle trajectory data and lane data, including:

[0148] Based on the pre-acquired historical vehicle trajectory data and lane data, the relative position of the vehicle and the candidate lane is obtained;

[0149] Based on the cross-attention mechanism and the relative position of the vehicle and the candidate lane, the correlation between the candidate lane centerline and the vehicle's historical trajectory is calculated.

[0150] The target lane of the vehicle is obtained based on the correlation between the candidate lane centerline and the vehicle's historical trajectory.

[0151] In this embodiment, directly sampling all lanes around the target vehicle without consideration and generating corresponding vehicle trajectory clusters would result in an excessively large number of vehicle trajectory clusters, increasing the complexity of the prediction model. Therefore, to reduce the sampling space of the predicted trajectory clusters, this embodiment utilizes a cross-attention mechanism to build a target lane prediction sub-model as shown in Figure 5. First, the target lane that the target vehicle is most likely to travel in is selected, and then vehicle trajectory clusters are generated based on this target lane.

[0152] In one embodiment, obtaining the target lane for a vehicle primarily relies on the relative positional relationship between the vehicle's historical trajectory and the centerlines of surrounding lanes, i.e., the relative position of the vehicle and the candidate lane. To characterize this relative positional relationship, the Frenet coordinates s and d of the vehicle's historical trajectory along the centerline of each candidate lane in the neighboring lane set are first calculated. Here, s represents the longitudinal distance traveled along the candidate lane centerline, and d represents the lateral distance perpendicular to the candidate lane centerline. For curved lanes, the Frenet coordinate system, compared to Cartesian coordinates, can more clearly and directly describe the relative positional relationship between the vehicle and the lane. After describing the relative positional relationship between the vehicle and the candidate lane using Frenet coordinates, this embodiment can utilize a cross-attention mechanism to handle the relationship between the sequence of the vehicle's historical trajectory and the sequence of path points on the candidate lane centerlines. Both cross-attention and self-attention mechanisms belong to attention networks, which model and capture the dependencies of the input sequence through (Q, K, V) triples. Depending on the different input sources of the three vectors Q, K, and V, in this embodiment, the candidate lane centerline can be used as the query vector Q for the cross-attention mechanism, and the vehicle historical trajectory can be used as the key vector K and the value vector V, thereby constructing the self-attention mechanism and the cross-attention mechanism.

[0153] In one embodiment, Q, K, and V in the self-attention mechanism are obtained by multiplying the same feature input vector X by three different weight matrices W. Q W K W V The generated vector has three identical dimensions, i.e., Q = XW. Q K = XW K V = XW V The weight matrix can be obtained through training a neural network. The essence of the self-attention mechanism is to calculate the mutual influence between elements within the input vector X, which can be implemented using a Transformer network. The query vector Q in the cross-attention mechanism is determined by the lane centerline X. A K and V are derived from the vehicle's historical trajectory X. B The transformed sequence is primarily used to model the relationship between two different sequences, i.e., Q = X. A W Q K = X B W K V = X B W V .

[0154] Therefore, in the target lane prediction sub-model, the lane centerline X A The query vector Q for the cross-attention mechanism, and the vehicle's historical trajectory X BAs the key vector K and value vector V, they are used together to perform cross-attention operations to obtain the correlation between the lane centerline and the vehicle's historical trajectory. The specific steps are as follows:

[0155] 1) Feature mapping: Q = X A W Q K = X B W K V = X B W V X A X is the center line of the lane. B For vehicle historical trajectory;

[0156] 2) Similarity Calculation: The similarity score between the query vector Q and the key vector K is calculated using the dot product of Q and K. These similarity scores represent the similarity between the lane centerline X and the key vector K. A Each element in the X-ray is related to the vehicle's historical trajectory. B The relationship between all elements in the table can be represented as:

[0157] Where S is the similarity score matrix, It is a scaling factor used to prevent the dot product value from becoming too large;

[0158] 3) Attention weight calculation: The similarity score is normalized using the Softmax function to obtain the attention weight, which reflects the lane centerline X. A Each element in the X-ray is related to the vehicle's historical trajectory. B The importance of all elements in the equation can be expressed as:

[0159] W a A→B =Softmax(S A→B )

[0160] Among them, W a It is attention weight; W a A→B Indicates the lane center line X A Each element in the X-ray is related to the vehicle's historical trajectory. B The importance of all elements in the process, i.e., attention weights.

[0161] 4) Weighted Summation: The attention weights are weighted and summed with the value vector V to obtain the output representation after cross-attention adjustment, which can be expressed as: O A→B =W a A→B V

[0162] In the formula, O A→B The dimension of O is consistent with the dimension of the query vector Q. Through the aforementioned attention mechanism, O A→BEach element in the model that corresponds one-to-one with the query vector Q contains the association information between the key vector K and the value vector V, thus modeling the lane centerline X. A and vehicle historical trajectory X B The interrelationship between them.

[0163] In one embodiment, after encoding the lane centerline and the vehicle's historical trajectory to obtain their interrelationship, all candidate lanes can be constructed as an [n,c]-dimensional vector: each candidate lane is represented by a c-dimensional vector, and this is sequentially increased to n lanes to form an [n,c]-dimensional lane array. Here, n is the number of candidate lanes, and c is the feature dimension of the lane. Next, a Transformer self-attention network is used to learn the relationships and differences between the candidate lanes, and a Multilayer Perceptron (MLP) and Softmax are used to output the probability distribution of the candidate lanes. Finally, considering possible prediction errors, this embodiment achieves a balance between sampling efficiency and prediction accuracy, selecting the top 3 lanes with the highest probabilities as the predicted target lanes for the vehicle.

[0164] In one embodiment, the trajectory cluster prediction sub-model is used to obtain an initial predicted trajectory cluster for the vehicle based on pre-acquired historical vehicle trajectory data and the vehicle's target lane, including:

[0165] Construct the Frenet coordinate system for the lane centerline of the target lane of the vehicle;

[0166] In the Frenet coordinate system of the lane centerline, spatial sampling of the vehicle target lane is performed based on the local path planning method to obtain the first path polynomial curve;

[0167] Transform the first path polynomial curve into Cartesian coordinates to obtain the second path polynomial curve.

[0168] Curvature detection is performed on the second path polynomial curve, and path polynomial curves that meet the preset requirements are selected to obtain the initial predicted trajectory cluster of the vehicle.

[0169] It should be noted that, in order to generate a predicted trajectory cluster that conforms to the road topology and vehicle kinematics, the concept of local path planning is applied to spatially sample the target lane, and the initial predicted trajectory cluster of the vehicle is expressed using a polynomial curve, which may include the following steps:

[0170] 1) Sampling is performed in the Frenet coordinate system along the lane centerline, where s represents the longitudinal distance traveled along the lane centerline and d represents the lateral distance (i.e., side distance) perpendicular to the lane centerline. The target lane centerline is fitted as a function of parameter s using a cubic spline curve:

[0171] In the formula, a x ,bx ,c x , a y ,b y ,c y , Here, x and y represent the spline curve coefficients, and x and y are the values ​​of the lane centerline point in the Cartesian coordinate system. Parameterizing the lane centerline facilitates subsequent reconstruction of the initial predicted vehicle trajectory cluster to the Cartesian coordinate system via s-interpolation.

[0172] 2) Construct the longitudinal distance s and lateral distance d of the initial predicted trajectory cluster of the vehicle in Frenet coordinates as fourth-order and fifth-order polynomials respectively for the prediction duration t:

[0173] In the formula, for a fourth-degree polynomial, the initial states can be solved simultaneously. and terminal state Five equations are established for solving, where S0 in the above states represents the longitudinal distance at the initial moment. The first derivative of the longitudinal distance at the initial moment (initial longitudinal velocity) is given. The second derivative of the longitudinal distance at the initial moment (initial longitudinal acceleration) is given. This represents the first derivative (longitudinal velocity at the end of the trajectory) at time t. This represents the second derivative (terminal longitudinal acceleration) at time t, the endpoint of the trajectory. Similarly, the coefficients of the fifth-order polynomial for the lateral distance d are obtained by combining the initial state... and terminal state Six equations are established for solving, where d0 represents the lateral distance at the initial time. The first derivative of the lateral distance at the initial moment (initial lateral velocity) is given. d represents the second derivative of the lateral distance at the initial moment (initial lateral acceleration). t This represents the lateral distance at time t at the end of the trajectory. The first derivative of the lateral distance at time t at the end of the trajectory (initial lateral velocity) represents the initial lateral velocity. The second derivative of the lateral distance at time t at the end of the trajectory (initial lateral acceleration), a4, a3, a2, a1, a0, b5, b4, b3, b2, b1 and b0 represent the coefficients of the polynomial, that is, the unknowns to be solved in the polynomial equation system.

[0174] 3) Sample different terminal states S t and terminal state D tBy combining these parameters, a series of first path polynomial curves in Frenet coordinates can be generated. Then, by interpolating using the lane centerline parameterized in step 1), the curves are restored to Cartesian coordinates to obtain the second path polynomial curves. In this embodiment, the sampling range for the end-velocity longitudinal velocity is set as follows:

[0175] In the formula, It is the current velocity (i.e., the initial longitudinal velocity), which is obtained by the difference between historical trajectory points; t pred For prediction in the time domain; a max,brake and a max,acc These represent the maximum deceleration and maximum acceleration, respectively. Then, the end velocity is sampled at a resolution of 1 km / h, and the end lateral distance is sampled at a resolution of 0.5 m. By combining longitudinal and lateral measurements, the driving area of ​​the vehicle along the target lane is sampled, and a second path polynomial curve is generated.

[0176] 4) Perform curvature detection on the second path polynomial curve and eliminate unreasonable candidate trajectories with excessive curvature. The curvature of the trajectory cluster can be calculated using the circumcircle curvature method of triangles. Finally, based on the calculation results, select path polynomial curves that meet the preset requirements to obtain the initial predicted trajectory cluster of the vehicle.

[0177] See Figure 6 for details, where the light green trajectory cluster represents feasible trajectories that meet the curvature constraints, and the blue trajectory cluster represents infeasible trajectories. The curvature calculation using the circumcircle curvature method is performed as follows:

[0178] Please refer to Figure 7. First, assume that A, B, and C are three consecutive discrete points on the predicted trajectory, and a, b, and c are their opposite sides. In ΔABC, by the Law of Cosines, we can obtain:

[0179] Then, the curvature k can be obtained from the law of sines:

[0180] Where R represents the radius of the circumcircle of ΔABC.

[0181] In one embodiment, the trajectory evaluation output sub-model is used to construct a vehicle trajectory cluster based on the initial predicted vehicle trajectory cluster and evaluate the vehicle trajectory cluster based on spatiotemporal interaction features, calculate the probability distribution of the vehicle trajectory cluster, and obtain the predicted trajectory, including:

[0182] The vehicle trajectory cluster is constructed by unifying the heterogeneous information of the initial predicted trajectory cluster and the historical trajectory of the vehicle.

[0183] The vehicle trajectory clusters and spatiotemporal interaction features are integrated and encoded through a cross-attention mechanism, and the temporal motion features of the vehicle trajectory clusters are captured by a preset Transformer self-attention model. Finally, the probability distribution of the vehicle trajectory clusters is output through an MLP multilayer perceptron and a Softmax layer.

[0184] Predicted trajectories are obtained based on the probability distribution of vehicle trajectory clusters.

[0185] It's important to note that on structured roads, human drivers first determine which lane surrounding vehicles are in, and then, based on this, assess their relative positions to their own vehicle. Therefore, lane guidance information is crucial for trajectory prediction. However, encoding multi-source heterogeneous information, such as lane markings and the historical trajectories of adjacent vehicles, and using neural networks to learn their interrelationships presents a significant challenge. This is because lane markings are a series of waypoints, and unlike the historical trajectory information of adjacent vehicles, the waypoint sequence does not implicitly contain speed or acceleration features. Furthermore, in general, there is more than one lane surrounding the target vehicle, which greatly hinders the network model from learning the relationship between the vehicle's historical trajectory and the correct lane during training.

[0186] To address the aforementioned technical issues, this embodiment stitches together the historical trajectory of the vehicle to be predicted with each predicted trajectory (i.e., all predicted trajectories) in its initial predicted trajectory cluster to construct a complete vehicle trajectory cluster from the starting point of the historical trajectory to the ending point of the predicted trajectory. The vehicle trajectory cluster itself is a motion trajectory containing speed and acceleration information. Furthermore, when generating the initial predicted trajectory cluster, the road topology is also considered, reflecting the changing trends of lane direction and position. Therefore, the historical trajectory of the vehicle to be predicted and its initial predicted trajectory cluster are stitched and recombined according to Figure 8, achieving a unified representation of the heterogeneous information of static map information and dynamic vehicle motion trajectories. In Figure 8, P0-P4 represent the historical trajectory of the vehicle to be predicted, and P5-P19 represent the initial predicted trajectory cluster of the vehicle to be predicted.

[0187] Furthermore, the self-attention mechanism of the Transformer can be used to learn the continuous motion features from the vehicle's historical trajectory to the predicted vehicle trajectory cluster. The complete vehicle trajectory cluster constructed above is the x, y coordinate in the Cartesian coordinate system. In a scenario guided by the lane centerline, transforming the x, y coordinates to Frenet coordinates s and d based on the lane centerline can obtain a more direct and essential relative positional relationship, thereby accelerating the training of the neural network. However, due to special scenarios such as curves, the distance s projected by the xy trajectory along the lane centerline is not linearly related to the vehicle speed. Therefore, the trajectory feature vector designed in this embodiment uses the more essential vehicle speed v and the distance d in the direction perpendicular to the lane centerline to characterize the vehicle's forward and backward motion trend. According to Figure 8, each candidate predicted trajectory is directly concatenated after the historical trajectory. Historical trajectory dimension: [3,5], where 3 represents the feature vector [x,y,v]; predicted trajectory dimension: [3,15]; concatenated complete trajectory dimension: [3,20]. After concatenation, the feature vector [x,y,v] will be further converted into the feature vector [v,d], and the dimension of the concatenated trajectory will be [2,20].

[0188] Furthermore, by unifying the representation of road topology information and vehicle trajectory information, the temporally complete predicted vehicle trajectory cluster includes both the vehicle's historical trajectory and the initial predicted trajectory cluster. Therefore, by inputting each temporally complete trajectory, the motion features of the trajectory points before and after the trajectory sequence can be encoded through a self-attention mechanism. This allows for the identification of the best candidate trajectory with the most matching motion trends, thus enabling the construction of a trajectory evaluation output sub-model to evaluate the predicted vehicle trajectory cluster and obtain the final probability distribution and predicted trajectory of the vehicle trajectory cluster.

[0189] Furthermore, following the above ideas, the trajectory evaluation output sub-model can be constructed as the evaluation model shown in Figure 9, which specifically includes three parts: behavioral interaction feature fusion encoding, predicted trajectory cluster self-attention feature extraction, and probability distribution decoding. Specifically: First, after extracting the multi-scale spatiotemporal interaction features of traffic participants, the predicted vehicle trajectory cluster and spatiotemporal interaction features are integrated and encoded through Cross-Attention, enabling the vehicle trajectory cluster to pay attention to the behavioral interaction information of traffic participants. Here, the predicted trajectory cluster serves as the query Q vector of the Cross-Attention module, and the spatiotemporal interaction features serve as the key vector K and value vector V of the Cross-Attention module. The output of the Cross-Attention module maintains the same dimension as the predicted vehicle trajectory cluster. Then, a standard Transformer self-attention network is used to capture the temporal motion features of the vehicle trajectory cluster, including feature [v,d], i.e., vehicle speed v and distance d in the direction perpendicular to the lane centerline. Additionally, the acceleration and the rate of change of distance d can be indirectly learned. Finally, the probability distribution evaluation of the predicted vehicle trajectory clusters is modeled as a classification problem. The probability distribution of the predicted vehicle trajectory clusters is output through a multilayer perceptron (MLP) and a softmax layer, ultimately yielding a complete multimodal predicted vehicle trajectory cluster and its probability distribution that fully represents the uncertain behavior of the predicted vehicle. The softmax normalized exponential function, a commonly used classifier in deep neural networks, outputs the probabilities corresponding to different trajectories.

[0190] This embodiment can predict the future trajectories of surrounding vehicles, taking into account road topology and vehicle kinematic constraints. It ultimately yields a multimodal predicted trajectory cluster and its probability distribution that fully characterizes the uncertain behavior of the predicted vehicle. The generated vehicle trajectory cluster revolves around the vehicle's most probable predicted trajectory, covering almost all possible vehicle movement intentions. A corresponding probability is generated for each trajectory in the predicted trajectory cluster, offering the advantage of accurate long-term multimodal trajectory prediction. This provides more accurate information on the movement of surrounding obstacles for subsequent decision-making and planning stages, enabling more reliable collision risk assessments. Ultimately, this generates safe and intelligent driving decisions, improving the safety and ride comfort of intelligent driving.

[0191] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A trajectory prediction method based on spatiotemporal interaction feature fusion, characterized in that, Includes the following steps: Real-time collection of information on traffic participants on the road; Construct a traffic scenario representation model based on traffic participant information; Construct a multi-scale spatiotemporal graph convolutional model to extract the spatiotemporal interaction features of traffic participants based on the traffic scene representation model; A trajectory prediction model is constructed to predict trajectories based on spatiotemporal interaction features and obtain the predicted trajectories.

2. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 1, characterized in that, The traffic scene representation model includes an Euclidean distance adjacency matrix, a potential collision adjacency matrix, and an adaptive adjacency matrix; the construction of the traffic scene representation model based on traffic participant information includes: Based on traffic participant information, a Euclidean distance adjacency matrix is ​​constructed, specifically represented as follows: In the formula, It is an Euclidean distance adjacency matrix An element in the equation represents the adjacency relationship between traffic participant i and traffic participant j at time t; D represents the distance between traffic participant i and traffic participant j at time t; close This represents the distance threshold between neighboring participants; 1 indicates that traffic participant i and traffic participant j are neighboring participants at time t; 0 indicates that traffic participant i and traffic participant j are not neighboring participants at time t. Based on traffic participant information, the current speed and direction of each participant are obtained, resulting in multiple directed line segments. A future trajectory model of the traffic participants is then constructed based on these directed line segments, specifically as follows: In the formula, L pred Indicates the length of a directed line segment; u cur The current speed of the traffic participant; t pred For the prediction time domain; θ represents the direction of the future trajectory; θ cur Current direction for traffic participants; The potential collision adjacency matrix is ​​constructed based on the future trajectory model of traffic participants, and is specifically represented as follows: In the formula, It is a potential collision adjacency matrix An element in the equation represents the potential collision relationship between traffic participant i and traffic participant j at time t. In this calculation, traffic participant i is the vehicle itself; AB is the future trajectory of the vehicle calculated based on the future trajectory model of the traffic participants; CD is the future trajectory of the current traffic participant calculated based on the future trajectory model of the traffic participants; if there is an intersection point (x, y), it means that the future trajectory of the vehicle and the future trajectory of the current traffic participant overlap, and there is a potential collision relationship. The value is assigned to 1; Based on traffic participant information, an adaptive adjacency matrix is ​​constructed, specifically represented as follows: In the formula, It is a potential adaptive adjacency matrix An element in E represents the adaptive relationship between traffic participant i and traffic participant j at time t; i This indicates that traffic participant i is the source node, and E j This indicates that traffic participant j is the target node; ReLU is the activation function; SoftMax is the normalization function; and the adaptive adjacency matrix is ​​used. Used to capture the dependency between the source node and the target node embedding, that is, to represent the interaction between traffic participant i and traffic participant j.

3. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 1, characterized in that, The construction of a multi-scale spatiotemporal graph convolutional model, which extracts spatiotemporal interaction features of traffic participants based on a traffic scene representation model, includes: A multi-scale spatiotemporal graph convolutional model consisting of multiple stacked dilated TCN-GCN spatiotemporal layers is constructed. In this model, the GCN in each spatiotemporal layer is used to capture the interaction relationships between traffic participants at different time scales, and the dilated TCN in each spatiotemporal layer is used to capture the movement trends of traffic participants over time. The multi-scale spatiotemporal graph convolutional model extracts the spatiotemporal interaction features of traffic participants based on the traffic scene representation model; Wherein: Dilated TCN represents a dilated temporal convolutional network; GCN represents a graph convolutional network.

4. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 3, characterized in that, In the multi-scale spatiotemporal graph convolution model, for any dilated TCN, the dilation convolution operation F(t) at time t is specifically expressed as: In the formula: Q represents a one-dimensional input sequence; f represents a filter; a is the element index in the filter; k1 is the size of the filter; d represents the dilation factor, which is used to control the distance between convolution kernel elements; in the multi-scale spatiotemporal graph convolution model, the d value of the odd-dilated TCN-GCN spatiotemporal layer is 1, and the d value of the even-dilated TCN-GCN spatiotemporal layer is 2.

5. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 3, characterized in that, In the multi-scale spatiotemporal graph convolution model, for any GCN, its output spatiotemporal interaction feature Z is specifically represented as: In the formula, k2 represents the diffusion step; This represents the power series of the Euclidean distance adjacency matrix; Represents the power series of the potential collision adjacency matrix; H represents the power series of the adaptive adjacency matrix; H is the output of the dilated TCN. This is the learnable parameter matrix in the convolution operation; Indicates the last observation timestamp t obs The Euclidean distance adjacency matrix; Indicates the last observation timestamp t obs The potential collision adjacency matrix; Indicates the last observation timestamp t obs The adaptive adjacency matrix; the Euclidean distance adjacency matrix, the potential collision adjacency matrix, and the adaptive adjacency matrix constitute the traffic scene representation model.

6. A trajectory prediction method based on spatiotemporal interactive feature fusion according to any one of claims 1 to 5, characterized in that, The construction of the trajectory prediction model, which enables the trajectory prediction model to predict trajectories based on spatiotemporal interaction features and obtain predicted trajectories, includes: Construct a trajectory prediction model with an encoder-decoder structure; wherein both the encoder and the decoder contain LSTM neural networks; The spatiotemporal interaction features are input into the encoder, and after being learned and processed by the LSTM neural network, the contextual feature information is obtained. The contextual feature information is input into the decoder, processed and learned by the LSTM neural network, and combined with the vehicle's current position to obtain multiple sequence points; Trajectory prediction is performed based on multiple sequence points to obtain the predicted trajectory.

7. A trajectory prediction method based on spatiotemporal interactive feature fusion according to any one of claims 1 to 5, characterized in that, The construction of the trajectory prediction model, which enables the trajectory prediction model to predict trajectories based on spatiotemporal interaction features and obtain predicted trajectories, includes: A trajectory prediction model is constructed, comprising a target lane prediction sub-model, a trajectory cluster prediction sub-model, and a trajectory evaluation output sub-model; wherein: The target lane prediction sub-model is used to obtain the vehicle's target lane based on pre-acquired vehicle historical trajectory data and lane data; The trajectory cluster prediction sub-model is used to obtain the initial predicted trajectory cluster of the vehicle based on the pre-acquired historical trajectory data of the vehicle and the vehicle's target lane. The trajectory evaluation output sub-model is used to construct a vehicle trajectory cluster based on the initial predicted trajectory cluster of the vehicle and evaluate the vehicle trajectory cluster based on spatiotemporal interaction features, calculate the probability distribution of the vehicle trajectory cluster, and obtain the predicted trajectory.

8. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 7, characterized in that, The target lane prediction sub-model is used to obtain the vehicle's target lane based on pre-acquired historical vehicle trajectory data and lane data, including: Based on the pre-acquired historical vehicle trajectory data and lane data, the relative position of the vehicle and the candidate lane is obtained; Based on the cross-attention mechanism and the relative position of the vehicle and the candidate lane, the correlation between the candidate lane centerline and the vehicle's historical trajectory is calculated. The target lane of the vehicle is obtained based on the correlation between the candidate lane centerline and the vehicle's historical trajectory.

9. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 7, characterized in that, The trajectory cluster prediction sub-model is used to obtain the initial predicted trajectory cluster of the vehicle based on pre-acquired historical trajectory data and the vehicle's target lane, including: Construct the Frenet coordinate system for the lane centerline of the target lane of the vehicle; In the Frenet coordinate system of the lane centerline, spatial sampling of the vehicle target lane is performed based on the local path planning method to obtain the first path polynomial curve; Transform the first path polynomial curve into Cartesian coordinates to obtain the second path polynomial curve. Curvature detection is performed on the second path polynomial curve, and path polynomial curves that meet the preset requirements are selected to obtain the initial predicted trajectory cluster of the vehicle.

10. The trajectory prediction method based on spatiotemporal interactive feature fusion according to claim 7, characterized in that, The trajectory evaluation output sub-model is used to construct a vehicle trajectory cluster based on the initial predicted trajectory cluster and evaluate the vehicle trajectory cluster based on spatiotemporal interaction features, calculate the probability distribution of the vehicle trajectory cluster, and obtain the predicted trajectory, including: The vehicle trajectory cluster is constructed by unifying the heterogeneous information of the initial predicted trajectory cluster and the historical trajectory of the vehicle. The vehicle trajectory clusters and spatiotemporal interaction features are integrated and encoded through a cross-attention mechanism, and the temporal motion features of the vehicle trajectory clusters are captured by a preset Transformer self-attention model. Finally, the probability distribution of the vehicle trajectory clusters is output through an MLP multilayer perceptron and a Softmax layer. Predicted trajectories are obtained based on the probability distribution of vehicle trajectory clusters.

Images