Autonomous-driving trajectory generation method and apparatus, device, medium, product, and vehicle

By processing various vehicle sensor data through a multimodal large language model, a visualized autonomous driving trajectory is generated, which solves the problem that text-based output is difficult to apply in existing technologies, and improves the accuracy and interpretability of the autonomous driving system.

WO2026233454A1PCT designated stage Publication Date: 2026-11-12ZHEJIANG GEELY HLDG GRP CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/095981
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-09
Filing Date
2026-05-09
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing large language models primarily output text in the field of autonomous driving, making them difficult to apply directly to actual driving and thus having a relatively small impact.

Method used

A multimodal large language model is adopted. By acquiring data from various vehicle sensors, the model is encoded using a visual transformer, trajectory autoencoder, and map encoder. Feature fusion is performed using a multilayer perceptron, and a thought chain and visual expert model are introduced for training to generate a visualized autonomous driving trajectory.

Benefits of technology

It improves the accuracy and reliability of autonomous driving systems, enhances their adaptability to complex environments, and increases the interpretability of generated trajectories and the transparency of driving decisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026095981_12112026_PF_FP_ABST
    Figure CN2026095981_12112026_PF_FP_ABST
Patent Text Reader

Abstract

An autonomous-driving trajectory generation method and apparatus, a device, a medium, a product, and a vehicle. The method comprises: acquiring multimodal data acquired by vehicle-mounted sensors, the multimodal data comprising data of the plurality of different data types; inputting the data of the plurality of different data types into encoders corresponding to the respective data types and performing encoding processing to obtain multimodal features having a unified feature representation, the multimodal features comprising features of a plurality of different data types; performing feature fusion on the features of the plurality of different data types to obtain a fused feature-space vector; and inputting the feature-space vector into a multimodal large model and performing inference, and using a trajectory decoder to convert an inference result outputted by the multimodal large model into an autonomous-driving trajectory.
Need to check novelty before this filing date? Find Prior Art

Description

Autonomous driving trajectory generation methods, devices, equipment, media, products and vehicles Cross-reference to related applications

[0001] This application claims priority to Chinese patent application No. 202510591687.9, filed on May 9, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to, but is not limited to, the field of autonomous driving technology, and particularly to a method, apparatus, device, storage medium, program product, and vehicle for generating autonomous driving trajectories. Background Technology

[0003] With the continuous development of autonomous driving technology, how to achieve autonomous driving planning has become a current research hotspot.

[0004] In recent years, with the continuous expansion of application scenarios for large language models, there have been successful precedents of applying large language models to the field of autonomous driving. Leveraging the advantages of large language models in generalization reasoning, logical deduction, and contextual modeling, applying large language models to the field of autonomous driving is gradually becoming a trend. Summary of the Invention

[0005] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0006] This application provides one or more embodiments of a method, apparatus, electronic device, storage medium, program product, and vehicle for generating autonomous driving trajectories.

[0007] According to a first aspect of the embodiments of this application, a method for generating an autonomous driving trajectory is provided, comprising: acquiring multimodal data collected by vehicle-mounted sensors; the multimodal data including data of various different data types; inputting the data of various different data types into encoders corresponding to their respective data types for encoding processing to obtain multimodal features with unified feature representation; the multimodal features including features of various different data types; fusing the features of various different data types to obtain a fused feature space vector; inputting the feature space vector into a multimodal large model for inference, and using a trajectory decoder to convert the inference result output by the multimodal large model into an autonomous driving trajectory.

[0008] In some embodiments, acquiring the multimodal data collected by the vehicle-mounted sensors includes at least one of the following: acquiring video images of the vehicle's surrounding environment collected by the vehicle-mounted image acquisition device; acquiring vehicle motion data collected by the vehicle-mounted sensors; and acquiring vehicle navigation data received by the vehicle-mounted navigation device.

[0009] In some embodiments, the step of inputting the data of various different data types into encoders corresponding to their respective data types for encoding processing to obtain multimodal features with a unified feature representation includes at least one of the following: inputting the video image into a visual transformer for encoding processing to obtain image features; inputting the vehicle motion data into a trajectory autoencoder for encoding processing to obtain motion features; and inputting the vehicle navigation data into a map encoder for encoding processing to obtain navigation features; wherein the image features, the motion features, and the navigation features have a unified feature representation.

[0010] In some embodiments, the step of fusing features of the multiple different data types to obtain a fused feature space vector includes: obtaining basic prompt words configured to guide the multimodal large model to reason in the expected reasoning direction; and fusing the basic prompt words with features of the multiple different data types to obtain the fused feature space vector.

[0011] In some embodiments, the multimodal large model also outputs text information related to the current driving situation, so that the autonomous driving system or the driver can make driving decisions based on the text information.

[0012] In some embodiments, the text information includes at least one of a description of the vehicle's surrounding environment, a description of the vehicle's motion state, and a description of navigation instructions.

[0013] In some embodiments, the method further includes: acquiring training samples; wherein the training samples include input samples and output labels, the input samples include jointly modeled temporal behavior data and environmental perception data, and the output labels include the expected autonomous driving trajectory under the temporal behavior data and the environmental perception data; training the multimodal large model based on the training samples; wherein a thought chain is introduced during the training of the multimodal large model, and the thought chain is constructed with multiple inference tasks configured to help the multimodal large model learn the mapping relationship between the input samples and the output labels.

[0014] In some embodiments, the plurality of reasoning tasks include: an intention instruction task for reasoning the expected driving behavior under the input sample; a scene description task for reasoning the scene description of the environmental perception data in the input sample; and an object labeling task for reasoning the annotation of objects of interest present in the environmental perception data in the input sample.

[0015] In some embodiments, a visual expert model is also introduced during the training of the multimodal large model. The visual expert model is configured to identify objects of interest in the environmental perception data to obtain recognition results, and input the recognition results into the multimodal large model to assist the multimodal large model in learning the mapping relationship between the input samples and the output labels.

[0016] In some embodiments, the inference results output by the multimodal large model include a series of sequentially consecutive longitudinal dynamic control quantities and lateral motion planning parameters; wherein, the longitudinal dynamic control quantities include quantified intensity of throttle or braking operations; and the lateral motion planning parameters include the angle of the vehicle's steering mechanism.

[0017] In some embodiments, the multimodal large model includes an end-to-end multimodal large model.

[0018] According to a second aspect of the embodiments of this application, an apparatus for generating an autonomous driving trajectory is provided, comprising: an acquisition unit configured to acquire multimodal data collected by vehicle-mounted sensors; the multimodal data including data of various different data types; an encoding unit configured to input the data of various different data types into encoders corresponding to their respective data types for encoding processing to obtain multimodal features with unified feature representation; the multimodal features including features of various different data types; a fusion unit configured to fuse the features of various different data types to obtain a fused feature space vector; and a generation unit configured to input the feature space vector into a multimodal large model for inference, and use a trajectory decoder to convert the inference result output by the multimodal large model into an autonomous driving trajectory.

[0019] According to a third aspect of the present application, an electronic device is provided, comprising: at least one processor; at least one memory configured to store processor-executable instructions, wherein the at least one memory and the at least one processor are communicatively connected to each other; and the at least one processor executes the above-described method by invoking the executable instructions.

[0020] According to a fourth aspect of the embodiments of this application, a non-transitory machine-readable storage medium is provided, the non-transitory machine-readable storage medium storing machine-readable instructions, which, when called and executed by at least one processor, implement the above-described method.

[0021] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by at least one processor, implements the above-described method.

[0022] According to a sixth aspect of the present application, a vehicle is provided, comprising: an apparatus for generating an autonomous driving trajectory as described in the second aspect above; or an electronic device as described in the third aspect above; or a non-transitory machine-readable storage medium as described in the fourth aspect above; or a computer program product as described in the fifth aspect above; or at least one processor; and at least one memory communicatively connected to the at least one processor, wherein the at least one memory stores computer-executable instructions, and the at least one processor executes the above-described method by executing the computer-executable instructions.

[0023] It should be understood that the above general description and the following detailed description are exemplary and illustrative only, and are not intended to limit this application. Other aspects will become clear after reading and understanding the accompanying drawings and detailed description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the specific embodiments of this application, the accompanying drawings used in the description of the specific embodiments will be briefly introduced below. The accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 is a schematic diagram of the architecture of a multimodal large model provided by an exemplary embodiment of this application.

[0026] Figure 2 is a flowchart of an autonomous driving trajectory generation method provided by an exemplary embodiment of this application.

[0027] Figure 3 is a schematic diagram of training a large multimodal model provided by an exemplary embodiment of this application.

[0028] Figure 4 is a schematic diagram of the structure of an electronic device containing an autonomous driving trajectory generation device provided in an exemplary embodiment of this application.

[0029] Figure 5 is a block diagram of an automatic driving trajectory generation device provided by an exemplary embodiment of this application. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this application as detailed in the appended claims.

[0031] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0032] As mentioned earlier, one possible approach to generating autonomous driving trajectories in the field of autonomous driving is to transform the trajectory planning task into a language modeling problem, thereby utilizing large language models to generate autonomous driving trajectories based on geographic coordinates. However, existing large language models ultimately output text, which can only serve as a formal cue and has little impact on actual autonomous driving.

[0033] In view of this, embodiments of this application provide a scheme for generating autonomous driving trajectories. The scheme for generating autonomous driving trajectories will be described in detail below with reference to the accompanying drawings.

[0034] Please refer to Figures 1 and 2. Figure 1 is a schematic diagram of the architecture of a multimodal large model provided by an exemplary embodiment of this application, and Figure 2 is a flowchart of a method for generating an autonomous driving trajectory provided by an exemplary embodiment of this application. As shown in Figure 2, the method may include the following steps 210 to 240.

[0035] In step 210, multimodal data collected by the vehicle-mounted sensors is acquired; the multimodal data includes data of various different types.

[0036] The embodiments of this application use a multimodal large language model (MLLM), which is referred to as "multimodal large model" in this paper.

[0037] Multimodal large models extend the semantic understanding capabilities of traditional large-scale language models (LLMs) to the field of visual cognition. Leveraging the advantages of LLMs in generalization reasoning, logical deduction, and contextual modeling, MLLMs construct the foundational framework for cross-modal cognition, enabling visual-language cross-modal alignment.

[0038] In this embodiment of the application, it is first necessary to acquire data of various different types collected by several different vehicle-mounted sensors. These data are usually referred to as multimodal data.

[0039] In one exemplary embodiment, acquiring the multimodal data collected by the vehicle-mounted sensors may include at least one of the following: acquiring video images of the vehicle's surrounding environment collected by the vehicle-mounted image acquisition device; acquiring vehicle motion data collected by the vehicle-mounted sensors; and acquiring vehicle navigation data received by the vehicle-mounted navigation device.

[0040] In this embodiment, the image acquisition device may include, but is not limited to, cameras mounted on the vehicle. Multiple cameras may be present and arranged at different locations on the vehicle to capture video images of the vehicle's surroundings from different angles. For example, cameras may include a front camera for capturing the environment in front of the vehicle, a rear camera for capturing the environment behind the vehicle, side cameras for capturing the environment on both sides of the vehicle, and so on. Thus, multiple cameras can capture 360° surround-view images, providing richer and more accurate environmental references for generating autonomous driving trajectories.

[0041] Vehicle motion data includes, but is not limited to, vehicle speed, acceleration, heading angle (used to characterize the vehicle's orientation), and vehicle attitude. Vehicle motion data can be used to describe the vehicle's motion state and environmental changes.

[0042] Vehicle navigation data can include map data provided by navigation satellites, vehicle positioning data, and more accurate autonomous driving trajectories can be planned using vehicle navigation data.

[0043] In step 220, the data of the various different data types are input into the encoders corresponding to their respective data types for encoding processing to obtain multimodal features with unified feature representation; the multimodal features include features of various different data types.

[0044] In this embodiment, since the multimodal data is collected by different vehicle-mounted sensors, these data have significant differences in data standards such as scale and format, making them difficult to use directly. Therefore, they can be encoded using their respective encoders to unify the feature representation between the multimodal data.

[0045] In an exemplary embodiment, the step of inputting the various data types into encoders corresponding to their respective data types for encoding processing to obtain multimodal features with a unified feature representation may include at least one of the following: inputting video images into a vision transformer for encoding processing to obtain image features; inputting vehicle motion data into a trajectory autoencoder for encoding processing to obtain motion features; and inputting vehicle navigation data into a map encoder for encoding processing to obtain navigation features; wherein the image features, motion features, and navigation features have a unified feature representation.

[0046] As shown in Figure 1, video images (e.g., image data) can be input into a Vision Transformer (ViT) for encoding. A Vision Transformer is a model based on a self-attention mechanism, capable of effectively capturing complex features and relationships in video images.

[0047] Vehicle motion data can be input into a trajectory autoencoder (TAE) for encoding. The trajectory autoencoder specifically includes a combined trajectory autoencoder (TAE) and an ego encoder (Ego Encoder).

[0048] TAE is a deep learning model used to encode and decode time-series data and extract high-level features from it. The vehicle motion data mentioned above is a type of time-series data, which contains vehicle motion data collected by onboard sensors over a continuous period of time (e.g., 5 seconds).

[0049] The goal of an autoencoder is to encode the self-state (such as position, velocity, attitude, etc.) of vehicle motion data and map it into a latent space. Through this latent space, the self-state can be modeled, thereby enabling responses during the decision-making process.

[0050] In this application embodiment, TAE and autoencoder are used together. TAE focuses more on how to extract high-level features from time series data, while autoencoder focuses on modeling the self-state of the vehicle. The combination of the two helps to improve the perception, prediction and decision-making capabilities of autonomous driving.

[0051] Vehicle navigation data can be input into a map encoder (such as SD Map Encoder) for encoding. The map encoder is responsible for parsing and understanding the map data and providing navigation support for the vehicle.

[0052] After being processed by their respective encoders, the aforementioned multimodal data can be converted into multimodal features with a unified feature representation.

[0053] In step 230, the features of the various data types are fused to obtain a high-dimensional feature space vector.

[0054] In the embodiments of this application, feature fusion of multimodal data can be performed using a multilayer perceptron (MLP). An MLP can consist of multiple layers (such as an input layer, a hidden layer, and an output layer), and each layer can contain several neurons. These neurons are connected by weights. Since there is no loop between different layers, it belongs to a feedforward neural network.

[0055] The input layer is used to receive multimodal features, where each data type of feature in the multimodal features can correspond to a neuron.

[0056] The hidden layer is located between the input layer and the output layer. Neurons in the hidden layer can non-linearly transform the output values ​​of neurons in the input layer to extract relationships between multimodal features. This non-linear transformation can be achieved through an activation function.

[0057] The output layer is used to integrate the processed multimodal features into a unified, high-dimensional feature space vector in order to better capture the complex relationships between multimodal data.

[0058] In an exemplary embodiment, the step of fusing features of the various different data types to obtain a high-dimensional feature space vector may further include: obtaining basic prompt words for guiding the multimodal large model to reason in the expected direction; and fusing the basic prompt words with features of the various different data types to obtain a high-dimensional feature space vector.

[0059] As shown in Figure 1, when the multilayer perceptron performs feature fusion, it also includes basic prompts (i.e., prompt data). The basic prompts can be preset instructions or questions used to guide the output of the multimodal large model. By incorporating the basic prompts into the feature space vector, the multimodal large model can understand the requirements of the reasoning task, thereby better generating the expected autonomous driving trajectory.

[0060] In step 240, the feature space vector is input into the multimodal large model for inference, and the inference result output by the multimodal large model is converted into an autonomous driving trajectory using a trajectory decoder.

[0061] As shown in Figure 1, the fused feature space vector can be input into a multimodal large model for inference.

[0062] By integrating visual image data, motion-related vehicle motion data, and vehicle navigation data into a multimodal large model, the multimodal large model can more comprehensively understand the current driving situation and output inference results that conform to the current driving situation.

[0063] On the one hand, the fusion of multimodal data can not only improve the accuracy and reliability of autonomous driving systems, but also enhance their adaptability to complex environments.

[0064] On the other hand, regarding the inference results output by the multimodal large model, this application embodiment adds a trajectory decoder on the basis of the multimodal large model. The trajectory decoder can generate a visualized autonomous driving trajectory using the inference results of the multimodal large model.

[0065] In an exemplary embodiment, as shown in Figure 1, in addition to outputting inference results for the trajectory decoder to generate autonomous driving trajectories, the multimodal large model can also output text information related to the current driving situation, so that the autonomous driving system or driver can make driving decisions based on the text information.

[0066] For example, the text information includes at least one of a description of the vehicle's surrounding environment, a description of the vehicle's motion state, and a description of navigation instructions.

[0067] Text reasoning leverages the powerful language generation capabilities of multimodal large models to transform complex driving information into easily understandable textual forms, thereby improving the interpretability of generated autonomous driving trajectories and providing decision support for autonomous driving or drivers.

[0068] In one exemplary embodiment, the multimodal large model may include an end-to-end multimodal large model.

[0069] End-to-end multimodal big model, also known as end-to-end integrated architecture multimodal big model, improves overall system performance compared to traditional layered architecture multimodal big model.

[0070] In traditional layered architectures, each functional module is optimized independently, and features need to be transferred between different functional modules during model inference. In contrast, end-to-end architectures achieve global optimization by building a shared feature extraction backbone network and by eliminating redundant links in feature transfer between functional modules, thereby improving the utilization of computing resources. Thus, compared with traditional layered architectures, overall system performance is improved.

[0071] In practical applications, multimodal large models mainly rely on the capabilities of the base model itself and lack knowledge of autonomous driving. Therefore, the inference results given may not conform to driving logic and have poor interpretability.

[0072] Therefore, this application improves the training process of the multimodal large model based on the embodiment shown in Figure 2. A thought chain is introduced during the training process to guide the fine-tuning of the multimodal large model in the field of autonomous driving, as described in the following embodiments.

[0073] In an exemplary embodiment, the method further includes: acquiring training samples; wherein the training samples include input samples and output labels, the input samples include temporal behavior data and environmental perception data for joint modeling, and the output labels include the expected autonomous driving trajectory under the temporal behavior data and environmental perception data; training a multimodal large model based on the training samples; wherein, during the training of the multimodal large model, a thought chain is introduced, in which multiple inference tasks are constructed to help the multimodal large model learn the mapping relationship between input samples and output labels. In the embodiments of this application, joint modeling refers to placing two or more data from different sources and different modalities (e.g., temporal behavior data, environmental perception data, etc.) in the same model framework for unified processing and correlation analysis, rather than processing them separately and then simply merging them.

[0074] In the schematic diagram of training a multimodal large model shown in Figure 3, by jointly modeling temporal behavioral data and environmental perception data, the probability distribution of the expected autonomous driving trajectory in the future is predicted, and a complete perception-decision closed loop is formed.

[0075] In this embodiment, the temporal behavior data can cover vehicle motion data over a continuous historical period (such as driving speed, acceleration, heading angle, vehicle attitude, etc., as shown above). Environmental perception data can include video images and spatiotemporal context information.

[0076] During training, the input samples from the training samples are used as model inputs and fed into the multimodal large model that incorporates the thinking chain. Multiple reasoning tasks constructed using the thinking chain guide the multimodal large model to progressively learn the mapping relationship between the input samples and the corresponding output labels.

[0077] In one exemplary embodiment, the plurality of inference tasks include: an Intent Command task, a Scene Description task, and a Major Objects annotation task.

[0078] In this embodiment of the application, the intent instruction task is used to infer the expected driving behavior under the input sample.

[0079] Anticipated driving behavior can refer to clearly stating the vehicle's expected direction of movement in the current scenario. For example: continuing to drive along the current lane, or performing operations such as turning left, turning right, or going straight; determining whether to maintain the current speed, or to perform operations such as deceleration or acceleration.

[0080] These intent commands provide clear guidance for subsequent autonomous driving trajectory planning, ensuring the consistency and safety of driving decisions.

[0081] In this embodiment of the application, the scene description task is used to infer scene descriptions of environmental perception data in the input sample.

[0082] A scenario description can refer to a concise and clear description of the current driving scenario, taking into account factors such as the status of traffic lights, the dynamics of surrounding vehicles or pedestrians, and lane markings. For example: "The traffic light ahead is red, there are no other vehicles passing through the left and right lanes, but a pedestrian is crossing the pedestrian crossing on the right"; "The traffic light ahead is green, and there is a car preparing to merge into this lane from the left lane."

[0083] These scenario descriptions help to fully grasp the current driving environment and provide strong support for making accurate driving decisions.

[0084] In this embodiment, the object annotation task is used to infer the annotation of objects of interest present in the environmental perception data of the input sample.

[0085] Object labeling refers to identifying and labeling road users, i.e., the objects that require key attention during autonomous driving, and specifying their exact locations in the video images of the driving scenario. For each object, a brief description is provided, outlining its current behavior and its importance to driving decisions. For example: Pedestrian: "Located on the right-hand crosswalk, crossing the road to the left. Importance: Slow down to ensure pedestrian safety." Vehicle: "A car on the left is accelerating, preparing to merge into this lane. Importance: Avoid it; may need to adjust speed or lane."

[0086] Through the aforementioned reasoning tasks, not only can detailed driving intentions and scene descriptions be generated, but traffic participants and their behaviors can also be accurately identified, providing more accurate and safer decision support for autonomous driving systems.

[0087] The reasoning process described above simulates the thinking patterns of human drivers, enhancing the transparency and explainability of autonomous driving and improving its ability to handle complex driving situations.

[0088] In one exemplary embodiment, a visual expert model is also introduced during the training of the multimodal large model. In another exemplary embodiment, in addition to being applied to the training process of the multimodal large model, the visual expert model can also be further used as a video preprocessing model in the method for generating autonomous driving trajectories to assist the multimodal model in generating autonomous driving trajectories.

[0089] As shown in Figure 3, the visual expert model is used to identify objects of interest in the environmental perception data and inputs the identification results into the multimodal large model to assist the multimodal large model in learning the mapping relationship between input samples and output labels. In some embodiments of this application, the visual expert model can independently generate explanations of the location and behavior of key targets, that is, identify, locate, and interpret the spatial location and behavioral state (e.g., direction of movement, speed, intention, etc.) of the main objects (or key targets, such as pedestrians, vehicles, traffic signs, etc.) that have an important impact on driving decisions in the scene, thereby assisting the multimodal large model in generating driving trajectories. The multimodal large model can comprehensively infer the overall environmental understanding based on video images and vehicle motion data, that is, a global semantic understanding of the current driving scene. The overall environmental understanding may include, for example, a description of overall information such as weather conditions, road types, and traffic environment. With the assistance of the visual expert model, the multimodal large model can generate driving trajectories based on the understanding of the overall environment, and finally generate trajectory images.

[0090] The embodiments of this application can accurately and efficiently identify objects (traffic participants, such as pedestrians, other vehicles, traffic lights, lane signs, etc.) in environmental perception data (such as video images) by adding a visual expert model, thus helping the multimodal large model to perform the above-mentioned reasoning tasks.

[0091] For example, for an intention command task, if the visual expert model recognizes that the current vehicle is in a straight lane, the current traffic light is red for straight traffic, and there are pedestrians crossing the crosswalk, then the multimodal large model can infer that the expected driving behavior is to brake and stop at the stop line in the straight lane to wait for the green light.

[0092] For the scene description task, based on the recognition results of the aforementioned visual expert model, the multimodal large model can generate the scene description "The current straight-ahead traffic light is red, and there are pedestrians crossing the crosswalk".

[0093] For object labeling tasks, based on the recognition results of the aforementioned visual expert model, the multimodal large model can label the positions of traffic lights, pedestrians, and other vehicles on video images.

[0094] In an exemplary embodiment, the inference results output by the multimodal large model may include several sets of longitudinal dynamic control quantities and lateral motion planning parameters that are sequentially consecutive; wherein, the longitudinal dynamic control quantities include the intensity of quantified throttle or braking operations; and the lateral motion planning parameters include the angle of the vehicle's steering mechanism.

[0095] Unlike traditional direct trajectory regression methods in local coordinate systems, this embodiment designs a dual-channel intermediate representation that includes longitudinal dynamics control quantities and lateral motion planning parameters. The dual-channel intermediate representation refers to an intermediate representation method for autonomous driving trajectory planning, comprising two independent control channels: a longitudinal dynamics control quantity channel, used to quantify longitudinal dynamics control quantities such as the intensity of throttle or braking operations, which then map speed change requirements; and a lateral motion planning parameter channel, used to describe lateral motion planning parameters such as the angle of the steering mechanism, which then reflect the curvature adjustment range of the autonomous driving trajectory. Thus, by decoupling longitudinal and lateral control requirements, the interpretability and flexibility of trajectory planning are improved. In this embodiment, the intensity of throttle or braking operations may include, but is not limited to, the depth or force of pressing the accelerator or brake pedal. The angle of the steering mechanism is a core control parameter for autonomous driving lateral motion planning, specifically referring to the angle used in the vehicle steering system to change the driving direction, reflecting the curvature adjustment range of the autonomous driving trajectory, and needs to be dynamically adjusted according to vehicle speed. The steering mechanism angle can include the steering wheel angle, the front wheel angle, the angle corresponding to the rack displacement, and the steering motor angle. The steering mechanism angle can accurately map the vehicle's driving intention. Whether it is straight driving (e.g., 0 degrees), slight lane change (e.g., ±(5 to 10) degrees), normal turning (e.g., ±(15 to 30) degrees), or sharp turning (e.g., ±(30 to 45) degrees), the expected trajectory tracking and / or path planning can be achieved through the corresponding steering angle control.

[0096] The design of the aforementioned dual-channel intermediate representation is based on modeling human driving behavior: longitudinal control determines the speed of movement, while lateral parameters constrain the direction of motion. This dual-channel intermediate representation mechanism not only improves the interpretability of the decision-making process but also reduces the curse of dimensionality problem caused by high-dimensional feature inputs during autonomous driving trajectory prediction through parameter decoupling (as the dimensionality of input features increases, problems such as data sparsity, computational complexity, and overfitting become more severe).

[0097] Returning to step 240 above, the multimodal large model can generate several sets of longitudinal dynamic control quantities S and lateral motion planning parameters C for the next T seconds based on the input feature space vector and the mapping relationship learned under the guidance of the aforementioned thought chain. Each set of longitudinal dynamic control quantities and lateral motion planning parameters can be regarded as the control variables of a trajectory point. Then, the trajectory decoder converts these several sets of longitudinal dynamic control quantities S and lateral motion planning parameters C for the next T seconds into an autonomous driving trajectory, so that the autonomous driving system can control the vehicle's movement based on the autonomous driving trajectory.

[0098] In an exemplary embodiment of this application, an electronic device capable of implementing the above-described method is also provided.

[0099] Figure 4 is a schematic structural diagram of an electronic device provided by an exemplary embodiment of this application. Referring to Figure 4, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for services. One or more embodiments of this application can be implemented in software, such as at least one processor reading a corresponding computer program from at least one non-volatile memory into memory and then running it. Of course, in addition to software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0100] Please refer to Figure 5. In one software implementation, an apparatus for generating autonomous driving trajectories is provided. The apparatus includes: an acquisition unit 510, an encoding unit 520, a fusion unit 530, and a generation unit 540.

[0101] The acquisition unit 510 is configured to acquire multimodal data collected by the vehicle-mounted sensors; the multimodal data includes data of various different types.

[0102] The encoding unit 520 is configured to input the data of the various different data types into the encoder corresponding to each data type for encoding processing to obtain multimodal features with unified feature representation; the multimodal features include features of various different data types.

[0103] The fusion unit 530 is configured to fuse features of the various different data types to obtain a high-dimensional feature space vector.

[0104] The generation unit 540 is configured to input the feature space vector into a multimodal large model for inference, and use a trajectory decoder to convert the inference result output by the multimodal large model into an autonomous driving trajectory.

[0105] In some embodiments, the acquisition unit 510 is specifically configured to perform at least one of the following: acquire video images of the vehicle's surrounding environment acquired by the vehicle-mounted image acquisition device; acquire vehicle motion data acquired by the vehicle-mounted sensors; and acquire vehicle navigation data received by the vehicle-mounted navigation device.

[0106] In some embodiments, the encoding unit 520 is specifically configured to perform at least one of the following: inputting video images into a vision transformer for encoding processing to obtain image features; inputting vehicle motion data into a trajectory autoencoder for encoding processing to obtain motion features; inputting vehicle navigation data into a map encoder for encoding processing to obtain navigation features; wherein the image features, motion features, and navigation features have a unified feature representation.

[0107] In some embodiments, the fusion unit 530 is configured to: obtain basic prompt words configured to guide the multimodal large model to reason in the expected reasoning direction; and fuse the basic prompt words with features of the various different data types to obtain a high-dimensional feature space vector.

[0108] In some embodiments, the multimodal large model also outputs text information related to the current driving situation, so that the autonomous driving system or the driver can make driving decisions based on the text information.

[0109] In some embodiments, the text information includes at least one of a description of the vehicle's surrounding environment, a description of the vehicle's motion state, and a description of navigation instructions.

[0110] In some embodiments, the apparatus further includes: a sample acquisition unit for acquiring training samples; wherein the training samples include input samples and output labels, the input samples include jointly modeled temporal behavior data and environmental perception data, and the output labels include the expected autonomous driving trajectory under the temporal behavior data and environmental perception data; and a model training unit for training a multimodal large model based on the training samples; wherein a thought chain is introduced during the training of the multimodal large model, and the thought chain contains multiple inference tasks configured to help the multimodal large model learn the mapping relationship between input samples and output labels.

[0111] In some embodiments, the multiple reasoning tasks include: an intention instruction task for reasoning the expected driving behavior under the input sample; a scene description task for reasoning the scene description of the environmental perception data in the input sample; and an object annotation task for reasoning the annotation of objects of interest present in the environmental perception data of the input sample.

[0112] In some embodiments, a visual expert model is also introduced during the training of the multimodal large model. The visual expert model is configured to identify objects of interest in the environmental perception data to obtain recognition results, and input the recognition results into the multimodal large model to assist the multimodal large model in learning the mapping relationship between input samples and output labels.

[0113] In some embodiments, the inference results output by the multimodal large model include several sets of longitudinal dynamic control quantities and lateral motion planning parameters that are sequentially consecutive; wherein, the longitudinal dynamic control quantities include the intensity of quantified throttle or braking operations; and the lateral motion planning parameters include the angle of the vehicle's steering mechanism.

[0114] In some embodiments, the multimodal large model includes an end-to-end multimodal large model.

[0115] For details on the implementation process of the functions and roles of each module in the above-mentioned device, please refer to the implementation process of the corresponding steps in the above-mentioned method for generating autonomous driving trajectories. For relevant details, please refer to the description of the method implementation method. It will not be repeated here.

[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the units or modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0117] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0118] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0119] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0120] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transient computer-readable media, such as modulated data signals and carrier waves.

[0121] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes said element.

[0122] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0123] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” used in one or more embodiments of this application and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0124] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this application, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0125] The above description is merely an example of one or more embodiments of this application and is not intended to limit the scope of the one or more embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of one or more embodiments of this application should be included within the protection scope of one or more embodiments of this application.

Claims

1. A method for generating an autonomous driving trajectory, comprising: Acquire multimodal data collected by vehicle-mounted sensors; The multimodal data includes data of various different types; The data of the various different data types are input into the encoders corresponding to their respective data types for encoding processing to obtain multimodal features with a unified feature representation; the multimodal features include features of various different data types; The features of the various different data types are fused to obtain a fused feature space vector. The feature space vector is input into a multimodal large model for inference, and the inference result output by the multimodal large model is converted into an autonomous driving trajectory using a trajectory decoder.

2. The method according to claim 1, wherein, The acquisition of multimodal data collected by vehicle-mounted sensors includes at least one of the following: Acquire video images of the vehicle's surrounding environment from in-vehicle image acquisition devices; Acquire vehicle motion data collected by onboard sensors; Acquire vehicle navigation data received by the in-vehicle navigation device.

3. The method according to claim 2, wherein, The step of inputting data of various different data types into encoders corresponding to their respective data types for encoding processing to obtain multimodal features with unified feature representation includes at least one of the following: The video image is input into a visual converter for encoding processing to obtain image features; The vehicle motion data is input into a trajectory autoencoder for encoding processing to obtain motion features; The vehicle navigation data is input into a map encoder for encoding processing to obtain navigation features; The image features, motion features, and navigation features have a unified feature representation.

4. The method according to any one of claims 1 to 3, wherein, The step of fusing features from multiple different data types to obtain a fused feature space vector includes: Obtain the basic prompt words configured to guide the multimodal large model to reason in the expected direction; The basic prompt words are fused with features from various different data types to obtain the fused feature space vector.

5. The method according to any one of claims 1 to 4, wherein, The multimodal large model also outputs text information related to the current driving situation, so that the autonomous driving system or the driver can make driving decisions based on the text information.

6. The method according to claim 5, wherein, The text information includes at least one of the following: a description of the vehicle's surrounding environment, a description of the vehicle's motion state, and a description of navigation instructions.

7. The method according to any one of claims 1 to 6, further comprising: Acquire training samples; wherein the training samples include input samples and output labels, the input samples include jointly modeled temporal behavior data and environmental perception data, and the output labels include the expected autonomous driving trajectory under the temporal behavior data and the environmental perception data; The multimodal large model is trained based on the training samples; wherein, during the training of the multimodal large model, a thought chain is introduced, in which multiple inference tasks are constructed to help the multimodal large model learn the mapping relationship between the input samples and the output labels.

8. The method according to claim 7, wherein, The multiple reasoning tasks include: The task of reasoning about the intended driving behavior given the input sample; The task of inferring a scene description based on the environmental perception data in the input sample; The reasoning task involves labeling objects of interest present in the environmental perception data of the input sample.

9. The method according to claim 7 or 8, wherein, During the training of the multimodal large model, a visual expert model is also introduced. The visual expert model is configured to identify objects of interest in the environmental perception data to obtain recognition results, and input the recognition results into the multimodal large model to assist the multimodal large model in learning the mapping relationship between the input samples and the output labels.

10. The method according to any one of claims 1 to 9, wherein, The inference results output by the multimodal large model include several sets of longitudinal dynamic control variables and lateral motion planning parameters that are sequentially continuous. The longitudinal dynamic control quantity includes the quantified intensity of throttle or braking operation; The lateral motion planning parameters include the angle of the vehicle's steering mechanism.

11. The method according to any one of claims 1 to 10, wherein, The multimodal large model includes an end-to-end multimodal large model.

12. An apparatus for generating an autonomous driving trajectory, comprising: The acquisition unit is configured to acquire multimodal data collected by onboard sensors; The multimodal data includes data of various different types; The encoding unit is configured to input the data of the various different data types into the encoder corresponding to each data type for encoding processing to obtain multimodal features with a unified feature representation; the multimodal features include features of various different data types; The fusion unit is configured to fuse features of the multiple different data types to obtain a fused feature space vector. The generation unit is configured to input the feature space vector into a multimodal large model for inference, and use a trajectory decoder to convert the inference results output by the multimodal large model into an autonomous driving trajectory.

13. An electronic device, comprising: At least one processor; At least one memory configured to store processor-executable instructions, wherein the at least one memory and the at least one processor are communicatively connected to each other; The at least one processor implements the method for generating an autonomous driving trajectory as described in any one of claims 1 to 11 by running the executable instructions.

14. A non-transitory machine-readable storage medium, wherein, The non-transitory machine-readable storage medium stores machine-readable instructions that, when executed by at least one processor, implement the method for generating an autonomous driving trajectory as described in any one of claims 1 to 11.

15. A computer program product comprising a computer program that, when executed by at least one processor, implements the method for generating an autonomous driving trajectory as described in any one of claims 1 to 11.

16. A vehicle comprising: The apparatus for generating autonomous driving trajectories according to claim 12, or The electronic device according to claim 13, or The non-transitory machine-readable storage medium according to claim 14, or The computer program product according to claim 15, or At least one processor; and at least one memory communicatively connected to the at least one processor, the at least one memory storing computer-executable instructions, the at least one processor executing the computer-executable instructions to perform the method for generating an autonomous driving trajectory as described in any one of claims 1 to 11.