Vehicle control method and apparatus, and vehicle control unit and storage medium

By fusing vehicle sensor data through a target deep neural network, the system can identify static obstacles and predict dynamic obstacles, solving the accuracy and safety issues of multimodal perception technology in vehicle environmental perception and improving the adaptability and safety of autonomous vehicles in complex environments.

WO2026237392A1PCT designated stage Publication Date: 2026-11-19CHERY AUTOMOBILE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2025/146518
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-06-23
Filing Date
2025-12-29
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing multimodal perception technologies have low accuracy in vehicle environmental perception, resulting in lower vehicle driving safety and difficulty in adapting to complex environments.

Method used

By acquiring sensor data from the vehicle and using a target deep neural network for environmental perception fusion, static obstacles can be identified and the movement trajectories of dynamic obstacles can be predicted, thereby achieving comprehensive perception and intelligent decision-making regarding the vehicle's surrounding environment.

Benefits of technology

It improves the accuracy of environmental perception and the vehicle's adaptability to complex environments, thereby enhancing the driving safety of autonomous vehicles.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2025146518_19112026_PF_FP_ABST
    Figure CN2025146518_19112026_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the present application are a vehicle control method and apparatus, and a vehicle control unit and a storage medium. The method comprises: acquiring sensor data of a current vehicle; on the basis of the sensor data, determining environmental perception fused data by means of a target deep neural network; on the basis of the environmental perception fused data, performing obstacle detection, and obtaining obstacle detection result data by means of the target deep neural network, wherein the obstacle detection result data comprises first data and second data, the first data indicates a recognition result and a classification result for a static obstacle in a surrounding environment of the vehicle, and the second data indicates a trajectory recording result and a path prediction result for a dynamic obstacle in the surrounding environment; and on the basis of the obstacle detection result data, controlling the vehicle to travel on the basis of a travelling action indicated by the obstacle detection result data.
Need to check novelty before this filing date? Find Prior Art

Description

Vehicle control methods, devices, vehicle controllers and storage media

[0001] This disclosure is based on and claims priority to Chinese Patent Application No. 202510845213.2, filed on June 23, 2025, entitled "Vehicle Control Method, Apparatus, Vehicle Controller and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of vehicle technology, and in particular to a vehicle control method, device, vehicle controller and storage medium. Background Technology

[0003] With the rapid development of intelligent driving technology, vehicle perception of its surroundings has become a crucial aspect of vehicle control. Currently, some vehicles employ multimodal perception technologies to achieve this. For example, data is collected simultaneously by different types of sensors, and the sensor data is fused and processed to achieve comprehensive perception of the surrounding environment. However, these multimodal perception technologies have relatively low accuracy in environmental perception, resulting in lower vehicle safety and difficulty in adapting to complex environments. Summary of the Invention

[0004] This application provides a vehicle control method, device, vehicle controller, and storage medium, which improves environmental perception accuracy, thereby enhancing vehicle driving safety and adaptability to complex environments. The technical solution is as follows:

[0005] On the one hand, a vehicle control method is provided, the method comprising:

[0006] Acquire sensor data of the current vehicle, including environmental image data, obstacle radar data, and 3D point cloud data;

[0007] Based on the sensor data, environmental perception fusion data is determined through a target deep neural network. The environmental perception fusion data indicates various features in the environmental image data, the obstacle radar data, and the three-dimensional point cloud data.

[0008] Obstacle detection is performed based on the environmental perception fusion data. Obstacle detection result data is obtained through the target deep neural network. The obstacle detection result data includes first data and second data. The first data indicates the identification and classification results of static obstacles in the surrounding environment of the vehicle, and the second data indicates the trajectory recording results and path prediction results of dynamic obstacles in the surrounding environment.

[0009] Based on the obstacle detection result data, the vehicle is controlled to drive according to the driving actions indicated by the obstacle detection result data.

[0010] In some embodiments, the obstacle detection based on the environmental perception fusion data, and obtaining obstacle detection result data through the target deep neural network, includes:

[0011] The target deep neural network identifies the position and size of each static obstacle in the surrounding environment from the environmental perception fusion data, and performs category identification on each static obstacle to obtain the first data;

[0012] The target deep neural network identifies the motion trajectories of various dynamic obstacles in the surrounding environment from the environmental perception fusion data. Based on the motion trajectories and current speeds of the various dynamic obstacles, the motion paths of the various dynamic obstacles are predicted to obtain the second data.

[0013] In some embodiments, acquiring the sensor data of the current vehicle includes:

[0014] The system acquires environmental image data collected by the vehicle's camera, obstacle radar data collected by the vehicle's millimeter-wave radar, and three-dimensional point cloud data collected by the vehicle's lidar.

[0015] In some embodiments, the environmental perception fusion data includes first fusion data and second fusion data; the step of determining the environmental perception fusion data based on the sensor data using a target deep neural network includes:

[0016] The sensor data is input into the target deep neural network for feature extraction to obtain image features, distance features, and spatial features. The image features indicate obstacles in the surrounding environment, the distance features indicate the movement speed of the obstacles and the spatial distance between them and the vehicle, and the spatial features indicate the spatial distribution and shape characteristics of the obstacles.

[0017] The first fused data is obtained by fusing the image features, distance features, and spatial features using the target deep neural network.

[0018] The sensor data is processed using the target deep neural network to obtain the second fused data.

[0019] In some embodiments, the step of fusing the image features, the distance features, and the spatial features through the target deep neural network to obtain the first fused data includes:

[0020] The feature weights are adjusted for the image features, the distance features, and the spatial features, respectively.

[0021] The target deep neural network is used to perform weighted fusion of the image features, the distance features, and the spatial features based on the feature weights to obtain the environmental perception fusion data.

[0022] In some embodiments, controlling the vehicle to drive based on driving actions indicated by the obstacle detection result data includes at least one of the following:

[0023] The vehicle is controlled to travel based on the driving path corresponding to the obstacle detection result data.

[0024] The vehicle is controlled to travel at a speed corresponding to the obstacle detection result data.

[0025] The vehicle is controlled to avoid obstacles indicated by the obstacle detection result data.

[0026] In some embodiments, the method further includes:

[0027] Based on continuously collected sensor data, environmental perception fusion data, obstacle detection result data, and driving action execution result data, the parameters of the target deep neural network are updated.

[0028] On the other hand, a vehicle control device is provided, the device comprising:

[0029] The acquisition module is used to acquire sensor data of the current vehicle, including environmental image data, obstacle radar data, and three-dimensional point cloud data.

[0030] The fusion module is used to determine environmental perception fusion data based on the sensor data and through a target deep neural network. The environmental perception fusion data indicates various features in the environmental image data, the obstacle radar data, and the three-dimensional point cloud data.

[0031] The detection module is used to detect obstacles based on the environmental perception fusion data and obtain obstacle detection result data through the target deep neural network. The obstacle detection result data includes first data and second data. The first data indicates the identification and classification results of static obstacles in the surrounding environment of the vehicle, and the second data indicates the trajectory recording results and path prediction results of dynamic obstacles in the surrounding environment.

[0032] The control module is used to control the vehicle to drive according to the driving actions indicated by the obstacle detection result data.

[0033] In some embodiments, the detection module is configured to identify the position and size of each static obstacle in the surrounding environment from the environmental perception fusion data using the target deep neural network, classify each static obstacle to obtain the first data; and identify the motion trajectory of each dynamic obstacle in the surrounding environment from the environmental perception fusion data using the target deep neural network, predict the motion path of each dynamic obstacle based on the motion trajectory and current motion speed of each dynamic obstacle to obtain the second data.

[0034] In some embodiments, the acquisition module is used to acquire environmental image data collected by the vehicle's camera, obstacle radar data collected by the vehicle's millimeter-wave radar, and three-dimensional point cloud data collected by the vehicle's lidar.

[0035] In some embodiments, the environmental perception fusion data includes first fusion data and second fusion data; the fusion module includes:

[0036] The feature extraction submodule is used to input the sensor data into the target deep neural network for feature extraction to obtain image features, distance features and spatial features. The image features indicate obstacles in the surrounding environment, the distance features indicate the movement speed of the obstacles and the spatial distance between them and the vehicle, and the spatial features indicate the spatial distribution and shape features of the obstacles.

[0037] The feature fusion submodule is used to perform multimodal feature fusion on the image features, the distance features, and the spatial features through the target deep neural network to obtain the first fused data;

[0038] The timing processing submodule is used to perform timing data processing on the sensor data through the target deep neural network to obtain the second fused data.

[0039] In some embodiments, the feature fusion submodule is used to adjust the feature weights for the image features, the distance features, and the spatial features respectively; and to perform weighted fusion of the image features, the distance features, and the spatial features based on the feature weights through the target deep neural network to obtain the environment perception fusion data.

[0040] In some embodiments, the control module is configured to perform at least one of the following:

[0041] The vehicle is controlled to travel based on the driving path corresponding to the obstacle detection result data.

[0042] The vehicle is controlled to travel at a speed corresponding to the obstacle detection result data.

[0043] The vehicle is controlled to avoid obstacles indicated by the obstacle detection result data.

[0044] In some embodiments, the apparatus further includes:

[0045] The update module is used to update the parameters of the target deep neural network based on continuously collected sensor data, environmental perception fusion data, obstacle detection result data, and driving action execution result data.

[0046] On the other hand, a vehicle controller is provided, which includes a processor and a memory. The memory is used to store at least one computer program, which is loaded and executed by the processor to implement the vehicle control method in the embodiments of this application.

[0047] On the other hand, a vehicle control system is provided, the system including a vehicle controller and a server connected to the vehicle controller, both the vehicle controller and the server including a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded and executed by the processor to implement the vehicle control method in the embodiments of this application.

[0048] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to implement the vehicle control method in the embodiments of this application.

[0049] On the other hand, a computer program product is provided, including a computer program that is executed by a processor to implement the vehicle control method in the embodiments of this application.

[0050] This application provides a vehicle control method that overcomes the limitations of single sensors by fusing information from multiple types of sensors, achieving more comprehensive and accurate environmental perception. By employing a target deep neural network for feature extraction and data fusion of sensor data, it improves the comprehensive processing capability of data from different modalities. This not only accurately identifies and classifies static obstacles but also records and predicts the movement trajectories of dynamic obstacles in real time, realizing intelligent detection of both static and dynamic obstacles. This approach enhances the perception accuracy and real-time decision-making capabilities of autonomous vehicles in complex environments, improves the vehicle's adaptability to complex scenarios, and thus enhances driving safety. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 is a schematic diagram of the implementation environment of a vehicle control method provided in an embodiment of this application;

[0053] Figure 2 is a flowchart of a vehicle control method provided in an embodiment of this application;

[0054] Figure 3 is a schematic diagram of an overall process provided in an embodiment of this application;

[0055] Figure 4 is a block diagram of a vehicle control device provided in an embodiment of this application;

[0056] Figure 5 is a block diagram of another vehicle control device provided in an embodiment of this application;

[0057] Figure 6 is a schematic diagram of the structure of a vehicle controller provided in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0059] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor is there any limitation on the quantity or execution order.

[0060] In this application, the term "at least one" means one or more, and "multiple" means two or more.

[0061] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sensor data involved in this application were obtained with full authorization.

[0062] Figure 1 is a schematic diagram of the implementation environment of a vehicle control method provided in an embodiment of this application. Referring to Figure 1, the implementation environment includes a parameter acquisition device 101, a vehicle controller 102, and a server 103. The parameter acquisition device 101 and the vehicle controller 102 are deployed in the vehicle, while the server 103 is located outside the vehicle. The parameter acquisition device 101 and the vehicle controller 102 can be directly or indirectly connected via wired or wireless communication, and the vehicle controller 102 and the server 103 can be directly or indirectly connected via wireless communication; this application does not impose any limitations on this connection.

[0063] In some embodiments, the parameter acquisition device 101 is used to acquire sensor data of the vehicle. The parameter acquisition device 101 includes at least one of a camera, millimeter-wave radar, and LiDAR (Light Detection and Ranging). The camera is used to acquire images of the vehicle's surrounding environment, converting the surrounding environment into digital image signals; the millimeter-wave radar is used to acquire information such as the distance, speed, and orientation of obstacles around the vehicle, sensing the environment by emitting millimeter waves and receiving reflected signals; the LiDAR is used to acquire three-dimensional point cloud data of the vehicle's surrounding environment, constructing spatial coordinate information of the environment by emitting laser beams and measuring reflection time. The parameter acquisition device 101 may also include a data transmission module for sending the vehicle's sensor data to the vehicle controller 102.

[0064] In some embodiments, the vehicle controller 102 acquires sensor data of the vehicle and sends the sensor data to the server 103. The vehicle controller 102 acquires the vehicle's sensor data from the parameter acquisition device 101. This application does not limit the data acquisition method or the data transmission method.

[0065] In some embodiments, server 103 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. This application does not limit this.

[0066] Figure 2 is a flowchart of a vehicle control method provided in an embodiment of this application. The method is executed by a vehicle control system, which includes a vehicle controller and a server connected to the vehicle controller.

[0067] Referring to Figure 2, the method includes the following steps:

[0068] 201. Acquire the sensor data of the current vehicle, including environmental image data, obstacle radar data, and 3D point cloud data.

[0069] In this embodiment, the vehicle controller acquires sensor data of the current vehicle from the vehicle's parameter acquisition device. The parameter acquisition device includes multiple different types of sensors, such as cameras, millimeter-wave radar, and lidar.

[0070] In some embodiments, each type of sensor is used for a specific type of information acquisition. Accordingly, environmental image data acquired by the vehicle's camera, obstacle radar data acquired by the vehicle's millimeter-wave radar, and 3D point cloud data acquired by the vehicle's lidar are obtained. Specifically, the camera is used to acquire real-time images of the vehicle's surrounding environment for the identification of road signs, traffic signals, lane lines, and obstacles; the millimeter-wave radar is used to acquire distance and speed information of surrounding obstacles for the detection of dynamic targets, such as vehicles and pedestrians; and the lidar is used to acquire 3D point cloud data of the surrounding environment to indicate the spatial structure of the surrounding environment and accurately perceive the spatial position and shape of obstacles. By combining multiple sensors, the shortcomings of single sensors are overcome, such as the influence of lighting on cameras, the low resolution of millimeter-wave radar, and the high cost of lidar, ensuring reliability in various scenarios.

[0071] Since the data frequencies and formats of various sensors differ, preprocessing is performed on the sensor data after acquiring the current vehicle's sensor data to ensure the timeliness and synchronization of data from different modalities. Optionally, a time synchronization module can be used to align the sensor data and perform operations such as denoising, scaling, and format conversion on the acquired raw data. For example, denoising is performed on environmental image data, filtering is performed on obstacle radar data, and spatial registration is performed on 3D point cloud data to ensure consistency of multimodal data; these details will not be elaborated further here.

[0072] In some embodiments, after the vehicle controller obtains the sensor data of the current vehicle in step 201, it sends the sensor data to a server (such as a cloud server), and the server executes subsequent steps 202 and 203. This application embodiment uses this as an example for illustrative purposes. In some embodiments, subsequent steps 202 and 203 can also be executed by the vehicle controller or other modules in the vehicle; this application embodiment does not impose any limitations on this.

[0073] In this embodiment, the target deep neural network is used for multimodal data fusion, including a multi-channel input, a feature extraction layer, a data fusion layer, and an output layer. The target deep neural network is also used for environmental perception and intelligent decision-making on the fused data. In other words, the target deep neural network refers to a trained deep neural network model used to achieve data fusion and obstacle detection. The target deep neural network can output environmental perception fused data during data fusion and obstacle detection result data during obstacle detection. A detailed description follows.

[0074] 202. Based on sensor data, environmental perception fusion data is determined through a target deep neural network. The environmental perception fusion data indicates various features in environmental image data, obstacle radar data, and 3D point cloud data.

[0075] In this embodiment of the application, the server determines environmental perception fusion data based on sensor data through a target deep neural network.

[0076] In some embodiments, feature extraction and feature fusion are performed on sensor data of different modalities. Accordingly, sensor data is input into a target deep neural network for feature extraction to obtain image features, distance features, and spatial features. Image features indicate obstacles in the surrounding environment, distance features indicate the speed of movement of obstacles and the spatial distance between them and the vehicle, and spatial features indicate the spatial distribution and shape characteristics of obstacles. The target deep neural network then performs multimodal feature fusion on the image features, distance features, and spatial features to obtain first fused data. That is, after the feature extraction layer, features from all modalities are fused in the data fusion layer, and then the first fused data is output by the fully connected layer (i.e., the output layer).

[0077] Each type of sensor data is processed separately to extract key features. For example, in the image feature extraction channel, a convolutional neural network (CNN) is used to process environmental image data to extract high-level image features indicating road edges, traffic signs, pedestrians, vehicles, etc.; in the distance feature extraction channel, a graph convolutional network (GCN) is used to process obstacle radar data to obtain distance features indicating the distance, speed, etc., of dynamic obstacles; and in the spatial feature extraction channel, a 3D convolutional neural network (3D-CNN) is used to process 3D point cloud data to extract spatial features indicating the spatial distribution and shape of obstacles.

[0078] In this way, data from various sensors are input into the target deep neural network, which extracts image features, distance features, and spatial features respectively. The fusion layer realizes unified processing of multimodal data, generating first fused data with rich dimensional information, which enhances the understanding of the vehicle's surrounding environment.

[0079] In some embodiments, adaptive fusion is achieved by dynamically adjusting feature weights. Accordingly, feature weights are adjusted for image features, distance features, and spatial features respectively; the image features, distance features, and spatial features are weighted and fused based on the feature weights using a target deep neural network to obtain environmental perception fusion data.

[0080] In this approach, based on the importance of different modalities, the weights of different modal features are adaptively adjusted through an attention mechanism or weighting mechanism to ensure the complementarity and synergy of modal information in complex scenes, and to ensure the effective fusion of key features. The reliability of various sensors differs in different scenarios; for example, LiDAR performs better in tunnels, while cameras provide richer information in clear weather. Weights are dynamically allocated based on real-time sensor data. For instance, in rainy conditions or poor lighting, the weight of image features related to cameras is reduced, while the weight of distance features related to millimeter-wave radar is increased to ensure the accuracy of environmental perception. This approach avoids the information redundancy or loss problems caused by fixed-weight fusion and improves the ability to cope with complex and changing environments.

[0081] In some embodiments, sensor data is processed using a target deep neural network to obtain second fused data. For example, by combining time-series networks such as Long Short-Term Memory Networks (LSTM), the temporal dependencies of multimodal sensor data can be processed, thereby enhancing adaptability to dynamic scenes.

[0082] In some embodiments, the environmental perception fusion data includes first fusion data and second fusion data. The above is an exemplary description of determining the environmental perception fusion data based on sensor data and using a target deep neural network in this context.

[0083] 203. Obstacle detection is performed based on environmental perception fusion data. Obstacle detection result data is obtained through a target deep neural network. The obstacle detection result data includes first data and second data. The first data indicates the identification and classification results of static obstacles in the vehicle's surrounding environment, and the second data indicates the trajectory recording results and path prediction results of dynamic obstacles in the surrounding environment.

[0084] In this embodiment, the server uses a target deep neural network to perform obstacle detection based on fused environmental perception data, obtaining obstacle detection result data. For example, the fused features are input into a fully connected network to output results such as object detection, classification, and trajectory prediction of the vehicle's surrounding environment.

[0085] In some embodiments, environmental perception and intelligent decision-making are performed based on fused data, categorized into static and dynamic obstacles. Accordingly, a target deep neural network is used to identify the position and size of each static obstacle in the surrounding environment from the fused environmental perception data, and classifies each static obstacle to obtain first data. The target deep neural network is also used to identify the motion trajectories of each dynamic obstacle in the surrounding environment from the fused environmental perception data, and based on the motion trajectories and current speeds of each dynamic obstacle, the movement paths of each dynamic obstacle are predicted to obtain second data.

[0086] In other words, based on environmental perception fusion data, static target detection is performed, such as the identification and classification of static obstacles like road boundaries, driving areas, and parking signs; and dynamic target tracking is performed, such as tracking and path prediction of moving vehicles, pedestrians, and other dynamic obstacles through long short-term memory networks, ensuring real-time performance and accuracy.

[0087] In this approach, vehicles can achieve precise perception of complex driving environments with the support of multimodal sensor data. This not only enables accurate obstacle detection and classification but also allows for the prediction of potential risks and subsequent driving maneuvers. This method separates the processing of static and dynamic obstacles. For static obstacles, identifying their location, size, and category provides precise spatial constraints for path planning; for dynamic obstacles, predicting future paths based on historical trajectories and current speed provides a reference for the next driving maneuver. This approach not only improves detection efficiency but also enhances the tracking stability of dynamic targets, allowing for proactive adjustments to the vehicle's driving actions.

[0088] 204. Based on obstacle detection data, control the vehicle to drive according to the driving actions indicated by the obstacle detection data.

[0089] In this embodiment, the server sends obstacle detection result data to the vehicle controller, which then applies the perception and decision-making results to the intelligent driving system to achieve intelligent control of the vehicle's driving actions. That is, based on the obstacle detection result data, a deep reinforcement learning module is used to make real-time decisions, providing the intelligent driving vehicle with safer and more reliable route planning and control commands.

[0090] In some embodiments, controlling the vehicle to drive based on driving actions indicated by obstacle detection result data includes at least one of the following: controlling the vehicle to drive based on a driving path corresponding to the obstacle detection result data; controlling the vehicle to drive based on a driving speed corresponding to the obstacle detection result data; and controlling the vehicle to avoid obstacles indicated by the obstacle detection result data.

[0091] In other words, based on obstacle detection data, the vehicle is controlled to automatically avoid obstacles, such as adjusting speed and path according to obstacle detection results; and the vehicle is also controlled to plan paths, such as combining environmental perception fusion data and dynamic obstacle movement path prediction to optimize the vehicle's driving path to ensure safety and efficiency. This multi-dimensional control strategy makes the vehicle's driving actions more rational and improves driving safety.

[0092] In some embodiments, the parameters of the target deep neural network are updated based on continuously collected sensor data, environmental perception fusion data, obstacle detection results, and driving action execution results. For example, the target deep neural network is continuously optimized based on the deviation data between the identification results of static and dynamic obstacles and the actual scene, the comparison between the prediction results of potential risks and the actual occurrence, and the execution results of operations such as automatic obstacle avoidance and path planning. In this way, through real-time feedback and the self-learning capability of deep neural networks, the perception and decision-making processes can be dynamically adjusted according to the continuously collected data, thereby enhancing the adaptability to complex environments.

[0093] For ease of description, the above scheme can be divided into the following levels, each representing a different module from basic data acquisition and algorithm processing to final application. These levels are data acquisition (as in step 201), feature extraction (as in step 202), data fusion (as in step 202), perception and decision-making (as in step 203), application implementation (as in step 204), and feedback optimization.

[0094] To facilitate the description of the overall process, please refer to Figure 3, which is a schematic diagram of an overall process provided by an embodiment of this application. First, sensor data of the current vehicle is acquired using parameter acquisition devices such as cameras, millimeter-wave radar, and lidar. This sensor data is input into the target deep neural network for feature extraction, yielding the following features: image features of roads, pedestrians, and vehicles; distance and speed information of dynamic targets; and spatial distribution and shape features of obstacles. Data fusion is performed on these extracted features, including multimodal data fusion and time-series data processing. Perception and decision-making are performed based on the fused data, including static target detection and dynamic target tracking. Intelligent driving applications, namely automatic obstacle avoidance path planning, are implemented based on the results of perception and decision-making. Throughout this process, the target deep neural network continuously undergoes feedback and self-learning.

[0095] This application provides a vehicle control method that overcomes the limitations of a single sensor by fusing information from multiple types of sensors, achieving more comprehensive and accurate environmental perception. By using a target deep neural network to extract features and fuse data from sensor data, the method improves the comprehensive processing capability of data from different modalities. It can not only accurately identify and classify static obstacles but also record and predict the movement trajectories of dynamic obstacles in real time, achieving intelligent detection of both static and dynamic obstacles. This approach enhances the perception accuracy and real-time decision-making capability of autonomous vehicles in complex environments, improves the vehicle's adaptability to complex scenarios, and thus enhances vehicle driving safety.

[0096] A more detailed analysis follows.

[0097] First, this solution automatically learns features from multimodal data (such as cameras, millimeter-level radar, and lidar), enabling the acquisition of richer environmental information from multiple dimensions, including vision, distance, and speed, thereby enhancing the understanding and perception of complex scenes. By fusing data from multiple sensors, stable environmental perception can be maintained even when individual sensors malfunction or experience significant data noise, improving robustness and the safety of autonomous driving.

[0098] Secondly, this scheme utilizes an end-to-end deep learning framework (i.e., a target deep neural network) to directly extract, fuse, classify, and make decisions from raw sensor data. Compared to traditional modular algorithm architectures, this framework not only reduces manual intervention in intermediate processes and simplifies data processing, but also accelerates processing speed through model optimization, meeting the stringent requirements of real-time performance and efficient perception in intelligent driving systems. Furthermore, this scheme exhibits strong scalability, adapting to different types of sensors and complex scenario requirements, making it suitable for the continuous evolution and upgrading of future intelligent driving systems. The algorithm improves processing efficiency and generalization ability.

[0099] In summary, this solution enables efficient and accurate obstacle detection, road recognition, and dynamic target tracking based on multimodal data. It can dynamically adapt to different scenarios and weather conditions. By learning the complementarity of different modalities and adjusting their weights, it improves the perception robustness and accuracy in complex driving environments (such as rainy or foggy weather, nighttime, or high-speed driving). It can achieve efficient and accurate perception and decision-making in complex and dynamic driving environments, thereby improving adaptability to complex scenarios and environmental perception accuracy. This ensures safe driving of vehicles under various weather and lighting conditions, providing reliable support for the safety and intelligence of autonomous driving.

[0100] Furthermore, the aforementioned multimodal perception algorithm was integrated into the intelligent driving vehicle system and tested in real-world road environments. Test scenarios included urban streets, highways, and complex intersections. Results showed that the algorithm could stably and accurately identify obstacles under various lighting, weather, and traffic conditions, and the system response speed met real-time requirements.

[0101] Figure 4 is a block diagram of a vehicle control device provided in an embodiment of this application. This device is used to execute the steps of the vehicle control method described above. Referring to Figure 4, the vehicle control device includes: an acquisition module 401, a fusion module 402, a detection module 403, and a control module 404.

[0102] The acquisition module 401 is used to acquire the sensor data of the current vehicle, including environmental image data, obstacle radar data, and three-dimensional point cloud data.

[0103] The fusion module 402 is used to determine environmental perception fusion data based on sensor data and through a target deep neural network. The environmental perception fusion data indicates various features in environmental image data, obstacle radar data, and 3D point cloud data.

[0104] The detection module 403 is used to detect obstacles based on environmental perception fusion data. It obtains obstacle detection result data through a target deep neural network. The obstacle detection result data includes first data and second data. The first data indicates the identification and classification results of static obstacles in the vehicle's surrounding environment, and the second data indicates the trajectory recording results and path prediction results of dynamic obstacles in the surrounding environment.

[0105] The control module 404 is used to control the vehicle to drive based on the obstacle detection result data.

[0106] In some embodiments, the detection module 403 is used to identify the position and size of each static obstacle in the surrounding environment from the environmental perception fusion data through the target deep neural network, perform category identification on each static obstacle, and obtain first data; and to identify the motion trajectory of each dynamic obstacle in the surrounding environment from the environmental perception fusion data through the target deep neural network, and predict the motion path of each dynamic obstacle based on the motion trajectory and current motion speed of each dynamic obstacle, and obtain second data.

[0107] In some embodiments, the acquisition module 401 is used to acquire environmental image data collected by the vehicle's camera, obstacle radar data collected by the vehicle's millimeter-wave radar, and three-dimensional point cloud data collected by the vehicle's lidar.

[0108] In some embodiments, FIG5 is a block diagram of another vehicle control device provided in this application.

[0109] The environmental perception fusion data includes first fusion data and second fusion data; the fusion module 402 includes:

[0110] The feature extraction submodule 4021 is used to input sensor data into the target deep neural network for feature extraction, and obtain image features, distance features and spatial features. The image features indicate obstacles in the surrounding environment, the distance features indicate the speed of movement of obstacles and the spatial distance between them and the vehicle, and the spatial features indicate the spatial distribution and shape features of obstacles.

[0111] The feature fusion submodule 4022 is used to perform multimodal feature fusion on image features, distance features and spatial features through the target deep neural network to obtain the first fused data;

[0112] The timing processing submodule 4023 is used to perform timing data processing on sensor data through the target deep neural network to obtain the second fused data.

[0113] In some embodiments, the feature fusion submodule 4022 is used to adjust the feature weights for image features, distance features and spatial features respectively; and to perform weighted fusion of image features, distance features and spatial features based on feature weights through a target deep neural network to obtain environmental perception fusion data.

[0114] In some embodiments, the control module 404 is configured to perform at least one of the following:

[0115] The vehicle is controlled to travel based on the driving path corresponding to the obstacle detection result data.

[0116] The vehicle is controlled to travel at the speed corresponding to the obstacle detection result data.

[0117] Control the vehicle to avoid obstacles indicated by obstacle detection data.

[0118] In some embodiments, the apparatus further includes:

[0119] The update module 405 is used to update the parameters of the target deep neural network based on continuously collected sensor data, environmental perception fusion data, obstacle detection result data, and driving action execution result data.

[0120] This application provides a vehicle control device that overcomes the limitations of a single sensor by fusing information from multiple types of sensors, achieving more comprehensive and accurate environmental perception. By employing a target deep neural network for feature extraction and data fusion from sensor data, it improves the comprehensive processing capability of data from different modalities. This not only enables accurate identification and classification of static obstacles but also allows for real-time recording and path prediction of the movement trajectories of dynamic obstacles, achieving intelligent detection of both static and dynamic obstacles. This approach enhances the perception accuracy and real-time decision-making capabilities of autonomous vehicles in complex environments, improves the vehicle's adaptability to complex scenarios, and thus enhances driving safety.

[0121] It should be noted that the vehicle control device provided in the above embodiments is only illustrated by the division of the above functional modules when running the application program. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the vehicle control system or vehicle controller can be divided into different functional modules to complete all or part of the functions described above. In addition, the vehicle control device and the vehicle control method embodiments provided in the above embodiments belong to the same concept, and their implementation process can be found in the method embodiments, which will not be repeated here.

[0122] Figure 6 is a schematic diagram of a vehicle controller provided in an embodiment of this application. The vehicle controller 600 includes a main control module 601 and a CAN (Controller Area Network) interface 602. The main control module 601 is connected to the CAN interface 602.

[0123] The main control module 601 typically includes a processor and a memory. The memory is used to store at least one computer program, which is loaded and executed by the processor to implement the vehicle control method in the embodiments of this application.

[0124] The processor can include one or more processing cores, such as a 6-core processor. The processor can be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor can also include a main processor and coprocessors. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor can integrate a GPU (Graphics Processing Unit) for rendering and drawing the content required to be displayed on the vehicle's screen. In some embodiments, the processor can also include an AI (Artificial Intelligence) processor for handling computational operations related to machine learning. The memory can include one or more computer-readable storage media, which can be non-transitory. The memory can also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, a non-transitory computer-readable storage medium in the memory is used to store at least one computer program, which is executed by a processor to implement the vehicle control method provided in the method embodiments of this application.

[0125] The CAN interface 602 can include multiple controller interfaces for communication with the motor controller, transmission controller, and steering controller, respectively. The main control module 601 can connect to multiple controllers through these interfaces and send control commands to them to control the vehicle. The motor controller, transmission controller, and steering controller are all actuators in the vehicle. The motor controller controls the motor speed and torque output to achieve speed control and power output. The transmission controller adjusts the vehicle's gears to match the current driving speed and power requirements. The steering controller controls the vehicle's steering system to achieve steering control.

[0126] The CAN interface 602 may also include interfaces for communicating with multiple parameter acquisition devices such as lidar, cameras, millimeter-level radar, and ultrasonic sensors. The main control module 601 can connect to multiple parameter acquisition devices through the above-mentioned interfaces to obtain data sent by multiple parameter acquisition devices, thereby realizing the acquisition of data on the vehicle's surrounding environment.

[0127] The CAN interface 602 may also include an interface for communicating with a server. The main control module 601 can connect to the server through the above interface, send sensor data to the server, and receive various data sent by the server.

[0128] Those skilled in the art will understand that the structure shown in Figure 6 does not constitute a limitation on the vehicle controller 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0129] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the vehicle control method described in the above embodiments. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, or an optical data storage device, etc.

[0130] This application also provides a computer program product, including a computer program that is executed by a processor to implement the vehicle control method in this application.

[0131] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0132] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A vehicle control method in which, The method includes: Acquire sensor data of the current vehicle, including environmental image data, obstacle radar data, and 3D point cloud data; Based on the sensor data, environmental perception fusion data is determined through a target deep neural network. The environmental perception fusion data indicates various features in the environmental image data, the obstacle radar data, and the three-dimensional point cloud data. Obstacle detection is performed based on the environmental perception fusion data. Obstacle detection result data is obtained through the target deep neural network. The obstacle detection result data includes first data and second data. The first data indicates the identification and classification results of static obstacles in the surrounding environment of the vehicle, and the second data indicates the trajectory recording results and path prediction results of dynamic obstacles in the surrounding environment. Based on the obstacle detection result data, the vehicle is controlled to drive according to the driving actions indicated by the obstacle detection result data.

2. The method of claim 1, wherein, The obstacle detection based on the environmental perception fusion data, through the target deep neural network, yields obstacle detection result data, including: The target deep neural network identifies the position and size of each static obstacle in the surrounding environment from the environmental perception fusion data, and performs category identification on each static obstacle to obtain the first data; The target deep neural network identifies the motion trajectories of various dynamic obstacles in the surrounding environment from the environmental perception fusion data. Based on the motion trajectories and current speeds of the various dynamic obstacles, the motion paths of the various dynamic obstacles are predicted to obtain the second data.

3. The method of claim 1, wherein, The acquisition of current vehicle sensor data includes: The system acquires environmental image data collected by the vehicle's camera, obstacle radar data collected by the vehicle's millimeter-wave radar, and three-dimensional point cloud data collected by the vehicle's lidar.

4. The method of claim 1, wherein, The environmental perception fusion data includes first fusion data and second fusion data; the process of determining the environmental perception fusion data based on the sensor data using a target deep neural network includes: The sensor data is input into the target deep neural network for feature extraction to obtain image features, distance features, and spatial features. The image features indicate obstacles in the surrounding environment, the distance features indicate the movement speed of the obstacles and the spatial distance between them and the vehicle, and the spatial features indicate the spatial distribution and shape characteristics of the obstacles. The first fused data is obtained by fusing the image features, distance features, and spatial features using the target deep neural network. The sensor data is processed using the target deep neural network to obtain the second fused data.

5. The method of claim 4, wherein, The first fused data is obtained by fusing multimodal features of the image features, the distance features, and the spatial features through the target deep neural network, including: The feature weights are adjusted for the image features, the distance features, and the spatial features, respectively. The target deep neural network is used to perform weighted fusion of the image features, the distance features, and the spatial features based on the feature weights to obtain the environmental perception fusion data.

6. The method of claim 1, wherein, Controlling the vehicle to drive based on the driving actions indicated by the obstacle detection result data includes at least one of the following: The vehicle is controlled to travel based on the driving path corresponding to the obstacle detection result data. The vehicle is controlled to travel at a speed corresponding to the obstacle detection result data. The vehicle is controlled to avoid obstacles indicated by the obstacle detection result data.

7. The method of claim 1, wherein, The method further includes: Based on continuously collected sensor data, environmental perception fusion data, obstacle detection result data, and driving action execution result data, the parameters of the target deep neural network are updated.

8. A vehicle control device, wherein, The device includes: The acquisition module is used to acquire sensor data of the current vehicle, including environmental image data, obstacle radar data, and three-dimensional point cloud data. The fusion module is used to determine environmental perception fusion data based on the sensor data and through a target deep neural network. The environmental perception fusion data indicates various features in the environmental image data, the obstacle radar data, and the three-dimensional point cloud data. The detection module is used to detect obstacles based on the environmental perception fusion data and obtain obstacle detection result data through the target deep neural network. The obstacle detection result data includes first data and second data. The first data indicates the identification and classification results of static obstacles in the surrounding environment of the vehicle, and the second data indicates the trajectory recording results and path prediction results of dynamic obstacles in the surrounding environment. The control module is used to control the vehicle to drive according to the driving actions indicated by the obstacle detection result data.

9. An integrated vehicle controller, wherein, The vehicle controller includes a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded by the processor and executed as the vehicle control method according to any one of claims 1 to 7.

10. A computer readable storage medium, wherein, The computer-readable storage medium is used to store at least one computer program for performing the vehicle control method according to any one of claims 1 to 7.

11. A computer program product, wherein, The product stores at least one piece of program code, which is loaded and executed by a processor to implement the vehicle control method according to any one of claims 1 to 7.