Travelable area planning method and apparatus

Abstract

The present application provides a travelable area planning method and an apparatus, used for reasoning the environment around a vehicle in combination with crowdsourced data, so as to plan a more accurate travelable area for the vehicle on the basis of limited obstacle information. The method comprises: acquiring crowdsourced device data and crowdsourced collection data, the crowdsourced device data comprising data of a plurality of second devices, the crowdsourced collection data comprising data collected by sensors in the plurality of second devices, and the second devices being also referred to as crowdsourced devices; performing data encoding, comprising encoding the crowdsourced device data to obtain device path encoding data, and encoding the crowdsourced collection data to obtain environmental semantic encoding data; and then inputting the device path encoding data and the environmental semantic encoding data into a neural network to output a first travelable area, the neural network being used for extracting the association relationship between the environmental semantic encoding data and the device path encoding data, and determining the first travelable area on the basis of the association relationship.

Description

A method and apparatus for planning drivable areas Technical Field

[0001] This application relates to the field of intelligent driving, and more particularly to a method and apparatus for planning drivable areas. Background Technology

[0002] In intelligent driving functions, determining a vehicle's route typically requires identifying specific information about surrounding targets, such as obstacle locations, road structures, and other vehicle information. Existing solutions often rely on sensor field of view or accuracy, or the accuracy and robustness of perception algorithms, to determine the route. However, if there are errors in target recognition, blind spots in obstacle handling, or obstacles exceeding the sensor's field of view, the vehicle may be unable to anticipate obstacle locations at high speeds, leading to situations where obstacle avoidance is impossible and compromising safe driving. Therefore, how to more accurately determine a drivable area for the vehicle based on limited obstacle information is a pressing issue that needs to be addressed. Summary of the Invention

[0003] This application provides a drivable area planning method and apparatus, which combines crowdsourced data to infer the environment around a vehicle, thereby planning a more accurate drivable area for the vehicle based on limited obstacle information.

[0004] In view of this, in a first aspect, this application provides a drivable area planning method applied to a first device, comprising: acquiring crowdsourced device data and crowdsourced collection data, wherein the crowdsourced device data includes data from multiple second devices, and the crowdsourced collection data includes data collected by sensors in multiple second devices, the second devices may also be referred to as crowdsourced devices, such as vehicles, drones, robots, and other drivable intelligent devices; subsequently performing data encoding, including encoding the crowdsourced device data to obtain device path encoding data, and encoding the crowdsourced collection data to obtain environmental semantic encoding data; subsequently inputting the device path encoding data and the environmental semantic encoding data into a neural network, outputting a first drivable area, the first drivable area being the area where the first device can driv, the neural network being used to extract the correlation between the environmental semantic encoding data and the device path encoding data, and determining the first drivable area based on the correlation.

[0005] Typically, crowdsourced devices can collect information about their surrounding environment while in operation, and they usually operate within drivable areas. In this embodiment, crowdsourced device data and collected data can be used to infer the vehicle's environmental conditions. Even if the vehicle has blind spots or obstacles detected by sensors, the road structure can be accurately inferred based on limited information about the source of these obstacles, thus avoiding blind spots and planning a more accurate drivable area.

[0006] In one possible implementation, the aforementioned crowdsourced data collection includes crowdsourced point cloud data, which is data obtained by fusing data collected by sensors of multiple second devices. The crowdsourced device data package includes crowdsourced traffic flow data, which includes path information for multiple second devices. In this application embodiment, the crowdsourced data can specifically include crowdsourced traffic flow data and crowdsourced point cloud data. Crowdsourced traffic flow data can represent the path information of the crowdsourced devices and can be used to determine the actually traversable areas of the devices in the environment. Crowdsourced point cloud data can be used to infer information about the surrounding environment, including specific road structures and obstacle information, for more accurate environmental reasoning.

[0007] In one possible implementation, the aforementioned encoding of crowdsourced device data to obtain device path encoding data may include: performing vector encoding on crowdsourced vehicle flow data to obtain device path encoding data, that is, encoding the vectors in the crowdsourced vehicle flow data, so that the obtained device path encoding data can more accurately represent the passable areas in the environment from the vector dimension; the aforementioned encoding of crowdsourced collected data to obtain environmental semantic encoding data includes: performing semantic encoding on crowdsourced point cloud data to obtain environmental semantic encoding data, that is, encoding the semantics of the point cloud representing the environment, so as to more accurately identify information such as road structures or obstacles in the environment.

[0008] In one possible implementation, the aforementioned method further includes: aligning crowdsourced point cloud data and crowdsourced device data in the same space to obtain aligned crowdsourced point cloud data and aligned crowdsourced device data; subsequently, slicing the aligned crowdsourced point cloud data and aligned crowdsourced device data in the same space to obtain multiple point cloud data slices and multiple device data slices; correspondingly, the aforementioned vector encoding of crowdsourced vehicle flow data to obtain device path encoding data can be performed by: performing vector encoding on each device data slice in the multiple device data slices to obtain device path encoding data corresponding to each device data slice; correspondingly, the aforementioned semantic encoding of crowdsourced point cloud data to obtain environmental semantic encoding data includes: performing semantic encoding on each point cloud data slice in the multiple point cloud data slices to obtain environmental semantic encoding data corresponding to each point cloud data slice. In this embodiment, crowdsourced point cloud data and crowdsourced device data can be aligned and sliced. During encoding, the slices can be encoded at the granularity. Therefore, the appropriate slice size can be selected according to the actual computing power, thereby dividing the crowdsourced data into granularities that are compatible with the instance computing power, so as to improve the utilization rate of the device's computing resources and improve the device's inference efficiency.

[0009] In one possible implementation, the aforementioned inputting device path encoding data and environmental semantic encoding data into a neural network to output a first drivable area may include: inputting environmental semantic encoding data corresponding to each point cloud data slice and device path encoding data corresponding to each device data slice into a neural network to obtain a region corresponding to each slice; and fusing the regions corresponding to each slice to obtain a first drivable area.

[0010] In this embodiment of the application, after encoding at the slice level, the passable area in each slice can be inferred at the slice level. By fusing the areas in all slices, the drivable area of ​​the device can be obtained, thereby realizing the reasoning of the passable area of ​​the device at the slice level.

[0011] In one possible implementation, the aforementioned vector encoding includes encoding a vector representing a device path, where the vector includes at least one of position, velocity, or direction. In this application embodiment, the vector representing the path of a crowdsourced device can be encoded into model-recognizable coded data, which can be used to identify accessible areas for the device in the environment.

[0012] In one possible implementation, the aforementioned semantic encoding includes encoding at least one of point clouds, point cloud locations or semantic categories, or relationships between point clouds. In this application, crowdsourced point cloud data can be semantically encoded to infer the entities represented by each point cloud, thereby accurately inferring road structures or obstacles through the relationships between multiple point clouds.

[0013] In one possible implementation, the aforementioned method further includes: obtaining a second drivable area, the second drivable area being an area determined based on data from the first device; and fusing the first drivable area and the second drivable area to obtain a third drivable area.

[0014] In this embodiment of the application, a second drivable area determined by real-time data can also be obtained. By fusing the first drivable area and the second drivable area, a more accurate third drivable area can be obtained, avoiding situations where drivable areas are impassable due to blind spots or perception errors in real-time perception, and improving the accuracy of the final output drivable area.

[0015] In one possible implementation, the aforementioned method further includes: obtaining device path information for a first time period, the first time period including a preset time period closest to the current time; updating a first drivable area based on the device path information, and using the updated area as the new first drivable area.

[0016] In this embodiment, the drivable area is corrected by using the latest traffic flow on the road. For example, some areas may be identified as impassable, but may actually be drivable, thereby improving the accuracy of the final output drivable area.

[0017] In one possible implementation, the aforementioned first and second devices can be vehicles, or other intelligent devices such as robots or drones. Therefore, in this embodiment, data from crowdsourced vehicles can be used to plan more accurate drivable areas for the vehicle.

[0018] Secondly, this application provides a drivable area planning device, applied to a first device, comprising:

[0019] The acquisition module is used to acquire crowdsourced device data and crowdsourced collection data. The crowdsourced device data includes data from multiple second devices, and the crowdsourced collection data includes data collected by sensors in multiple second devices.

[0020] The path encoding module is used to encode crowdsourced device data to obtain device path encoding data;

[0021] The environment coding module is used to encode crowdsourced data to obtain environmental semantic coding data.

[0022] The planning module is used to input device path coding data and environmental semantic coding data into the neural network and output a first drivable area. The first drivable area is the area where the first device can drive. The neural network is used to extract the correlation between environmental semantic coding data and device path coding data and determine the first drivable area based on the correlation.

[0023] The effects achieved by the second aspect or any optional implementation of the second aspect can be referred to the description of the first aspect or any optional implementation of the first aspect, and will not be repeated hereafter.

[0024] In one possible implementation, the aforementioned crowdsourced data collection includes crowdsourced point cloud data, which is data obtained by fusing data collected by sensors of multiple second devices, and crowdsourced device data package crowdsourced vehicle flow data, which includes path information of multiple second devices.

[0025] In one possible implementation, the aforementioned path encoding module is specifically used to perform vector encoding on the crowdsourced vehicle flow data to obtain device path encoding data;

[0026] The environment coding module is specifically used to perform semantic coding on crowdsourced point cloud data to obtain environment semantic coding data.

[0027] In one possible implementation, the aforementioned apparatus further includes:

[0028] The alignment module is used to align crowdsourced point cloud data and crowdsourced device data in the same space to obtain aligned crowdsourced point cloud data and aligned crowdsourced device data.

[0029] The slicing module is used to slice the aligned crowdsourced point cloud data and the aligned crowdsourced device data in the same space to obtain multiple point cloud data slices and multiple device data slices.

[0030] The path encoding module is specifically used to perform vector encoding on each device data slice in multiple device data slices to obtain the device path encoding data corresponding to each device data slice.

[0031] The environment coding module is specifically used to perform semantic coding on each point cloud data slice in multiple point cloud data slices to obtain the environment semantic coding data corresponding to each point cloud data slice.

[0032] In one possible implementation, the aforementioned planning module is specifically used to: input the environmental semantic encoding data corresponding to each point cloud data slice and the device path encoding data corresponding to each device data slice into a neural network to obtain the region corresponding to each slice; and fuse the regions corresponding to each slice to obtain the first drivable region.

[0033] In one possible implementation, the aforementioned vector encoding includes encoding a vector representing a device path, the vector including at least one of position, velocity, or direction.

[0034] In one possible implementation, the aforementioned semantic encoding includes encoding at least one of point clouds, the location semantic category of point clouds, or the association between point clouds.

[0035] In one possible implementation, the aforementioned apparatus further includes a fusion module for: acquiring a second drivable area, the second drivable area being an area determined based on data from the first device; and fusing the first drivable area and the second drivable area to obtain a third drivable area.

[0036] In one possible implementation, the aforementioned device further includes a conflict suppression module, configured to: acquire device path information for a first time period, the first time period including a preset time period closest to the current time; update a first drivable area based on the device path information, and use the updated area as the new first drivable area.

[0037] In one possible implementation, the aforementioned first device and second device are vehicles.

[0038] Thirdly, this application provides a drivable area planning device, including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to execute processing-related functions in the drivable area planning method shown in any of the first aspects above. Optionally, the drivable area planning device may be a chip.

[0039] Fourthly, embodiments of this application provide a vehicle including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to perform processing-related functions in the method shown in any of the first aspects above.

[0040] Fifthly, embodiments of this application provide a digital processing chip or chip, the chip including a processing unit and a communication interface, the processing unit obtaining program instructions through the communication interface, the program instructions being executed by the processing unit, the processing unit being used to perform processing-related functions as described in the first aspect or any optional embodiment of the first aspect.

[0041] In a sixth aspect, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any optional implementation thereof.

[0042] In a seventh aspect, embodiments of this application provide a computer program product comprising a computer program / instructions, which, when executed by a processor, causes the processor to perform the method described in the first aspect or any optional implementation thereof. Attached Figure Description

[0043] Figure 1 is a schematic diagram of the architecture of the method provided in this application;

[0044] Figure 2 is a structural schematic diagram of a vehicle provided in an embodiment of this application;

[0045] Figure 3 is a flowchart illustrating a drivable area planning method provided in an embodiment of this application;

[0046] Figure 4 is a flowchart illustrating another drivable area planning method provided in an embodiment of this application;

[0047] Figure 5 is a flowchart illustrating another drivable area planning method provided in an embodiment of this application;

[0048] Figure 6 is a flowchart illustrating another drivable area planning method provided in an embodiment of this application;

[0049] Figure 7 is a schematic diagram of a neural network structure provided in an embodiment of this application;

[0050] Figure 8 is a schematic diagram of an application scenario of the method provided in this application;

[0051] Figure 9 is a schematic diagram of another application scenario of the method provided in this application;

[0052] Figure 10 is a schematic diagram of another application scenario of the method provided in this application;

[0053] Figure 11 is a structural schematic diagram of a drivable area planning device provided in an embodiment of this application;

[0054] Figure 12 is a structural schematic diagram of another drivable area planning device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0056] First, for ease of understanding, some concepts or categories involved in the embodiments of this application will be explained below.

[0057] (1) transformer

[0058] A transformer architecture is a feature extraction network that includes both an encoder and a decoder. Of course, in some cases, a transformer architecture may not include an encoder but may include a decoder.

[0059] Encoder: Learns features, such as pixel features, in the global receptive field through self-attention.

[0060] Decoder: Learns the features of the desired module, such as the features of the output box, through self-attention and cross-attention.

[0061] For example, the structure of a Transformer layer in an existing scheme may include a multi-head attention network and a feedforward network module. Taking natural language processing as an example, the multi-head attention network obtains corresponding weight values ​​by calculating the relevance between words, thus obtaining context-related word representations, which is the core part of the Transformer structure. The feedforward network further transforms the obtained representations to obtain the final output of the Transformer layer. In addition to these two important components, residual layers (ADD) and linear normalization (Norm) are also stacked on these two components to optimize the output of the Transformer layer.

[0062] (2) Attention mechanism

[0063] Attention mechanisms can quickly extract important features from sparse data. They provide an effective modeling approach for capturing global contextual information through QKV (Queries, Keys, Values). Assuming the input is Q(query), and the context is stored as key-value pairs (K, V), then the attention mechanism is essentially a mapping function from the query to a series of key-value pairs. The essence of the attention function can be described as a mapping from a query to a series of (key-value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in (K, V) form, then attention performs addressing by calculating the similarity between Q and K. The similarity calculated between Q and K reflects the importance of the extracted V values, i.e., the weights, and then a weighted sum is obtained to obtain the final feature value.

[0064] Attention calculation mainly consists of three steps. The first step is to calculate the similarity between the query and each key to obtain weights. Common similarity functions include dot product, concatenation, and perceptron. The second step typically uses a softmax function (which can normalize the weights, resulting in a probability distribution where the sum of all weight coefficients is 1, and also highlights the weights of important elements) to normalize these weights. Finally, the weights and their corresponding key values ​​are weighted and summed to obtain the final feature value. The specific calculation formula is as follows:

[0065] Where d is the dimension of matrix Q,K.

[0066] Furthermore, attention includes self-attention and cross-attention. Self-attention can be understood as a special type of attention where the inputs to the QKV features are consistent. Cross-attention, on the other hand, involves inconsistent inputs to the QKV features. Attention integrates the queried features as updated values ​​for the current features using the similarity between features (e.g., inner product) as weights. Self-attention is attention extracted based on the attention drawn from the feature map itself.

[0067] For convolutional networks, the kernel size limits the receptive field, often requiring multiple layers to focus on the entire feature map. Self-attention, on the other hand, has the advantage of global focus; it can obtain global spatial information about the feature map through simple lookups and assignments.

[0068] Secondly, the method provided in this application can be applied to various smart terminals, such as vehicles, robots, drones, or other electronic devices. For example, it can be applied to intelligent driving functions of vehicles, navigation of robots, and offline scenarios such as drone navigation.

[0069] For example, it can be applied to obstacle avoidance scenarios when there are emergency traffic situations in front of a vehicle, such as when a vehicle is overturned or out of control, to plan a safer driving path for the vehicle; or it can be deployed in a robot to plan a more reasonable obstacle avoidance path for the robot when there are moving obstacles in front of it. This application exemplifies the deployment of the method provided in the embodiments of this application in a vehicle as an example. The vehicle mentioned below can also be replaced by a robot, drone, or other electronic device, etc., which will not be described in detail below.

[0070] First, the method provided in this application can be applied to systems capable of collecting crowdsourced vehicles. For example, the system architecture used in the embodiments of this application can be as shown in Figure 1. This system architecture may include autonomous vehicles 100, other crowdsourced vehicles 200, and a cloud platform 300.

[0071] The number of crowdsourced vehicles 200 can be one or more. During operation, these vehicles can collect data using sensors, and each vehicle can upload this data to the cloud 300. The crowdsourced data can include data collected by the vehicles themselves, as well as data generated by the vehicles themselves. Typically, in real-world applications, a large number of crowdsourced vehicles can be deployed, and their data can be uploaded to the cloud 300 with user permission. In other words, after user permission, the vehicles upload information to the cloud 300, where the cloud aggregates the data uploaded by the crowdsourced vehicles 200 and transmits it to the self-owned vehicle 100.

[0072] It should be noted that in the following embodiments of this application, the first device mentioned includes the vehicle deployed by the method provided in the embodiments of this application, or a self-driving vehicle, and the second device includes crowdsourced vehicles, which will not be described in detail below.

[0073] Further, referring to Figure 2, which is a structural schematic diagram of a vehicle provided in an embodiment of this application, the vehicle 100 can be configured in an intelligent driving mode. For example, the vehicle 100 can control itself while in intelligent driving mode, and can determine the current state of the vehicle and its surrounding environment through human operation, determine whether there are obstacles in the surrounding environment, and control the vehicle 100 based on the obstacle information. When the vehicle 100 is in intelligent driving mode, the vehicle 100 can also be set to operate without human interaction.

[0074] Figure 2 is a functional block diagram of a vehicle 100 provided in an embodiment of this application. The vehicle 100 can be configured to a full or partial intelligent driving mode. For example, the vehicle 100 can obtain environmental information about its surroundings through the perception system 120, and obtain an intelligent driving strategy based on the analysis of the surrounding environmental information to achieve full intelligent driving, or present the analysis results to the user to achieve partial intelligent driving.

[0075] Vehicle 100 may include various subsystems, such as an infotainment system 110, a perception system 120, a decision control system 130, a drive system 140, and a computing platform 150. Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and component of vehicle 100 may be interconnected via wired or wireless means.

[0076] In some embodiments, the infotainment system 110 may include a communication system 111, an entertainment system 112, and a navigation system 113.

[0077] Communication system 111 may include wireless communication system 111, which can communicate wirelessly with one or more devices directly or via a communication network. For example, wireless communication system 111 may use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. Wireless communication system 111 may communicate using WiFi and wireless local area network (WLAN). In some embodiments, wireless communication system 111 may communicate directly with devices using infrared links, Bluetooth, or ZigBee. Wireless communication system 111 may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0078] In this embodiment of the application, communication between the vehicle and the cloud can be achieved through the communication system 111.

[0079] The entertainment system 112 may include a central control screen, a microphone, and speakers. Users can listen to the radio and play music within the vehicle using the entertainment system 112; or connect their mobile phones to the vehicle and project their screens onto the central control screen, which may be touch-sensitive, allowing users to operate the system. In some cases, the microphone can capture the user's voice signal, and analysis of this signal can enable the user to control certain aspects of the vehicle 100, such as adjusting the interior temperature. In other cases, music can be played to the user through the speakers.

[0080] The interactive interface provided in this application embodiment can be displayed in the infotainment system. Specifically, it can be displayed on devices in the vehicle, such as the central control screen, passenger screen, instrument panel screen, and head-up display, which can be used to display interactive interfaces to users. Users can perform interactive operations based on the interactive interface, which may include, but are not limited to, touch interaction, voice interaction, and gesture interaction, depending on the actual interaction methods supported by the vehicle.

[0081] The navigation system 113 may include map services to provide navigation for the vehicle 100, and the navigation system 113 may be used in conjunction with the vehicle's global positioning system 121 and inertial measurement unit 122. The map may be a two-dimensional map, a high-precision map, or a map constructed based on data collected during the vehicle's operation.

[0082] The perception system 120 may include several sensors for sensing information about the environment surrounding the vehicle 100. For example, the perception system 120 may include a global positioning system 121 (which may include a global navigation satellite system (GNSS), specifically GPS, BeiDou, or other positioning systems), an inertial measurement unit (IMU) 122, a lidar (LiDAR) 123, a millimeter-wave radar 124, an ultrasonic radar 125, and a camera device 126. The perception system 120 may also include sensors from the internal systems of the monitored vehicle 100 (e.g., an in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of the vehicle 100. The multimodal data from the vehicle mentioned below may include data collected by various units in the perception system 120, such as point cloud data collected by lidar or environmental images collected by the camera device.

[0083] The Global Positioning System 121 can be used to determine the geographical location of vehicle 100.

[0084] The inertial measurement unit 122 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In some embodiments, the inertial measurement unit 122 may be a combination of an accelerometer and a gyroscope.

[0085] The lidar 123 can use lasers to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the lidar 123 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components.

[0086] The millimeter-wave radar 124 can use radio signals to sense objects in the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing objects, the millimeter-wave radar 124 can also be used to sense the speed and / or direction of travel of objects.

[0087] The ultrasonic radar 125 can use ultrasonic signals to sense objects around the vehicle 100.

[0088] The camera device 126 can be used to capture image information of the surrounding environment of the vehicle 100. The camera device 126 may include a monocular camera, a binocular camera, a structured light camera, and a panoramic camera, etc. The image information acquired by the camera device 126 may include still image information or video stream information.

[0089] The decision control system 130 includes a computing system 131 that analyzes and makes decisions based on information acquired by the sensing system 120. The decision control system 130 also includes a vehicle controller 132 that controls the power system of the vehicle 100, and a steering system 133, a throttle 134, and a braking system 135 for controlling the vehicle 100.

[0090] The computing system 131 can process and analyze various information acquired by the perception system 120 to identify targets, objects, and / or features in the environment surrounding the vehicle 100. The targets may include pedestrians or animals, and the objects and / or features may include traffic signals, road boundaries, and obstacles. The computing system 131 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other techniques. In some embodiments, the computing system 131 may be used to map the environment, track objects, estimate object speeds, etc. The computing system 131 can analyze the acquired information and derive a control strategy for the vehicle.

[0091] The vehicle controller 132 can be used to coordinate the control of the vehicle's power battery and engine 141 to improve the power performance of the vehicle 100.

[0092] The steering system 133 can be used to adjust the forward direction of the vehicle 100. For example, in one embodiment, it can be a steering wheel system.

[0093] The throttle 134 is used to control the operating speed of the engine 141 and thus the speed of the vehicle 100.

[0094] Braking system 135 is used to control the deceleration of vehicle 100. Braking system 135 can use friction to slow down the rotational speed of wheel 144. In some embodiments, braking system 135 can convert the kinetic energy of wheel 144 into electric current. Braking system 135 may also take other forms to slow down the rotational speed of wheel 144 to control the speed of vehicle 100.

[0095] The drive system 140 includes components that provide powered motion to the vehicle 100. In one embodiment, the drive system 140 may include an engine 141, an energy source 142, a transmission system 143, and wheels 144. The engine 141 may be an internal combustion engine, an electric motor, an air-compressed engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. The engine 141 converts the energy source 142 into mechanical energy.

[0096] Examples of energy sources 142 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 142 may also provide energy to other systems of vehicle 100.

[0097] The drivetrain 143 transmits mechanical power from the engine 141 to the wheels 144. The drivetrain 143 may include a gearbox, a differential, and a drive shaft. In one embodiment, the drivetrain 143 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 144.

[0098] Some or all of the functions of vehicle 100 are controlled by computing platform 150. Computing platform 150 may include at least one processor 151, which can execute instructions 153 stored in a non-transitory computer-readable medium such as memory 152. In some embodiments, computing platform 150 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner.

[0099] Processor 151 can be any conventional processor, such as a commercially available CPU. Alternatively, processor 151 may also include a graphics processing unit (GPU), a field-programmable gate array (FPGA), a system-on-chip (SoC), an application-specific integrated circuit (ASIC), or a combination thereof. Processor 151 can be located on a device remote from the vehicle and can communicate wirelessly with the vehicle.

[0100] In some embodiments, memory 152 may contain instructions 153 (e.g., program logic) that can be executed by processor 151 to perform various functions of vehicle 100. Memory 152 may also contain additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the infotainment system 110, perception system 120, decision control system 130, and drive system 140.

[0101] In the method provided in this application embodiment, the steps performed by the vehicle can be executed by the computing platform 150. For example, the processor 151 can read the program stored in the memory 152, display the interactive interface on the display screen, and execute subsequent processes based on the user's interactive operation.

[0102] In addition to instruction 153, memory 152 may also store data such as road maps, route information, vehicle position, direction, speed, and other similar vehicle data, as well as other information. It may also store higher-precision maps obtained by the methods provided in this application embodiment. This information can be used by vehicle 100 and computing platform 150 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0103] The computing platform 150 can control the functions of the vehicle 100 based on inputs received from various subsystems, such as the drive system 140, the perception system 120, and the decision control system 130. For example, the computing platform 150 can utilize inputs from the decision control system 130 to control the steering system 133 to avoid obstacles detected by the perception system 120. In some embodiments, the computing platform 150 is operable to provide control over many aspects of the vehicle 100 and its subsystems.

[0104] Alternatively, one or more of these components may be installed separately from or associated with vehicle 100. For example, memory 152 may exist partially or completely separately from vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0105] Optionally, the above components are just an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 2 should not be construed as a limitation on the embodiments of this application.

[0106] The aforementioned vehicle 100 can be any vehicle or vehicle-mounted terminal capable of intelligent driving, such as a car, truck, motorcycle, bus, ship, airplane, helicopter, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, or train. This application embodiment does not impose any special limitations on this type of vehicle.

[0107] In intelligent driving functions, it is usually necessary to plan a safe driving area for the vehicle. Planning a driving area for a vehicle typically relies on the accuracy and robustness of the sensors and perception algorithms installed in the vehicle. However, blind spots or failures in recognizing road elements may lead to inaccurate identification of obstacles or road structures in the vehicle's environment, thus making it impossible to plan a more accurate driving area for the vehicle.

[0108] Therefore, this application provides a drivable area planning method that can utilize crowdsourced data to plan drivable areas, thereby planning more accurate and safer drivable areas for vehicles.

[0109] Referring to Figure 3, a flowchart of a drivable area planning method provided in an embodiment of this application is shown below.

[0110] 301. Obtain crowdsourcing device data and crowdsourcing collection data.

[0111] The crowdsourced device data includes data on the travel paths of multiple secondary devices, such as the location, speed, direction of movement, or acceleration of the secondary devices.

[0112] Crowdsourced data collection includes data collected by sensors in multiple secondary devices, specifically including images, point clouds, or raster data. For example, the crowdsourced data may specifically include bird's-eye view (BEV) images, 2D occupancy raster, 3D occupancy raster, multi-view images, binocular images, or depth images collected or perceived during vehicle operation.

[0113] Based on the architecture shown in Figure 1 above, the crowdsourcing device data and crowdsourcing collection data can specifically originate from the cloud. For example, if the method provided in this embodiment is deployed in a first device, this first device is connected to the cloud, and the cloud is connected to multiple crowdsourcing devices, i.e., second devices. These multiple crowdsourcing devices can upload data to the cloud, and the cloud stores the data uploaded by the crowdsourcing devices. When the first device requests crowdsourcing data from the cloud, the cloud can send the stored crowdsourcing data to the first device.

[0114] This application exemplifies the use of a crowdsourced vehicle as the crowdsourced device. The crowdsourced device data may include data about the crowdsourced vehicle, such as its position, speed, direction of movement, or acceleration. The crowdsourced data collection may include data collected by sensors in the crowdsourced vehicle, such as images, depth images, or point cloud data collected by the crowdsourced vehicle. This application exemplifies the use of point cloud data in the crowdsourced data collection as an example. The crowdsourced point cloud data referred to below can also be replaced with other types of data, which will not be elaborated further below.

[0115] Optionally, crowdsourced data can be preprocessed, such as by desensitizing or filtering, to obtain usable crowdsourced data.

[0116] Optionally, before encoding the crowdsourced device data and the crowdsourced collected data, the crowdsourced point cloud data and the crowdsourced device data can be aligned in the same space to obtain aligned crowdsourced point cloud data and aligned crowdsourced device data. For example, the crowdsourced device data and the crowdsourced point cloud data can both be mapped to the same space, and the data mapped to this space can be used as new crowdsourced device data and crowdsourced point cloud data.

[0117] 302. Encode the crowdsourced equipment data to obtain equipment path code data.

[0118] The information of crowdsourced vehicles included in the crowdsourced equipment data can be encoded to obtain coded data that can represent the driving path of the crowdsourced vehicles, namely, equipment path coded data.

[0119] Optionally, when encoding crowdsourced vehicle data, the aligned crowdsourced vehicle flow data can be vector-encoded to obtain vehicle path encoding data. Specifically, vectors in the crowdsourced vehicle data can be encoded, such as the vehicle's position, speed, direction of movement, or acceleration. The resulting device path encoding data can be used to represent the driving path of the crowdsourced vehicles.

[0120] Typically, when crowdsourced vehicles travel on roads, they usually travel within drivable roads. The travel paths of these vehicles can be used to identify which areas in the environment are drivable roads, allowing for the subsequent determination of the drivable areas for individual vehicles.

[0121] 303. Encode the crowdsourced data to obtain environmental semantic encoded data.

[0122] The crowdsourced data may include environmental information collected or sensed during the operation of the crowdsourced vehicle. In this embodiment, the environmental information collected or sensed by the crowdsourced vehicle can be encoded to obtain encoded data that can be used to represent the environment of the area through which the crowdsourced vehicle has traveled.

[0123] For example, taking crowdsourced data collection including crowdsourced point cloud data as an example, the point cloud information in the crowdsourced point cloud data is encoded to obtain environmental semantic encoded data that can be used to represent the environment.

[0124] Optionally, semantic encoding can be performed on the crowdsourced point cloud data to obtain environmental semantic encoded data. Specifically, this semantic encoding can include encoding the semantics of the point clouds in the crowdsourced point cloud data, such as encoding information such as the position (e.g., coordinates or height) or category of each point cloud, thereby obtaining environmental semantic encoded data that can represent the environment.

[0125] For example, data collected by crowdsourced vehicles can form information about obstacles or road structures in the environment. Through sensors installed in these vehicles, the resulting crowdsourced point cloud data aggregates the perception results from multiple vehicles. For instance, regarding road structures, the data collected by crowdsourced vehicles can yield a large number of point clouds related to the road environment. These point clouds can form information about environmental elements, and each point cloud's data can include coordinates or category information. The coordinates of a point cloud can include its spatial location, specifically two-dimensional, three-dimensional, or multi-dimensional coordinates. When the coordinates of a point cloud are three-dimensional or higher, they can include the point cloud's height or other information. The category of a point cloud can be the category of the target it represents, such as, but not limited to, curbs, lane lines, road signs, intersections, or traffic lights.

[0126] Furthermore, to accommodate the computing resources or computational timeliness of the computing devices deployed using the method provided in this application, the computational granularity can be reduced by slicing when encoding crowdsourced device data and crowdsourced point cloud data. For example, if the computing device has superior computing power or abundant computing resources and can process larger amounts of data, it can be sliced ​​into larger granularities; conversely, if the computing device has lower computing power or fewer computing resources, it can be sliced ​​into smaller granularities.

[0127] For example, optionally, after aligning the crowdsourced point cloud data and crowdsourced device data in the same space to obtain aligned crowdsourced point cloud data and aligned crowdsourced device data, the aligned crowdsourced point cloud data and aligned crowdsourced device data can be sliced ​​in the same space to obtain multiple point cloud data slices and multiple device data slices; each device data slice in the multiple device data slices is vector encoded to obtain device path encoded data corresponding to each device data slice; and each point cloud data slice in the multiple point cloud data slices is semantically encoded to obtain environmental semantic encoded data corresponding to each point cloud data slice.

[0128] Therefore, in this embodiment of the application, crowdsourcing device data and crowdsourcing point cloud data can be sliced ​​into smaller computational granularities to adapt to the computing power of computing devices or meet the required failure requirements.

[0129] 304. Input the device path encoding data and environmental semantic encoding data into the neural network and output the first drivable area.

[0130] The first drivable area is the first device, that is, the area where the vehicle can drive. The neural network is used to extract the correlation between environmental semantic coding data and device path coding data, and to determine the first drivable area based on the correlation.

[0131] Typically, the trajectory of a crowdsourced vehicle flow can represent the drivable area of ​​a vehicle in the actual environment, and the data collected by the crowdsourced vehicles can represent information such as road structure or obstacles in the environment. Therefore, in the embodiments of this application, the surrounding environment of a vehicle can be inferred from the data uploaded by the crowdsourced vehicles, which can achieve a more accurate understanding of the environment around the vehicle. For example, it can more accurately infer obstacles or road structures such as structured or unstructured roads around the vehicle, and thus determine the drivable area in the environment very accurately.

[0132] Optionally, device path information for the first time period can also be obtained, including the preset time period closest to the current time. The first drivable area is then updated based on the device path information, and the updated area is used as the new first drivable area. This can be understood as selecting the crowdsourced vehicle flow path within the most recent time period to correct the first drivable area, avoiding situations where changes in road structure or obstacles cause some areas to be impassable, or where new roads are built or new drivable areas exist, thus ensuring that the obtained first drivable area better matches the real-time environment.

[0133] Optionally, a second drivable area can be obtained, which is an area determined based on data from the first device. The first and second drivable areas are then merged to obtain a third drivable area. This can be understood as using data collected by vehicle sensors to identify the drivable area, i.e., the second drivable area, which can originate from the vehicle itself or from the cloud. Subsequently, the first and second drivable areas are merged, and the merged area is referred to as the third drivable area. The second drivable area can be updated using the first drivable area obtained from crowdsourced data, or vice versa, depending on the specific application scenario.

[0134] In addition to fusing the real-time predicted second drivable area with the drivable area in the cloud (referred to as post-fusion), the vehicle can also directly use the second drivable area as one of the inputs to the vehicle prediction model, along with the vehicle sensor data, and directly input it into the vehicle prediction model (i.e. the model deployed on the vehicle for predicting drivable areas) to output a larger real-time drivable area. This method of obtaining the third drivable area can be referred to as pre-fusion.

[0135] The foregoing has described the method flow provided by the embodiments of this application. The following is a more detailed description of the method flow provided by the embodiments of this application in conjunction with specific application scenarios.

[0136] Referring to Figure 4, a flowchart of another drivable area planning method provided in this application embodiment is shown below.

[0137] 401. Obtain crowdsourced point cloud data.

[0138] Crowdsourced point cloud data refers to information collected or identified by crowdsourced vehicles, including information about road elements or obstacles in the environment. For example, images, laser point clouds, or perceived road elements or obstacles collected by crowdsourced vehicles can be mapped into three-dimensional or two-dimensional space. Here, we take mapping to three-dimensional space as an example to obtain crowdsourced point cloud data.

[0139] The crowdsourced point cloud can be obtained directly using a LiDAR (Light Detection and Ranging) system installed in a crowdsourced vehicle, or it can be a 3D mapping point cloud obtained by fusing data such as BEV images, 2D occupancy grids, 3D occupancy grids, multi-view images, binocular images, or depth images.

[0140] 402. Obtain crowdsourced vehicle flow data.

[0141] Crowdsourced vehicle flow data, also known as the aforementioned crowdsourced device data, can represent the driving trajectory of crowdsourced vehicles, including information such as the location, direction, speed, or acceleration of the crowdsourced vehicles.

[0142] Typically, vehicle trajectory information over a period of time can be collected through methods such as location tracking of crowdsourced vehicle traffic and vehicle identification within crowdsourced data, such as collecting traffic flow data from the past week or month. Generally, more complete or more continuous trajectories can be selected as the trajectory for crowdsourced vehicle traffic.

[0143] 403. Perform point cloud registration and pose optimization.

[0144] To further improve the accuracy of subsequent processing, the crowdsourced point cloud data and crowdsourced vehicle flow data can be aligned in the same space and then further calibrated.

[0145] Specifically, the first step is to map the crowdsourced point cloud data and the crowdsourced vehicle flow data to the same space, which can be a two-dimensional space, a three-dimensional space, or a space with more dimensions. Alternatively, the coordinates of the crowdsourced point cloud data and the crowdsourced vehicle data can be mapped to align the coordinates of the crowdsourced point cloud data and the crowdsourced vehicle flow data.

[0146] Subsequently, further optimization was performed on the crowdsourced point cloud data and crowdsourced vehicle flow data in the same space.

[0147] For crowdsourced point cloud data, point cloud registration can be performed, which means merging the point cloud data collected by crowdsourced vehicles into the same space. Specifically, registration can be performed using registration algorithms, such as the three-point method, the four-point method, or the least squares method.

[0148] For crowdsourced vehicle traffic data, the position and orientation of the crowdsourced vehicles can be optimized, such as adjusting the vehicle's orientation to make it more reasonable, or adjusting the vehicle's steering angle to better conform to kinematic constraints.

[0149] Step 403 is optional. For example, in some scenarios, if the crowdsourced vehicle flow data and the crowdsourced point cloud data have already been aligned in the cloud, then no further alignment is needed here.

[0150] 404. Slice.

[0151] For crowdsourced point cloud data and crowdsourced vehicle flow data, in order to reduce the computational granularity of computing devices, the data can be sliced ​​to reduce the amount of computation in a single calculation process.

[0152] Specifically, crowdsourced point cloud data and crowdsourced vehicle flow data can be sliced ​​in the same space. The granularity of the slice can be determined by the computing performance of the computing device. For example, if the computing power of the computing device is low, or if the algorithm required by the data is high, the slice can be made with a smaller granularity to reduce the computing resources required for a single calculation.

[0153] For example, as shown in Figure 5, crowdsourced point cloud data and crowdsourced vehicle flow data can be sliced ​​in the same space. That is, when slicing, crowdsourced point cloud data and crowdsourced vehicle flow data are sliced ​​at the same location, thereby maintaining the spatial consistency between point cloud data and vehicle flow data within the same slice.

[0154] After slicing, the drivable area can be planned according to the granularity of the slices.

[0155] For example, referring to Figure 6, after slicing the data, semantic encoding and vector encoding are performed at the slice granularity. Steps 405 to 411 are described below with reference to Figure 6.

[0156] 405. Merging point cloud maps.

[0157] Point clouds collected by crowdsourced vehicles are fused to form a point cloud map. For example, since there are usually a large number of crowdsourced vehicles, there may be differences between the collected point cloud data. After aligning the point clouds in the same space, the point clouds collected by multiple crowdsourced vehicles can be further fused to obtain more concise and accurate point cloud data.

[0158] 406. Semantic Coding

[0159] Semantic encoding is performed on point cloud data, such as encoding the location, height, or category of the point cloud, to obtain the environmental semantic encoding data of the current slice. For example, the category might represent a vehicle, lane line, or road sign. Essentially, semantic encoding maps the point cloud into a two-dimensional space to facilitate subsequent model processing.

[0160] For example, semantic encoding can be represented as: feat pillar (x,y)=f1(height1,class1…), where height1 represents the point cloud height and class1 represents the point cloud class.

[0161] 407. Integrating traffic flow trajectories.

[0162] For crowdsourced vehicle traffic, the number of crowdsourced vehicles is usually large, and the trajectories of crowdsourced vehicle traffic may not completely overlap. Therefore, the trajectories of multiple crowdsourced vehicles can be merged to obtain a complete and wider coverage of the area occupied by the vehicle traffic trajectory.

[0163] 408. Vector encoding.

[0164] For traffic flow trajectories, vector encoding can be performed, that is, encoding the vectors representing the traffic flow to obtain the traffic flow path encoding data for the current slice. For example, the position, speed, or direction of vehicles can be encoded to obtain the encoded data representing the traffic flow trajectory. It can be understood that the position, speed, direction, etc. of the traffic flow trajectory can be mapped to a two-dimensional space corresponding to the semantic encoding of the point cloud.

[0165] 409. Input to transformer.

[0166] After obtaining the environmental semantic coding data and traffic path coding data at the slice granularity, the environmental semantic coding data and traffic path coding data of the current slice can be fused. The fusion method can be splicing or other fusion methods. The fused coding data is then input into the transformer to output the drivable area in the current slice.

[0167] Specifically, the transformer can output information about the drivable area, such as the boundaries, center point coordinates, length, width, or area size of the drivable area.

[0168] For example, the structure of a transformer can be as shown in Figure 7, which can include a self-attention module and a cross-attention module. The self-attention module can be used to extract the relationship between point cloud and traffic flow in the space from the input features. The cross-attention module can be used to extract the relationship between different encoded data and the relationship between the drivable area to output the complete drivable area.

[0169] 410. Conflict suppression.

[0170] Typically, the data uploaded by crowdsourced vehicles may include historical data with a long time interval from the current data, which can change in real-world scenarios, such as lane position or lane markings. In this application's implementation, traffic flow trajectory information from the most recent time period can be used to suppress conflicts in drivable areas, such as by increasing or decreasing the drivable area.

[0171] For example, if environmental changes occur, such as a lane that was closed six months ago but reopens this week, traffic flow data from the past week can be collected to determine if the area is passable and thus designated as a drivable zone. Conversely, if a section of road was under maintenance and temporarily closed last week, crowdsourced traffic flow data from that week can be used to determine if the area is impassable. If the drivable zone includes this section, it can be adjusted to prevent vehicles from traveling on impassable sections.

[0172] Therefore, in this embodiment, crowdsourced vehicle flow information within the latest time period can be used to suppress conflicts in drivable areas and improve the accuracy of the final output drivable area.

[0173] 411. Slice and aggregate to output the drivable area.

[0174] After encoding each slice and outputting its corresponding drivable area, the drivable areas of corresponding points in each slice can be aggregated to output a more complete drivable area.

[0175] During the slicing stage, since the crowdsourced data has been segmented, after inferring the drivable area at the granularity of the slice, the drivable areas corresponding to each slice need to be aggregated to generate a more complete drivable area in space.

[0176] Furthermore, it can display drivable areas to users, or generate control signals based on drivable areas to control the vehicle to drive in those areas, thereby enabling the vehicle to drive in drivable areas, avoiding blind spots and improving driving safety.

[0177] Therefore, in this embodiment, by fusing the obstacle point cloud inferred from crowdsourced vehicles with historical traffic flow, a stable and complete boundary of the drivable area in the environment can be inferred, avoiding route decision errors caused by missing obstacles. Furthermore, fusing crowdsourced point clouds can address the limited field of view of a single-vehicle sensor, making obstacle boundaries more complete and preventing the influence of a single-vehicle sensor's field of view on route planning at higher speeds or over longer distances. By fusing historical traffic flow, the complete drivable area boundary can be inferred based on the position, direction, and continuity of the traffic flow, reducing the influence of unstable obstacle boundaries and minimizing route decision errors caused by missing obstacle detection. By detecting conflicts between the latest traffic flow and the obstacle point cloud (e.g., the input point cloud may contain noise or have missing road surfaces), failure information and detection errors in historical detections can be identified, reducing interference from historical erroneous data and the impact of sensor detection errors. This can reduce the impact of actual changes in obstacles, road structure changes, or sensor observation errors on route decision-making. Furthermore, combining historical information with real-time information perceived by the vehicle can enhance the vehicle's field of vision in long-distance scenarios, enable reasonable and effective path planning in advance, expand the real-time perception field of a single vehicle, and provide accurate boundary information of drivable areas at a distance, especially on semi-structured and unstructured roads.

[0178] For example, roads can generally be divided into structured roads and unstructured roads, as shown in Figure 8. Structured roads include lane lines or road markings, while unstructured roads are drivable roads without lane lines or road markings. Typically, structured roads have complete and stable boundaries, while unstructured roads may lack complete and continuous boundary lines, making the identification of unstructured roads very difficult. For example, as shown in Figure 9, if the vehicle's perception data contains missing obstacles or observation errors, it may lead to a failure in recognizing the road structure. In this embodiment, information collected by crowdsourced vehicle traffic trajectories and crowdsourced vehicles is used to more accurately infer drivable areas, thus planning more accurate drivable areas for the vehicle. As shown in Figure 10, the method provided in this embodiment can accurately infer missing obstacles in the vehicle's perception data, thereby more accurately determining the boundaries of the drivable area.

[0179] The method flow provided in this application has been described above. The structure of the apparatus for performing the above method flow is described below.

[0180] Referring to Figure 11, this application provides a schematic diagram of a drivable area planning device according to an embodiment. This device can be applied to the aforementioned first device, or it can be used to perform the steps executed by the aforementioned first device, including:

[0181] The acquisition module 1101 is used to acquire crowdsourcing device data and crowdsourcing collection data. The crowdsourcing device data includes data from multiple second devices, and the crowdsourcing collection data includes data collected by sensors in multiple second devices.

[0182] The path encoding module 1102 is used to encode the crowdsourced device data to obtain device path encoding data;

[0183] The environment coding module 1103 is used to encode crowdsourced data to obtain environmental semantic coding data.

[0184] The planning module 1104 is used to input the device path coding data and the environmental semantic coding data into the neural network and output the first drivable area. The first drivable area is the area where the first device can drive. The neural network is used to extract the correlation between the environmental semantic coding data and the device path coding data and determine the first drivable area based on the correlation.

[0185] In one possible implementation, the aforementioned crowdsourced data collection includes crowdsourced point cloud data, which is data obtained by fusing data collected by sensors of multiple second devices, and crowdsourced device data package crowdsourced vehicle flow data, which includes path information of multiple second devices.

[0186] In one possible implementation, the aforementioned path encoding module 1102 is specifically used to perform vector encoding on the crowdsourced vehicle flow data to obtain device path encoding data.

[0187] The environment coding module 1103 is specifically used to perform semantic coding on crowdsourced point cloud data to obtain environment semantic coding data.

[0188] In one possible implementation, the aforementioned apparatus further includes:

[0189] Alignment module 1105 is used to align crowdsourced point cloud data and crowdsourced device data in the same space to obtain aligned crowdsourced point cloud data and aligned crowdsourced device data;

[0190] The slicing module 1106 is used to slice the aligned crowdsourced point cloud data and the aligned crowdsourced device data in the same space to obtain multiple point cloud data slices and multiple device data slices.

[0191] The path encoding module 1102 is specifically used to perform vector encoding on each device data slice in multiple device data slices to obtain device path encoding data corresponding to each device data slice;

[0192] The environment coding module 1103 is specifically used to perform semantic coding on each point cloud data slice in multiple point cloud data slices to obtain the environment semantic coding data corresponding to each point cloud data slice.

[0193] In one possible implementation, the aforementioned planning module is specifically used to: input the environmental semantic encoding data corresponding to each point cloud data slice and the device path encoding data corresponding to each device data slice into a neural network to obtain the region corresponding to each slice; and fuse the regions corresponding to each slice to obtain the first drivable region.

[0194] In one possible implementation, the aforementioned vector encoding includes encoding a vector representing a device path, the vector including at least one of position, velocity, or direction.

[0195] In one possible implementation, the aforementioned semantic encoding includes encoding at least one of point clouds, the location semantic category of point clouds, or the association between point clouds.

[0196] In one possible implementation, the aforementioned device further includes a fusion module 1107, configured to: acquire a second drivable area, the second drivable area being an area determined based on data from the first device; and fuse the first drivable area and the second drivable area to obtain a third drivable area.

[0197] In one possible implementation, the aforementioned device further includes a conflict suppression module 1108, configured to: acquire device path information for a first time period, the first time period including a preset time period closest to the current time; update a first drivable area based on the device path information, and use the updated area as the new first drivable area.

[0198] In one possible implementation, the aforementioned first device and second device are vehicles.

[0199] Figure 12 shows a schematic diagram of the hardware structure of a drivable area planning device 120 provided in an embodiment of this application. This drivable area planning device 120 can be used to implement the steps of the methods shown in Figures 3 to 10 above.

[0200] The drivable area planning device 120 shown in Figure 12 may include a processor 1201, a memory 1202, a communication interface 1203, and a bus 1204. The processor 1201, the memory 1202, and the communication interface 1203 can be connected via the bus 1204.

[0201] The processor 1201 is the control center of the drivable area planning device 120. It can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adapted to the actual application scenario.

[0202] As an example, processor 1201 may include one or more CPUs, and may also include other processors, such as the CPU, NPU or GPU shown in Figure 12.

[0203] The memory 1202 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0204] In one possible implementation, the memory 1202 may exist independently of the processor 1201. The memory 1202 can be connected to the processor 1201 via a bus 1204 and is used to store data, instructions, or program code. When the processor 1201 calls and executes the instructions or program code stored in the memory 1202, it can implement the methods provided in the embodiments of this application, such as the methods shown in Figures 3 to 10 above.

[0205] In another possible implementation, the memory 1202 can also be integrated with the processor 1201.

[0206] The communication interface 1203 is used for the drivable area planning device 120 to connect with other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 1203 may include a receiving unit for receiving data and a transmitting unit for transmitting data.

[0207] Bus 1204 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 12, but this does not indicate that there is only one bus or one type of bus.

[0208] It should be noted that the structure shown in FIG12 does not constitute a limitation on the drivable area planning device 120. In addition to the components shown in FIG12, the drivable area planning device 120 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0209] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0210] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0211] This application also provides a computer-readable storage medium storing a program for training a model or performing inference tasks, which, when run on a computer, causes the computer to perform all or part of the steps in the methods described in the embodiments shown in Figures 3 to 10 above.

[0212] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the aforementioned processor or processor functions, and one or more interfaces. When the digital processing chip integrates a memory, it can perform the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the method steps of any one or more of the foregoing embodiments based on the program code stored in the external memory.

[0213] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0214] The target sensing device or target sensing device provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the server to execute the method described in the embodiments shown in Figures 3-10 above. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, such as random access memory (RAM).

[0215] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0216] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0217] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0218] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0219] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0220] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps in this application does not imply that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved. The division of modules in this application is a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed may be through some ports, and the indirect coupling or communication connection between modules may be electrical or other similar forms, which are not limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules can be selected to achieve the purpose of the solution in this application according to actual needs.

Claims

1. A method for planning drivable areas, characterized in that, Applied to the first device, including: Acquire crowdsourced device data and crowdsourced collection data, wherein the crowdsourced device data includes data from multiple second devices, and the crowdsourced collection data includes data collected by sensors in the multiple second devices; The crowdsourced device data is encoded to obtain device path encoding data; The crowdsourced data is encoded to obtain environmental semantic encoded data; The device path encoding data and the environmental semantic encoding data are input into a neural network, and a first drivable area is output. The first drivable area is the area where the first device can drive. The neural network is used to extract the correlation between the environmental semantic encoding data and the device path encoding data, and to determine the first drivable area based on the correlation.

2. The method according to claim 1, characterized in that, The crowdsourced data collection includes crowdsourced point cloud data, which is data obtained by fusing data collected by the sensors of the multiple second devices. The crowdsourced device data package includes crowdsourced vehicle flow data, which includes path information of the multiple second devices.

3. The method according to claim 2, characterized in that, The process of encoding the crowdsourced device data to obtain device path encoding data includes: The crowdsourced vehicle flow data is vector-encoded to obtain the device path encoding data; The encoding of the crowdsourced data to obtain environmental semantic encoded data includes: The crowdsourced point cloud data is semantically encoded to obtain the environmental semantic encoded data.

4. The method according to claim 3, characterized in that, The method further includes: The crowdsourced point cloud data and the crowdsourced device data are aligned in the same space to obtain aligned crowdsourced point cloud data and aligned crowdsourced device data; The aligned crowdsourced point cloud data and the aligned crowdsourced device data are sliced ​​in the same space to obtain multiple point cloud data slices and multiple device data slices; The process of vector encoding the crowdsourced vehicle flow data to obtain the device path encoding data includes: Each device data slice in the plurality of device data slices is vector encoded to obtain device path encoded data corresponding to each device data slice; The step of semantically encoding the crowdsourced point cloud data to obtain the environmental semantically encoded data includes: Semantic encoding is performed on each of the multiple point cloud data slices to obtain the environmental semantic encoding data corresponding to each point cloud data slice.

5. The method according to claim 4, characterized in that, The step of inputting the device path encoding data and the environmental semantic encoding data into the neural network and outputting the first drivable area includes: The environmental semantic encoding data corresponding to each point cloud data slice and the device path encoding data corresponding to each device data slice are input into the neural network to obtain the region corresponding to each slice. The regions corresponding to each slice are merged to obtain the first drivable region.

6. The method according to any one of claims 3-5, characterized in that, The vector encoding includes encoding a vector representing a device path, the vector including at least one of position, velocity, or direction.

7. The method according to any one of claims 3-6, characterized in that, The semantic encoding includes encoding at least one of point clouds, the semantic category of point cloud location, or the association between point clouds.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Obtain a second drivable area, which is an area determined based on data from the first device; The first drivable area and the second drivable area are merged to obtain the third drivable area.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: Obtain device path information for a first time period, which includes a preset time period closest to the current time. The first drivable area is updated based on the device path information, and the updated area is used as the new first drivable area.

10. The method according to any one of claims 1-9, characterized in that, The first device and the second device are vehicles.

11. A drivable area planning device, characterized in that, Applied to the first device, including: The acquisition module is used to acquire crowdsourcing device data and crowdsourcing collection data. The crowdsourcing device data includes data from multiple second devices, and the crowdsourcing collection data includes data collected by sensors in the multiple second devices. The path encoding module is used to encode the crowdsourced device data to obtain device path encoded data; The environment coding module is used to encode the crowdsourced data to obtain environmental semantic coding data; The planning module is used to input the device path encoding data and the environmental semantic encoding data into a neural network and output a first drivable area, which is the area where the first device can drive. The neural network is used to extract the correlation between the environmental semantic encoding data and the device path encoding data and determine the first drivable area based on the correlation.

12. The apparatus according to claim 11, characterized in that, The crowdsourced data collection includes crowdsourced point cloud data, which is data obtained by fusing data collected by the sensors of the multiple second devices. The crowdsourced device data package includes crowdsourced vehicle flow data, which includes path information of the multiple second devices.

13. The apparatus according to claim 12, characterized in that, The path encoding module is specifically used to perform vector encoding on the crowdsourced vehicle flow data to obtain the device path encoding data; The environment coding module is specifically used to perform semantic coding on the crowdsourced point cloud data to obtain the environment semantic coding data.

14. The apparatus according to claim 13, characterized in that, The device further includes: The alignment module is used to align the crowdsourced point cloud data and the crowdsourced device data in the same space to obtain aligned crowdsourced point cloud data and aligned crowdsourced device data. The slicing module is used to slice the aligned crowdsourced point cloud data and the aligned crowdsourced device data in the same space to obtain multiple point cloud data slices and multiple device data slices. The path encoding module is specifically used to perform vector encoding on each of the plurality of device data slices to obtain device path encoding data corresponding to each device data slice; The environment coding module is specifically used to perform semantic coding on each of the multiple point cloud data slices to obtain the environment semantic coding data corresponding to each point cloud data slice.

15. The apparatus according to claim 14, characterized in that, The planning module is specifically used for: The environmental semantic encoding data corresponding to each point cloud data slice and the device path encoding data corresponding to each device data slice are input into the neural network to obtain the region corresponding to each slice. The regions corresponding to each slice are merged to obtain the first drivable region.

16. The apparatus according to any one of claims 13-15, characterized in that, The vector encoding includes encoding a vector representing a device path, the vector including at least one of position, velocity, or direction.

17. The apparatus according to any one of claims 13-16, characterized in that, The semantic encoding includes encoding at least one of point clouds, the semantic category of point cloud location, or the association between point clouds.

18. The apparatus according to any one of claims 11-17, characterized in that, The device further includes a fusion module for: Obtain a second drivable area, which is an area determined based on data from the first device; The first drivable area and the second drivable area are merged to obtain the third drivable area.

19. The apparatus according to any one of claims 11-18, characterized in that, The device further includes a conflict suppression module for: Obtain device path information for a first time period, which includes a preset time period closest to the current time. The first drivable area is updated based on the device path information, and the updated area is used as the new first drivable area.

20. The apparatus according to any one of claims 11-19, characterized in that, The first device and the second device are vehicles.

21. A drivable area planning device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the device performs the method as described in any one of claims 1-10.

22. A vehicle, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the device performs the method as described in any one of claims 1-10.

23. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 10.

24. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 10.

Images