Obstacle perception method, device, storage medium, and program product

Abstract

An obstacle perception method, relating to the technical field of intelligent assisted driving. The method comprises: acquiring unmanned aerial vehicle perception data, platform-side perception data, and an unmanned aerial vehicle relative pose, wherein the unmanned aerial vehicle relative pose is used for describing pose parameters of an unmanned aerial vehicle relative to a vehicle (10); on the basis of the unmanned aerial vehicle relative pose, transforming the unmanned aerial vehicle perception data into a platform-side coordinate system to obtain corresponding high-altitude perspective perception data; and fusing the high-altitude perspective perception data and the platform-side perception data to construct an obstacle perception map. By introducing the high-altitude perspective perception data provided by the unmanned aerial vehicle, the method effectively extends a perception range and provides more reliable and accurate obstacle information for an intelligent assisted driving system. Further disclosed are an obstacle perception system, a computer device, a computer-readable storage medium, a computer program product, and a movable platform.

Description

Obstacle sensing methods, devices, storage media and software products Technical Field

[0001] This application relates to the field of intelligent assisted driving technology, and in particular to an obstacle perception method, device, storage medium, program product and mobile platform. Background Technology

[0002] With the rapid development of intelligent driver assistance technology, the vehicle's environmental perception capability has become a key research direction. Environmental perception typically relies on a variety of sensors, such as lidar, millimeter-wave radar, and cameras, which can provide distance information, speed information, and visual feature data around the vehicle.

[0003] However, vehicle sensors (such as lidar and millimeter-wave radar) are primarily based on straight-line signal propagation, and their sensing range is significantly affected by line-of-sight obstruction. When vehicles travel on slopes, curves, overpasses, or other complex terrains, "blind spots" easily appear due to limited visibility, leading to missed obstacle detection and greatly impacting the safety of intelligent assisted driving systems.

[0004] Currently, the industry has not proposed a better solution to the above problems. Summary of the Invention

[0005] This application provides an obstacle sensing method, device, storage medium, program product, and mobile platform to at least solve one of the above-mentioned technical problems.

[0006] In a first aspect, embodiments of this application provide an obstacle perception method, comprising: acquiring drone perception data, platform-side perception data, and drone relative pose; the drone relative pose is used to describe the drone's pose parameters relative to a vehicle; converting the drone perception data to a platform-side coordinate system based on the drone relative pose to obtain corresponding high-altitude perspective perception data; and fusing the high-altitude perspective perception data and the platform-side perception data to construct an obstacle perception map.

[0007] Secondly, embodiments of this application provide an obstacle perception system, comprising: an acquisition unit for acquiring UAV perception data, platform-side perception data, and UAV relative pose; the UAV relative pose is used to describe the UAV's pose parameters relative to a vehicle; a conversion unit for converting the UAV perception data to a platform-side coordinate system based on the UAV relative pose to obtain corresponding high-altitude perspective perception data; and a fusion unit for fusing the high-altitude perspective perception data and the platform-side perception data to construct an obstacle perception map.

[0008] Thirdly, embodiments of this application provide a storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the obstacle perception methods described above in this application.

[0009] Fourthly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the obstacle sensing methods and systems described above in this application.

[0010] Fifthly, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to execute any of the above-described obstacle sensing methods and systems.

[0011] Sixthly, embodiments of this application provide a mobile platform, the mobile platform including a first sensor, a memory, and a processor; the first sensor is used to collect first sensing information; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement any of the obstacle sensing methods and systems described above in this application.

[0012] Optionally, the mobile platform further includes a second sensor mounted on the drone, the second sensor being used to collect second sensing information.

[0013] The beneficial effects of the embodiments of this application are at least as follows:

[0014] By incorporating high-altitude perspective perception data provided by drones, more comprehensive environmental information can be obtained in areas where platform-side sensors are limited (such as slopes, bridges, or turns with obstructed visibility), effectively expanding the perception range and eliminating blind spots created by traditional platform-side sensors. Furthermore, by converting the drone's perception data to the platform-side coordinate system based on its relative pose, the fusion of platform-side and drone perception data is ensured within the same coordinate system. This avoids data distortion or fusion errors caused by coordinate system inconsistencies, thereby improving the accuracy of perception data fusion. This provides more reliable and accurate obstacle information for intelligent assisted driving systems, further enhancing driving safety. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 shows a flowchart of an example of an obstacle sensing method according to an embodiment of this application;

[0017] Figure 2A shows a schematic diagram illustrating the effect of an example obstacle perception map according to an embodiment of this application;

[0018] Figure 2B shows a schematic diagram illustrating the effect of an example obstacle perception map according to an embodiment of this application;

[0019] Figure 3 shows an example of an operation flowchart for obtaining the relative pose of a drone according to an embodiment of this application;

[0020] Figure 4A shows a schematic diagram of an example of a user interface for inputting the relative pose of a drone according to an embodiment of this application.

[0021] Figure 4B shows a schematic diagram of an example of a user interface for inputting the relative pose of a drone according to an embodiment of this application.

[0022] Figure 5A shows an example of an operation flowchart for constructing an obstacle perception map for the same obstacle according to an embodiment of this application;

[0023] Figure 5B shows a schematic diagram illustrating the effect of an example obstacle perception map according to an embodiment of this application;

[0024] Figure 6 shows a schematic diagram of an example of obstacle perception map construction for the same obstacle;

[0025] Figure 7 shows an example of the operation flowchart for constructing obstacle perception maps for different obstacles;

[0026] Figure 8 shows a graphical interface schematic diagram of an example of an obstacle graphics component library according to an embodiment of this application;

[0027] Figure 9 shows a schematic diagram illustrating the effect of an example of dynamically displayed obstacle perception map according to an embodiment of this application;

[0028] Figure 10 shows a schematic diagram of an example of displaying the real-time status of a vehicle using a steering wheel display;

[0029] Figure 11 shows a structural block diagram of an example obstacle sensing system according to an embodiment of this application;

[0030] Figure 12 is a schematic diagram of the structure of an embodiment of the electronic device of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0032] It should also be noted that, in this document, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0033] The technical solutions in this application, including the collection, storage, use, processing, transmission, provision, and disclosure of users' personal information, comply with relevant laws and regulations and do not violate public order and good morals.

[0034] Figure 1 shows a flowchart of an example of an obstacle perception method according to an embodiment of this application.

[0035] Regarding the execution entity of the method in this application embodiment, it can be any controller or processor with computing or processing capabilities, executing the obstacle perception method. Addressing the problem that traditional vehicle-mounted sensors (such as LiDAR, millimeter-wave radar, and cameras) rely heavily on straight-line signal propagation and are easily affected by obstructions, resulting in an inability to comprehensively perceive the surrounding environment, the obstacle perception method provided in this application embodiment fuses high-altitude perspective data provided by a drone with the vehicle's own low-altitude perception data. This allows for multi-dimensional perception data fusion at different heights and angles, thereby improving the comprehensiveness of environmental perception. Thus, utilizing the drone's perception data effectively compensates for the blind spots of platform-side sensors in complex terrain, providing more spatial information and enhancing the ability to identify obstacles.

[0036] In some examples, the methods of this application embodiment can be integrated into electronic devices or terminals through software, hardware, or a combination of both, and the types of terminals or electronic devices can be diverse, such as mobile phones, tablets, or desktop computers. In some scenarios, they can also be installed in vehicle terminals to provide or enhance vehicle intelligent assistance functions. In other scenarios, they can be installed on mobile robot terminals to provide or enhance robot autonomous movement functions. In still other scenarios, the methods of this application embodiment can be installed on terminal devices of water vehicles such as ships and yachts to extend the perception range of water vehicles and assist in the driving functions of water vehicles. In this application embodiment, a vehicle (vehicle terminal) is used as an example of a mobile platform (platform terminal) for illustration.

[0037] As shown in Figure 1, in step S110, the UAV perception data, platform-side perception data, and UAV relative pose are acquired. The UAV relative pose is used to describe the pose parameters of the UAV relative to the mobile platform.

[0038] Taking the vehicle-side platform as an example, the details of the platform-side perception data involve collecting first-level depth sensing information through various sensors mounted on the vehicle. This depth information is then analyzed to obtain the corresponding platform-side perception data, which may include, but is not limited to, the distance, speed, and direction of obstacles, as well as the physical characteristics of obstacles (such as shape and size). The first sensors can be various sensors mounted on the vehicle, such as LiDAR, RADAR, and cameras. For example, LiDAR collects distance information of surrounding objects, RADAR detects the relative speed of objects, and cameras capture visual features of the surrounding environment, thus integrating these elements to form the platform-side perception data.

[0039] Regarding the details of the drone's perception data, second depth sensing information is collected through a second sensor mounted on the drone. This depth information is then analyzed to obtain the corresponding drone perception data, which may include, but is not limited to, information such as the spatial location, shape, and distance of ground obstacles. The second sensor can be various sensors mounted on the drone, such as high-definition cameras, LiDAR, or other environmental detection sensors. During flight, it collects data on the high-altitude environment surrounding the platform and views the ground and surrounding objects from different angles, supplementing the blind spots of the platform's sensors with this high-altitude perspective. It should be understood that the number of drones accompanying the platform can be one or more, and there is no limitation on this.

[0040] The relative pose of a drone can include its spatial position and orientation relative to the platform. This can be obtained through various sensing technologies, such as sensor information from inertial navigation and positioning systems. Alternatively, it can be determined based on settings (e.g., drone pose adjustment information, camera gimbal adjustment information), such as by determining the drone's follow-up information set by the user. Both of these methods fall within the scope of this application's embodiments.

[0041] In some examples of embodiments of this application, the relative pose of the drone includes at least one of the following: drone following angle, drone altitude above the ground, and drone following distance.

[0042] The drone following angle refers to the angle of the drone relative to the platform's direction of travel. It is typically expressed as the drone's heading deviation relative to the platform, describing the drone's rotation direction on the horizontal plane, i.e., the drone's orientation relative to the platform. It can be measured by an inertial measurement unit (IMU) or set by the user. The drone's altitude above the ground refers to the vertical distance between the drone and the ground, which can be collected by an altitude sensor or set by the user. The drone following distance refers to the horizontal straight-line distance between the drone and the platform, which can be calibrated by user settings or measured by sensors (such as LiDAR or visual tracking algorithms). Taking a vehicle as the platform, the combined effect of multiple key parameters of the drone's relative posture (following angle, altitude above the ground, and following distance) ensures precise coordination between the drone and the vehicle, enabling accurate posture adjustment and feedback in different environments.

[0043] In step S120, the UAV perception data is converted to the platform coordinate system according to the UAV's relative pose to obtain the corresponding high-altitude perspective perception data.

[0044] Specifically, based on the relative pose parameters between the UAV and the platform, coordinate transformation algorithms (such as rotation matrix, quaternion transformation, etc.) are used to transform the UAV's perception data from the UAV coordinate system to the platform coordinate system, ensuring the data synchronization between the UAV and platform sensors. The transformed data is then used to generate obstacle perception information from a high-altitude perspective, including the position and shape of obstacles captured from the UAV's viewpoint, as well as any overlap or differences with the platform's perception data.

[0045] In some implementations, coordinate system transformation is performed using quaternion transformation. Compared to rotation matrices, quaternions require less computation, providing a computationally efficient and numerically stable way to represent rotation, and avoiding numerical errors and gimbal lock issues found in rotation matrices.

[0046] For example, the quaternion q = (q0, q1, q2, q3) represents a rotation, where q0 is the real part of the quaternion, representing the scalar part of the rotation; and q1, q2, q3 are the imaginary parts of the quaternion, representing the vector part of the rotation.

[0047] Given the sensing data d of the UAV in its local coordinate system u And the quaternion pose q = (q0, q1, q2, q3) of the UAV relative to the platform, transforming the quaternion into a rotation matrix.

[0048] Specifically, the sensing data d in the platform coordinate system c The following formula can be used to calculate: d c =R·d u +T, Equation (1)

[0049] In the formula, R is the rotation matrix obtained by transforming the quaternion q, and d u It is the perception data in the UAV coordinate system, d c This refers to the perceived data in the platform's coordinate system, where T is the translation vector of the UAV relative to the platform, and T = (T x ,T y ,T z ) indicates the translational offset between the drone and the platform.

[0050] It should be noted that the aforementioned high-altitude perspective perception data can be updated based on sensor data and / or user-defined follow information. For example, the real-time quaternion pose of the UAV is dynamically calculated using the UAV's inertial measurement unit (IMU), GPS, and other sensor data (such as LiDAR, vision systems, etc.). Furthermore, if a user-defined UAV follow mode is used, the quaternion of the follow information can be adjusted based on user-defined pose information (e.g., angle and distance), and the real-time relative pose of the UAV can be automatically calculated.

[0051] In step S130, high-altitude perspective perception data and platform-side perception data are fused to construct an obstacle perception map.

[0052] Here, various non-restricted fusion algorithms, such as weighted fusion, Kalman filtering, or other fusion algorithms, can be used to comprehensively process the high-altitude view data provided by the UAV and the low-altitude data from the platform's sensors. Taking a vehicle as an example, when the vehicle cannot clearly perceive obstacles, the UAV's high-altitude view data can serve as supplementary information, enhancing the perception of distant or obstructed objects. Furthermore, based on the fused data, the obstacles are accurately located and their states estimated, thereby generating an obstacle perception map. This map includes the spatial distribution of obstacles in the vehicle's surrounding environment, reducing blind spots in traditional perception systems in complex terrain (e.g., slopes, turns). Preferably, the obstacle perception map also labels the type, size, and outline of each obstacle to support subsequent path planning and decision-making by the vehicle's intelligent driving system.

[0053] Through the embodiments of this application, high-precision obstacle perception maps are constructed by integrating perception data from high altitude and platform, eliminating blind spots of traditional sensors in complex terrain or driving environments, reflecting the dynamic changes of the surrounding environment of the platform in real time, enhancing the responsiveness and adaptability of the intelligent driving system, and ensuring the safety and stability of the platform under various driving conditions.

[0054] Figure 2A shows a schematic diagram illustrating the effect of an example obstacle perception map according to an embodiment of this application.

[0055] It should be noted that the form of the display used to show the obstacle perception map can be diverse, such as a central control screen in the vehicle or a HUD (Head-up Display). As shown in Figure 2A or 2B, the obstacle perception map is displayed through a display mounted in the center of the steering wheel body area. It should be understood that the display placement on the steering wheel in Figure 2A or 2B is only for example, and it can also be designed in other locations such as the wheel spoke area.

[0056] In some examples of embodiments of this application, the drone perception data is determined based on depth sensing information of the drone for at least one of the following: the vehicle driving lane, the adjacent lane of the vehicle driving lane, and the vehicle's surrounding environment information.

[0057] Referring to the example in Figure 2A, the UAV perception data includes depth sensing information from the UAV regarding adjacent lanes of the vehicle's driving lane. An obstacle perception map displays the outlines and locations of different obstacles (21, 22, and 23) in the environment surrounding vehicle 10. Obstacle 21 is an object with a narrow outline located in the left lane of the vehicle; obstacle 22 is an object with a triangular outline located in the right lane of the vehicle; and obstacle 23 is an object with a curved, elongated outline that spans both the current lane and another lane. Thus, by utilizing the UAV's depth perception of adjacent lanes, comprehensive obstacle information in both the current and adjacent lanes is achieved. By labeling the shapes and locations of obstacles, complete lane environment perception information is provided to the driver, improving driving safety.

[0058] Figure 2B shows a schematic diagram illustrating the effect of an example obstacle perception map according to an embodiment of this application.

[0059] Referring to the example in Figure 2B, the UAV perception data includes the depth sensing information of the UAV regarding the vehicle's surrounding environment, which shows the information of the mountains around the lane. Based on the UAV's high-altitude perspective perception data, it is possible to support the presentation of scene information such as sidewalks and mountains around the vehicle in the obstacle perception map. It also provides conditions for the intelligent assisted driving system to detect extreme events, such as when a pedestrian suddenly runs into the road, or when there is a landslide or falling rocks. Early warnings can be triggered in such cases, so that the intelligent assisted driving system can respond in time (such as timely deceleration / braking / turning to avoid the obstacle).

[0060] As shown in Figure 2B, the windshield 25 can display information such as the road conditions ahead, key obstacle icons, vehicle speed, and speed limit reminders in a semi-transparent overlay via a HUD, so as to avoid the driver frequently looking down to look at other display devices. The central control display terminal 27 can be used to display vehicle information, route planning, and media information lights. The obstacle perception map is displayed through the steering wheel display screen 29, focusing on the local information that is most important in the current driving situation.

[0061] Specifically, the display range of the obstacle perception map can also encompass scenes including drones and vehicles, which can be obtained through methods such as 3D modeling. Through this obstacle perception map, the relative positions of drones and vehicles can be viewed, and the display range can be zoomed in or out. In some examples, when a user needs a clearer view of obstacle information on the road, the display range can be switched to Figure 2A based on user interaction, thus meeting different user needs. Additionally, in some examples, the display range of the obstacle perception map can be automatically switched based on driving mode or drone control mode. For example, autonomous or manual driving modes prioritize the display range of Figure 2A, while switching to drone control mode displays the display range of Figure 2B.

[0062] Figure 3 shows an example operation flowchart for acquiring the relative pose of a drone according to an embodiment of this application. Here, the drone's following mode and camera gimbal are adjusted through user interaction with the vehicle-mounted display to achieve the recording of the drone's relative pose.

[0063] In step S310, drone controls and vehicle controls for indicating the drone and vehicle, respectively, are displayed on the first display.

[0064] Specifically, drone controls can display the drone as an icon or virtual model, including information such as the drone's direction, position, and altitude, to dynamically display the drone's real-time status. Vehicle controls can represent the vehicle's position and current direction of travel as a simplified vehicle model or icon, and identify the lane and vehicle body area.

[0065] In step S320, the relative pose of the drone is determined based on the relative pose of the drone control relative to the vehicle control.

[0066] In some implementations, the drone controls support various user interaction operations, such as dragging and rotating, to adjust the drone's pose relative to the vehicle.

[0067] For example, the user can adjust the relative position between the drone and the vehicle control by moving the drone control on the display, and input the user-set pose parameters into the drone flight control system through the vehicle system, thereby adjusting the drone's pose in real time.

[0068] In terms of business application scenarios, users can adjust the pitch and horizontal angles of the drone camera using the virtual slider or directional keys on the display, and generate corresponding gimbal adjustment commands to control and update the drone's spatial positioning.

[0069] The embodiments of this application realize the flexible configuration of the drone follow mode, which supports users to quickly adjust the drone's posture according to different driving scenarios (such as straight roads, slopes, turns, etc.), meet diverse drone follow perception needs, and provide support for obstacle perception in different environments.

[0070] For example, when driving on a straight road, the vehicle's path is relatively stable, and the drone should maintain a basically stable relative position to the vehicle to ensure smooth following. Position adjustments may focus on fine-tuning the horizontal position (such as maintaining a certain front-to-back distance and lateral position) and minor adjustments to the camera gimbal (to ensure visual stability in front of, behind, or to the side of the vehicle).

[0071] When driving on a slope, the vehicle's tilt angle changes, which may affect the drone's relative altitude and stability. To ensure that the drone can automatically adjust its relative altitude according to the slope, attitude adjustment may involve automatic vertical altitude compensation and gimbal pitch angle adjustment to adapt to the slope's undulations.

[0072] When turning, the vehicle's direction of travel changes drastically, which may cause a significant change in the relative position between the drone and the vehicle. At this time, the drone's rotation angle and position adjustment become particularly important. The attitude adjustment must ensure that the drone can flexibly adapt to the relative changes during the turning process and avoid following errors.

[0073] Figure 4A shows a schematic diagram of an example of a user interface for inputting the relative pose of a drone according to an embodiment of this application.

[0074] As shown in Figure 4A, the map display can simultaneously show a wide-view view of both car and drone icons. Users can interact with the drone icon, such as dragging and moving it, for example, moving the drone icon from the left rear of the vehicle to the right front. The map can also display the distance, altitude, and speed of the drone relative to the vehicle, and supports adjustments or updates based on user input.

[0075] Figure 4B shows a schematic diagram illustrating another example of a user interface for inputting the relative pose of a drone according to an embodiment of this application.

[0076] As shown in Figure 4B, the vehicle control 41 is displayed in the center of the user interface, and the drone control 42 can be dragged to move to different positions in the user interface.

[0077] In some implementations, the drone's relative pose is updated when a user interaction with the drone controls is detected to meet preset operation conditions. Here, preset operation conditions are used to detect the user's intention to adjust the drone's relative pose, thus avoiding accidental triggering. The rules for these preset operation conditions can be diverse, such as the user confirming the drone's position after movement or moving to a specific location or area, and can be set or adjusted according to business needs; no restrictions are placed here.

[0078] Referring to the example in Figure 4B, the display shows multiple marker points (a1, a2, etc.), each marker point indicating the relative pose of the corresponding UAV. It should be understood that although the marker points in the figure are arranged in a circular pattern, related variations, such as square or hexagonal arrangements, should also be considered within the scope of implementation of the embodiments of this application.

[0079] Referring to the example in Figure 4B, each marker point represents a set of preset drone following angles and following distances. When a user interaction is detected that moves the drone control to the target marker point, the drone's relative pose indicated by the target marker point is updated. For example, the drone following angles corresponding to a1 and a2 are the same, both 0 degrees, but the drone following distances are different. When the drone control is moved from a1 to a2, the drone following distance is adjusted and updated.

[0080] Preferably, after detecting that the user has moved the drone control to a specific marker point, the ground altitude adjustment slider b is displayed. At this time, by detecting the adjustment operation on line segment b, the flight altitude of the drone can be further adjusted, thereby realizing multi-dimensional comprehensive adjustment of the drone's relative attitude.

[0081] Through the above operations, the function of manually adjusting the relative pose of the drone based on user interaction was realized, which met the user's need for active adjustment in different obstacle environments.

[0082] As an optional or supplementary implementation of this application, the drone may also possess autonomous pose adjustment capabilities. Specifically, when it is detected that the obstacle tracking view indicated by the drone's perception data does not meet preset obstacle tracking conditions, the drone's relative pose is updated. Furthermore, when it is detected that the obstacle tracking view indicated by the drone's perception data does not meet preset obstacle tracking conditions, the pose of the sensors installed on the drone may also be updated, for example, by adjusting the camera gimbal parameters to optimize the camera's shooting range, thereby achieving the goal of meeting the obstacle tracking conditions.

[0083] In some implementations, the status of the drone and gimbal camera can also be manually intervened through human-computer interaction. Specifically, the drone status or gimbal camera status can be controlled by setting human-computer interaction conditions. For example, when the operation received by the drone icon (or other physical button) meets the preset operation conditions (such as clicking or long-pressing to display the settings interface and inputting relevant parameters), the drone status (e.g., takeoff, landing, rotation, and translation) or the gimbal camera status (e.g., pitch angle adjustment, scene zoom, and resolution setting) is updated.

[0084] For example, users can control the zoom level of a 3D scene using gestures (such as pinching or spreading two fingers on a touchscreen), rotating the scroll wheel on the steering wheel, rotating the central control knob, or voice commands (such as "zoom in"). When a user zooms in, the scene focus is concentrated on a small area around the vehicle, allowing for a clearer view of the shape and position of nearby obstacles. When a user zooms out, a wider range of environmental information is displayed, including distant roads, buildings, roadblocks, and other vehicles, enabling global situational awareness.

[0085] Here, the preset obstacle tracking conditions can be varied and can be set according to the field of view coverage or the field of view center offset.

[0086] For the field of view coverage, obstacles need to be located within the effective field of view of the drone camera, and their coverage in the field of view should meet the threshold requirements (e.g., occupying more than 20% of the field of view).

[0087] For example, the field of view of the drone camera is in the horizontal field of view (FOV). h and vertical FOV v Indicates, for example, FOV h =90°, FOV v =60°, the field of view is defined as the area within the horizontal and vertical angles along the camera's optical axis. Calculate the pixel percentage C of obstacles in the camera image. g If C g If the coverage is ≥20%, the coverage area meets the requirements; otherwise, the viewing angle needs to be adjusted.

[0088] For the field of view center offset, the center of the obstacle should be close to the center area of ​​the camera's field of view, and the offset angle should not exceed the set value (e.g., 30 degrees).

[0089] For example, calculate the center point P of the obstacle. obj The angle θ between the camera and the optical axis of the gimbal camera:

[0090] In the formula, D drone Let D be the optical axis direction vector of the gimbal camera. obj Let be the direction vector of the obstacle.

[0091] If θ≤30°, the obstacle is in the center of the camera, which meets the obstacle tracking condition; otherwise, the drone's following angle needs to be adjusted.

[0092] Therefore, through comprehensive analysis of coverage and offset angle, it is possible to accurately assess whether the obstacle tracking viewpoint meets the requirements, and when the requirements are not met, to implement autonomous pose adjustment or autonomously adjust the pose of the sensors set on the UAV, thereby improving the efficiency and stability of obstacle perception and tracking.

[0093] In addition, specific tracking designs can be implemented for specific types of obstacles. For example, for moving obstacles (such as pedestrians or other vehicles), drones need to ensure the response speed of the field of view and maintain continuous tracking of the target.

[0094] Figure 5A shows an example of an operation flowchart for constructing an obstacle perception map for the same obstacle according to an embodiment of this application.

[0095] As shown in Figure 5A, in step S510, the first obstacle information and the second obstacle information corresponding to the high-altitude perspective perception data and the platform-side perception data are analyzed respectively.

[0096] Here, feature extraction is performed on the high-altitude perspective perception data and the platform-side perception data to generate corresponding first obstacle information and second obstacle information. The obstacle information may include the spatial location of the obstacle (e.g., the coordinates of the obstacle's center point), shape features (e.g., the length, width, and height of the obstacle), and dynamic features (e.g., the speed and direction of movement of the obstacle).

[0097] In step S520, if the first obstacle information and the second obstacle information are detected to correspond to the same obstacle, the obstacle outline is reconstructed based on the first obstacle information and the second obstacle information, and the corresponding obstacle location is calculated.

[0098] In some implementations, the spatial positional differences between the first obstacle information and the second obstacle information are compared. For example, Euclidean distance is calculated, and if the Euclidean distance is less than a preset threshold, it is confirmed that the two sets of data correspond to the same obstacle.

[0099] At this point, the contours of the same obstacle are fused and reconstructed. Specifically, the top-view data collected by the UAV mainly provides the top contour and lateral distribution information of the obstacle. The point cloud data in the top view is projected onto a plane to extract the planar geometric information of the obstacle (such as length and width). The projected area is then clustered to separate the individual obstacles. Side-view data provided by vehicle-mounted sensors supplements the vertical height and side structure information of the obstacle. For example, LiDAR or millimeter-wave radar is used to measure the height and side shape of the obstacle, and stereo vision algorithms are used to extract the depth and shape of the obstacle from the vehicle-mounted camera images.

[0100] For example, point cloud stitching technology is used to fuse the top-view contour extracted from aerial perspective data with the side-view contour extracted from platform perspective data to generate a complete 3D obstacle contour. In some cases, if data conflicts exist (such as inconsistent shapes in the overlapping areas of aerial and platform perception data), the contour can be reconstructed using point cloud weighted fusion for the conflicting overlapping areas.

[0101] Regarding the computational details of obstacle localization, the final position can be calculated by weighted averaging the obstacle's position from both the aerial and platform-side perspectives. Furthermore, if the obstacle is moving, its speed and direction can be calculated, and its trajectory estimated based on perception data from multiple time points.

[0102] In step S530, an obstacle perception map is constructed based on the obstacle outline and obstacle location.

[0103] In some implementations, the reconstructed obstacle information is inserted into a 3D grid map, which includes detailed features of the obstacles such as location, shape, and dynamic state (stationary or moving), and can be updated in real time based on the dynamic perception data of vehicles and drones.

[0104] Figure 5B shows a schematic diagram illustrating the effect of an example obstacle perception map according to an embodiment of this application.

[0105] As shown in Figure 5B, the obstacle perception map displays the vehicle's surrounding environment, including the vehicle's position, obstacle positions, and distance markers. Specifically, the vehicle's position is marked at the center point of the display interface, and obstacle outlines are highlighted in red or other prominent colors, providing the driver with a visual reference for quickly identifying their spatial position relative to the vehicle. The shape of the obstacle corresponds to its actual modeled features, and icons representing corresponding traffic objects, such as vehicles, pedestrians, and roadblocks, can also be used. Furthermore, obstacle avoidance outlines can be displayed in areas where obstacles approach the vehicle to alert the driver to avoid these outlines. These obstacle avoidance outlines can be set at a preset distance along the outer edge of the obstacle's outline; this preset distance can be set based on the minimum range for avoiding collisions with obstacles as configured by the system.

[0106] In addition, it can display distance markers for the nearest obstacle, such as 30cm, 50cm, etc. These distance markers allow drivers to quickly understand the severity of potential collision risks.

[0107] In some implementations, the obstacle-aware map is also used to receive user input to zoom in and out on the 3D scene or objects around the vehicle. Furthermore, when a user zooms in on an obstacle, the system can automatically display or highlight key information about that obstacle in the zoomed-out interface, such as relative distance and clear outline details of the obstacle.

[0108] Figure 6 shows a schematic diagram of an example architecture for building obstacle-aware maps.

[0109] As shown in Figure 6, the UAV 610 includes a sensor data acquisition module 611 and a transmitter 612, which are mainly responsible for data acquisition and transmission. The vehicle 620 includes a vehicle-mounted sensor 621, a receiver 622, an HMI (Human Machine Interface) module 623, and a planning and control module 624, which are mainly responsible for data reception, processing, fusion, and application.

[0110] Specifically, the sensor data acquisition module 611 integrates sensor data from the UAV and other relevant data, such as camera position, RTK (Real Time Kinematic) data, timestamps, etc., and can also perform operations such as data compression and packaging.

[0111] After the sensor data acquisition module 611 completes the acquisition of the same batch of data, the transmitter 612 transmits data to the receiver 622 via wireless technology, transmitting the data from the UAV to the vehicle 620.

[0112] Deep analysis is performed on the sensor data collected by the UAV, and the depth information (high-altitude perspective depth map) from the perspective of the UAV sensor is analyzed. Through semantic recognition (high-altitude perspective semantic recognition), dynamic objects, static objects and other predefined sensing objects are identified.

[0113] The vehicle-mounted sensor 621 collects vehicle-mounted sensing information, analyzes the depth information (vehicle-mounted perspective depth map) from the perspective of the vehicle-mounted sensor, and identifies dynamic objects, static objects, and other predefined sensing objects through semantic recognition (vehicle-mounted perspective semantic recognition).

[0114] By utilizing the current location, RTK, and pose information of the UAV and the pose information of the vehicle, the coordinates of the perception results are transformed and uniformly converted to the coordinate system of the platform. The perception results of the UAV and the vehicle are then fused to form a complete perception fusion map.

[0115] After the fusion is completed, the perceived fused map is transmitted to the downstream HMI module 623 or planning and control module 624.

[0116] Through the embodiments of this application, a top-down global information of obstacles is provided using a high-altitude perspective, while the vehicle-side perspective supplements the side details. The fusion of the two generates a more accurate and complete obstacle perception map, providing high-quality data support for path planning, obstacle avoidance, and decision control in autonomous driving.

[0117] In the process of constructing an obstacle perception map by integrating high-altitude perspective perception data and platform-side perception data, the obtained high-altitude perspective perception data and platform-side perception data may contain first obstacle information and second obstacle information for the same obstacle. If the first obstacle information and the second obstacle information are detected to correspond to the same obstacle, the obstacle outline can be reconstructed based on the first obstacle information and the second obstacle information, and the corresponding obstacle location can be calculated. Based on the obstacle outline and the obstacle location, an obstacle perception map is constructed.

[0118] Figure 7 shows an example of the operation flowchart for constructing obstacle perception maps for different obstacles.

[0119] As shown in Figure 7, in step S710, the first obstacle type and the first obstacle position corresponding to the first obstacle information are parsed, and the second obstacle type and the second obstacle position corresponding to the second obstacle information are parsed.

[0120] Here, obstacle information includes the obstacle category and its coordinate position in the platform's coordinate system. Specifically, obstacle types are analyzed through semantic recognition to identify the types of obstacles, such as dynamic objects (pedestrians, vehicles, animals, etc.) and static objects (such as traffic signs, guardrails, etc.).

[0121] In step S720, a first obstacle graphic component and a second obstacle graphic component corresponding to the first obstacle type and the second obstacle type are determined, respectively.

[0122] In some implementations, a component library containing various obstacle graphics is designed, with each component representing a different type of obstacle using a specific shape, color, and size. In the example of the obstacle graphic component library shown in Figure 8, pedestrians are represented by human-shaped icons, while bicycles, trucks, and buses are represented by their respective icons. Furthermore, the corresponding graphic component is matched based on the parsed obstacle type.

[0123] In step S730, a first obstacle graphic component is marked at the first obstacle location, and a second obstacle graphic component is marked at the second obstacle location, thereby constructing an obstacle perception map.

[0124] Specifically, based on the obstacle's location, the corresponding graphic components are accurately labeled on the perception map. Combining the graphic components and location labels of all obstacles, a complete perception map is generated. The perception map supports real-time updates, ensuring that information about dynamic obstacles is continuously updated.

[0125] In some examples of embodiments of this application, an obstacle perception map is displayed stereoscopically using a second display. The second display can take various forms, such as a vehicle center console screen, a head-up display (HUD), or a steering wheel display, to stereoscopically display the perception map and indicate the outlines and locations of obstacles. This provides an intuitive, detailed, and real-time updated environmental perception map, offering decision support for the driver or autonomous driving system.

[0126] Regarding the details of presenting the obstacle perception map in 3D, on the one hand, the obstacle positions are projected and converted to the screen coordinate system of the second display. This ensures that when the vehicle or obstacle position changes, the corresponding display position on the display is updated in real time, guaranteeing that the obstacle's appearance on the display always matches its actual position. On the other hand, the display can be divided into different display areas, which are used to indicate the relative positions of the corresponding obstacles.

[0127] For example, in the second display, a target display area is determined that matches the location of the third obstacle, which is the location of the third obstacle in the obstacle perception map. For example, the second display may be divided into multiple display areas, each uniquely corresponding to the relative position of an obstacle; for instance, an obstacle located to the left front of the vehicle is displayed in the first display area, while an obstacle located to the right rear of the vehicle is displayed in the second display area. Then, based on the target display area, the third obstacle is displayed.

[0128] Therefore, by selecting the corresponding target display area based on the location of the obstacle for visualization, the driver can quickly and clearly grasp the key information about the obstacle.

[0129] Figure 9 shows a schematic diagram illustrating an example of the effect of dynamically displaying an obstacle perception map according to an embodiment of this application.

[0130] As shown in Figure 9, the second display is located on the rim area of ​​the steering wheel and can show three states: the obstacle is on the left, middle, or right side of the display. This allows for real-time detection of obstacles on the road and alerts the driver to their location based on the detected obstacle's position.

[0131] For example, when an obstacle is in the left area of ​​the display, it indicates that the obstacle is in the left lane. During an alert, the hazard icon may flash twice until the danger is over. When the obstacle is in the middle area of ​​the display, it indicates that the obstacle is in the same lane. When the obstacle is in the right area of ​​the display, it indicates that the obstacle is in the right lane. Preferably, the type of obstacle can also be identified, and the corresponding component can be invoked to display it in the appropriate area. Thus, through intuitive hazard display, the time required for users to receive real-time information feedback in automatic mode and make real-time decisions in the event of a perceived crisis can be shortened.

[0132] As a preferred embodiment, the second display integrated into the steering wheel can also display various vehicle statuses in real time.

[0133] Figure 10 shows a schematic diagram illustrating an example of displaying the real-time status of a vehicle using a steering wheel display.

[0134] As shown in Figure 10, the four lane-changing states are displayed using different gradient light signals. Specifically, the upper two sub-figures of Figure 10 represent successful left and right lane changes, respectively, while the lower two sub-figures represent failed left and right lane changes, respectively. Furthermore, the steering wheel display can also present other non-restricted vehicle information, such as vehicle speed or driving mode.

[0135] Therefore, by linking real-time status feedback display and operational decisions on the steering wheel-integrated display, the user's operating costs and safety in autonomous driving mode are reduced, thus providing a more humanized and functionally safe design definition for vehicle-machine interaction.

[0136] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0137] Figure 11 shows a structural block diagram of an example obstacle sensing system according to an embodiment of this application.

[0138] As shown in Figure 11, the obstacle perception system 1100 includes an acquisition unit 1110, a conversion unit 1120, and a fusion unit 1130.

[0139] The acquisition unit 1110 is used to acquire UAV perception data, platform-side perception data, and UAV relative pose; the UAV relative pose is used to describe the UAV's pose parameters relative to the vehicle.

[0140] The conversion unit 1120 is used to convert the UAV perception data to the platform coordinate system according to the relative pose of the UAV, so as to obtain the corresponding high-altitude perspective perception data.

[0141] The fusion unit 1130 is used to fuse the high-altitude perspective perception data and the platform-side perception data to construct an obstacle perception map.

[0142] In some implementations, the UAV perception data is determined based on depth sensing information from the UAV for at least one of the following: the vehicle's driving lane, adjacent lanes of the vehicle's driving lane, and information about the vehicle's surrounding environment.

[0143] In some implementations, acquiring the relative pose of the UAV includes:

[0144] Based on the first display, drone controls and platform controls are displayed for indicating the drone and the mobile platform, respectively;

[0145] The relative pose of the drone is determined based on the relative pose of the drone control relative to the platform control.

[0146] In some implementations, the system further includes:

[0147] The first pose update unit (not shown) is used to update the relative pose of the drone when a user interaction operation on the drone control is detected to meet a preset operation condition.

[0148] In some embodiments, the display shows multiple marker points, each marker point indicating the relative pose of the corresponding UAV;

[0149] The step of updating the relative pose of the drone upon detecting that a user interaction operation on the drone control meets preset operation conditions includes:

[0150] If a user interaction is detected to move the drone control to a target marker, the drone's relative pose is updated based on the target marker.

[0151] In some implementations, the relative pose of the drone includes at least one of the following: drone follow angle, drone altitude above the ground, and drone follow distance.

[0152] In some implementations, the system further includes:

[0153] The second pose update unit (not shown) is used to update the relative pose of the UAV or the pose of the sensors installed on the UAV when the obstacle tracking view indicated by the UAV perception data does not meet the preset obstacle tracking conditions.

[0154] In some implementations, the fusion of the high-altitude perspective perception data and the platform-side perception data to construct an obstacle perception map includes:

[0155] The first obstacle information and the second obstacle information corresponding to the high-altitude perspective perception data and the platform-side perception data are analyzed respectively;

[0156] If the first obstacle information and the second obstacle information are detected to correspond to the same obstacle, the obstacle outline is reconstructed based on the first obstacle information and the second obstacle information, and the corresponding obstacle location is calculated;

[0157] An obstacle perception map is constructed based on the obstacle outline and the obstacle location.

[0158] In some implementations, when the first obstacle information and the second obstacle information are detected to correspond to different obstacles, the method further includes:

[0159] The first obstacle type and the first obstacle location corresponding to the first obstacle information are analyzed, and the second obstacle type and the second obstacle location corresponding to the second obstacle information are analyzed.

[0160] Determine a first obstacle graphic component and a second obstacle graphic component that correspond to the first obstacle type and the second obstacle type, respectively;

[0161] The first obstacle graphic component is labeled at the location of the first obstacle, and the second obstacle graphic component is labeled at the location of the second obstacle, thereby constructing an obstacle perception map.

[0162] In some implementations, after fusing the high-altitude perspective perception data and the platform-side perception data to construct an obstacle perception map, the method further includes:

[0163] The obstacle perception map is displayed in stereo based on the second display.

[0164] In some embodiments, the step of displaying the obstacle perception map in stereoscopic form on a second display includes:

[0165] In the second display, a target display area matching the position of the third obstacle is determined; the position of the third obstacle is the position of the third obstacle in the obstacle perception map;

[0166] The third obstacle is displayed based on the target display area.

[0167] In some embodiments, the first display and / or the second display are mounted on the steering wheel of the vehicle.

[0168] In some embodiments, this application provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the obstacle perception methods described above.

[0169] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the obstacle perception methods described above.

[0170] In some embodiments, this application also provides an electronic device including: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform an obstacle perception method.

[0171] In some embodiments, this application also provides a mobile platform, wherein the mobile platform includes a first sensor, a memory, and a processor; the first sensor is used to collect first perception information; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, to implement the steps of an obstacle perception method. The mobile platform may be a vehicle, an intelligent robot, a ship, etc.; the first sensor, memory, and processor are mounted on the mobile platform. The first sensor may include components such as a camera or radar capable of acquiring platform-side perception data (first perception information).

[0172] In some embodiments, the mobile platform further includes a second sensor mounted on the drone, which is used to collect second perception information. The second sensor may include components such as cameras or radar that can acquire high-altitude perspective perception data (second perception information).

[0173] In some embodiments, the mobile platform further includes a drone on which the second sensor is mounted.

[0174] In some embodiments, the mobile platform further includes a hangar installed on the mobile platform for housing the drone.

[0175] Here, the hangar provides a safe environment for parking, charging, and maintenance of drones. It should be noted that the hangar structure can be diverse and customized according to vehicle type and drone specifications, and is equipped with an automated opening and closing mechanism, enabling drones to be quickly deployed and retrieved while in motion.

[0176] The obstacle sensing device described in the above embodiments of this application can be used to execute the obstacle sensing method described in the embodiments of this application, and accordingly achieve the technical effects achieved by the obstacle sensing method described in the above embodiments of this application, which will not be elaborated further here. In the embodiments of this application, the relevant functional modules can be implemented by a hardware processor.

[0177] Figure 12 is a schematic diagram of the hardware structure of an electronic device for performing an obstacle perception method according to another embodiment of this application. As shown in Figure 12, the device includes:

[0178] One or more processors 1210 and memory 1220, with one processor 1210 as an example in Figure 12.

[0179] The device for performing the obstacle sensing method may further include an input device 1230 and an output device 1240.

[0180] The processor 1210, memory 1220, input device 1230 and output device 1240 can be connected by a bus or other means. Figure 12 shows an example of connection by a bus.

[0181] The memory 1220, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the obstacle perception method in the embodiments of this application. The processor 1210 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 1220, thereby implementing the obstacle perception method in the above-described method embodiments.

[0182] The memory 1220 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the obstacle sensing device. Furthermore, the memory 1220 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1220 may optionally include memory remotely located relative to the processor 1210, and this remote memory may be connected to the obstacle sensing device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0183] The input device 1230 can receive input digital or character information and generate signals related to user settings and function control of the obstacle sensing device. The output device 1240 may include a display device such as a display screen.

[0184] The one or more modules are stored in the memory 1220, and when executed by the one or more processors 1210, they execute the obstacle perception method in any of the above method embodiments.

[0185] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0186] The electronic devices in this application embodiments exist in various forms, including but not limited to:

[0187] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0188] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0189] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

[0190] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0191] (5) Other electronic devices with data interaction functions.

[0192] In some embodiments, this application also provides a mobile platform on which the computer device described in any embodiment of this application is installed. The mobile platform includes, but is not limited to, vehicles, tracked robots, bipedal robots, quadrupedal robots, etc., wherein the vehicle can be a passenger car, pickup truck, truck, etc. It should be noted that the above are merely examples, and this application does not limit the specific form of the mobile platform.

[0193] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An obstacle sensing method, comprising: Acquire drone perception data, platform-side perception data, and drone relative pose; The relative pose of the UAV is used to describe the pose parameters of the UAV relative to the mobile platform. Based on the relative pose of the UAV, the UAV perception data is converted to the platform coordinate system to obtain the corresponding high-altitude perspective perception data. The high-altitude perspective perception data and the platform-side perception data are integrated to construct an obstacle perception map.

2. The method according to claim 1, wherein, The UAV perception data is determined based on depth sensing information of the UAV for at least one of the following: the vehicle's driving lane, the adjacent lanes of the vehicle's driving lane, and information about the vehicle's surrounding environment.

3. The method according to claim 1, wherein, The acquisition of the relative pose of the UAV includes: Based on the first display, drone controls and platform controls are displayed for indicating the drone and the mobile platform, respectively; The relative pose of the drone is determined based on the relative pose of the drone control relative to the platform control.

4. The method according to claim 3, further comprising: If the user interaction operation on the drone control is detected to meet the preset operation conditions, the relative pose of the drone is updated.

5. The method according to claim 4, wherein, The display shows multiple marker points, each marker point being used to indicate the relative pose of the corresponding UAV. The step of updating the relative pose of the drone upon detecting that a user interaction operation on the drone control meets preset operation conditions includes: If a user interaction is detected to move the drone control to a target marker, the drone's relative pose is updated based on the target marker.

6. The method according to any one of claims 1-5, wherein, The relative pose of the drone includes at least one of the following: drone following angle, drone altitude above the ground, and drone following distance.

7. The method according to claim 1, further comprising: If the obstacle tracking view indicated by the UAV perception data does not meet the preset obstacle tracking conditions, the relative pose of the UAV or the pose of the sensors installed on the UAV is updated.

8. The method according to any one of claims 1-7, wherein, The process of fusing the high-altitude perspective perception data and the platform-side perception data to construct an obstacle perception map includes: The first obstacle information and the second obstacle information corresponding to the high-altitude perspective perception data and the platform-side perception data are analyzed respectively; If the first obstacle information and the second obstacle information are detected to correspond to the same obstacle, the obstacle outline is reconstructed based on the first obstacle information and the second obstacle information, and the corresponding obstacle location is calculated; An obstacle perception map is constructed based on the obstacle outline and the obstacle location.

9. The method according to claim 8, wherein, If the first obstacle information and the second obstacle information are detected to correspond to different obstacles, the method further includes: The first obstacle type and the first obstacle location corresponding to the first obstacle information are analyzed, and the second obstacle type and the second obstacle location corresponding to the second obstacle information are analyzed. Determine a first obstacle graphic component and a second obstacle graphic component that correspond to the first obstacle type and the second obstacle type, respectively; The first obstacle graphic component is labeled at the location of the first obstacle, and the second obstacle graphic component is labeled at the location of the second obstacle, thereby constructing an obstacle perception map.

10. The method according to claim 3, wherein, After fusing the high-altitude perspective perception data and the platform-side perception data to construct an obstacle perception map, the method further includes: The obstacle perception map is displayed in stereo based on the second display.

11. The method according to claim 10, wherein, The method of displaying the obstacle perception map in stereoscopic form on a second display includes: In the second display, a target display area matching the position of the third obstacle is determined; the position of the third obstacle is the position of the third obstacle in the obstacle perception map; The third obstacle is displayed based on the target display area.

12. The method according to claim 10 or 11, wherein, The first display and / or the second display are mounted on the steering wheel of the vehicle.

13. An obstacle sensing system, comprising: The acquisition unit is used to acquire UAV perception data, platform-side perception data, and UAV relative pose; the UAV relative pose is used to describe the pose parameters of the UAV relative to the mobile platform. The conversion unit is used to convert the UAV perception data to the platform coordinate system according to the relative pose of the UAV, so as to obtain the corresponding high-altitude perspective perception data. The fusion unit is used to fuse the high-altitude perspective perception data and the platform-side perception data to construct an obstacle perception map.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein, The processor executes the computer program to implement the steps of the method according to any one of claims 1-12.

15. A computer-readable storage medium having a computer program / instructions stored thereon, wherein, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-12.

16. A computer program product comprising a computer program / instructions, wherein, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-12.

17. A mobile platform, wherein, The mobile platform includes a first sensor, a memory, and a processor; The first sensor is used to collect first sensing information; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the steps of the method as described in any one of claims 1-12.

18. The mobile platform according to claim 17, wherein, The mobile platform also includes a second sensor mounted on the drone, which is used to collect second sensing information.

19. The mobile platform according to claim 18, wherein, The mobile platform also includes a drone, on which the second sensor is mounted.

20. The mobile platform according to claim 18, wherein, The mobile platform further includes a hangar, which is installed on the mobile platform and used to house the drone.

Images