Method and system for displaying panoramic image of area in front of vehicle, storage medium, and electronic device

Abstract

A method and a system for displaying a panoramic image of an area in front of a vehicle, a storage medium, and an electronic device. The method comprises: acquiring an image of an area in front of a vehicle, and generating a panoramic display image on the basis of the image of the area in front of the vehicle (S1); acquiring a real-time vehicle speed and steering wheel angle data of the vehicle during travel, calculating a predicted travel trajectory of the vehicle on the basis of the real-time vehicle speed and the steering wheel angle data, and combining the predicted travel trajectory with the panoramic display image to generate a first panoramic image (S2); and acquiring information on an obstacle in front of the vehicle, determining a position of the obstacle by combining said information with the first panoramic image, annotating the position of the obstacle in the first panoramic image and generating a second panoramic image, and displaying the second panoramic image via a PHUD display module (S3).

Description

A method, system, storage medium, and electronic device for displaying a panoramic image of the front of a vehicle.

[0001] This application claims priority to Chinese Patent Application No. 202411840770.7, filed on December 13, 2024, entitled "A Method, System, Storage Medium and Electronic Device for Displaying Panoramic Images in Front of a Vehicle", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application belongs to the field of automotive intelligent driving technology, and in particular relates to a method and system for displaying a panoramic image in front of a vehicle. Background Technology

[0003] Among related technologies, the mainstream solution for displaying the driver's field of vision is to use a 360-degree camera system to capture images of the vehicle's surroundings and project them onto the central control screen. This system uses a front-facing camera to capture images of the area in front of the vehicle and then projects these images onto the central control screen, allowing the driver to clearly observe the environment in front of the vehicle, including blind spots and obstacles. The drawback of this solution is that displaying the front-facing images on the central control screen can distract the driver from their driving attention.

[0004] In related technologies, patent application CN202210438601.5 discloses a display system for an intelligent cockpit and an intelligent vehicle. The display system includes an immersive instrument panel, an A-pillar display screen, and a glasses-free 3D instrument panel. The immersive instrument panel displays blind spot image information, which is the image information in the driver's field of vision area obscured by the hood of the intelligent vehicle. The A-pillar display screen displays A-pillar blind spot image information, which is the image information in the driver's field of vision area obscured by the A-pillars on both sides of the intelligent vehicle. The glasses-free 3D instrument panel is set on the dashboard of the intelligent vehicle and displays map information of the intelligent vehicle's current location in a floating manner. Although it can display image information of the blind spot in front of the vehicle, there is a physical border, and the displayed frontal field of vision is relatively narrow. The floating display cannot better integrate with the real scene. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a method and system for displaying a panoramic image of the front of a vehicle. The method involves acquiring images of the front of the vehicle, processing these images to generate a panoramic display image, calculating the vehicle's predicted trajectory based on real-time vehicle speed and steering wheel angle data, and combining this with the panoramic display image to generate a first panoramic image, acquiring information about obstacles in front of the vehicle, and combining this information with the first panoramic image to determine the location of the obstacles. The location of the obstacles is then marked in the first panoramic image to generate a second panoramic image, which is displayed by a PHUD display module. This method effectively displays a panoramic image of the front of the vehicle to the driver, including the predicted trajectory and obstacle locations.

[0006] This application is achieved through the following technical solution:

[0007] In one aspect, embodiments of this application provide a method for displaying a panoramic image of the front of a vehicle, the method comprising:

[0008] Acquire the front view of the vehicle and generate a panoramic display image based on the front view;

[0009] The system acquires real-time vehicle speed and steering wheel angle data, calculates the predicted driving trajectory of the vehicle based on the real-time vehicle speed and steering wheel angle data, and generates a first panoramic image by combining the panoramic display image.

[0010] Obstacle information in front of the vehicle is acquired. The obstacle information is combined with the first panoramic image to determine the location of the obstacle in front of the vehicle. The location of the obstacle in front of the vehicle is marked in the first panoramic image to generate a second panoramic image. The second panoramic image is displayed by the PHUD display module.

[0011] Furthermore, when the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, a third panoramic image is generated based on the second panoramic image. The PHUD display module displays the third panoramic image to issue a warning signal to the driver.

[0012] Furthermore, the image acquisition module captures images of the front of the vehicle and converts them into front image data, which is then sent to the image processing module.

[0013] The image processing module extracts the front sub-images captured by each sub-camera of the image acquisition module based on the front image data.

[0014] Extract the front sub-images corresponding to the same frame of each front sub-image, perform image processing on each front sub-image, and determine the panoramic front image of the vehicle.

[0015] The front sub-images of each vehicle are extracted frame by frame, and the panoramic front image is determined frame by frame to synthesize the panoramic display image.

[0016] Furthermore, distortion correction processing is performed on the front image of the vehicle to correct image distortion caused by camera distortion;

[0017] Adjust the brightness and contrast of each vehicle front sub-image in the same frame to make the lighting of each vehicle front sub-image consistent.

[0018] A feature detector is used to extract feature points and descriptors from each front sub-image of the vehicle to characterize the features in the front sub-image of the vehicle.

[0019] The matching algorithm is used to match feature points between different front sub-images of vehicles and compare the similarity of descriptors to determine the correspondence between different front sub-images of vehicles.

[0020] Based on the matching results of feature points in the front sub-images, the homography matrix is ​​calculated, the front sub-images are aligned, and perspective transformation is performed using the homography matrix to map the front sub-images to the same coordinate system, thus correcting the front sub-images.

[0021] The corrected sub-images of the same frame are stitched together, and the overlapping areas of the stitched sub-images are processed to remove obvious seams at the stitching points.

[0022] Furthermore, based on the real-time vehicle speed and steering wheel angle data, a single-track model is used to calculate the vehicle's trajectory and generate a predicted trajectory.

[0023] The predicted driving trajectory and the panoramic display image are fused together to generate a second panoramic image.

[0024] Furthermore, information on obstacles in front of the vehicle is collected and sent to the obstacle warning module;

[0025] The obstacle warning module combines information about obstacles in front of the vehicle with the first panoramic image to determine the precise location and attributes of the obstacles; the information about obstacles in front of the vehicle includes distance, location, and attributes.

[0026] The location and attributes of obstacles in front of the vehicle are marked in the first panoramic image using boxes and / or text to generate a second panoramic image.

[0027] Furthermore, the data format of the obstacle information in front of the vehicle is converted to represent the spatial coordinates of the obstacle in front of the vehicle in the form of a point cloud, and the point cloud of the obstacle in front of the vehicle is determined.

[0028] Determine whether there is a spatial overlap between the vehicle's predicted trajectory and the point cloud of obstacles in front of the vehicle;

[0029] If there is a spatial overlap between the predicted driving trajectory and the point cloud of obstacles in front of the vehicle, a third panoramic image is generated based on the second panoramic image to issue a warning signal to the driver.

[0030] In another aspect, based on the same inventive concept, embodiments of this application also provide a panoramic imaging display system for the front of a vehicle, the system comprising: an image processing module, a trajectory prediction module, and an obstacle warning module;

[0031] The image processing module is used to acquire images of the front of the vehicle and generate panoramic display images based on these images.

[0032] The trajectory prediction module is used to acquire real-time vehicle speed and steering wheel angle data, calculate the predicted driving trajectory of the vehicle based on the real-time vehicle speed and steering wheel angle data, and generate the first panoramic image by combining the panoramic display image.

[0033] The obstacle warning module is used to acquire information about obstacles in front of the vehicle. The obstacle information is combined with the first panoramic image to determine the position of the obstacle in front of the vehicle, and the position of the obstacle in front of the vehicle is marked in the first panoramic image to generate a second panoramic image. It is also used to generate a third panoramic image based on the second panoramic image when the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory.

[0034] In another aspect, based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing one or more programs, which, when executed, can realize the aforementioned method for displaying panoramic images in front of a vehicle.

[0035] In another aspect, based on the same inventive concept, embodiments of this application also provide an electronic device, including a processor, a communication interface, the aforementioned computer-readable storage medium, and a communication bus; wherein the processor, the communication interface, and the computer-readable storage medium communicate with each other through the communication bus;

[0036] The processor is used to execute the program stored in the aforementioned computer-readable storage medium.

[0037] Compared with related technologies, this application has the following advantages:

[0038] 1. It reduces blind spots for drivers while driving, providing them with a wider field of vision in front of the vehicle, and the images displayed to the driver can be better integrated with the real scene.

[0039] 2. The visible area completely covers the front cabin of the vehicle, and the image is displayed more clearly on the windshield. It is not easily affected by external light and interference, and the driver can obtain key road information without shifting his gaze or being distracted, thus improving driving safety and convenience.

[0040] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0041] Figure 1 is a flowchart of a method for displaying a panoramic image of the front of a vehicle according to an embodiment of this application;

[0042] Figure 2 is a schematic diagram of a PHUD display provided in an embodiment of this application;

[0043] Figure 3 is a block diagram of a vehicle front panoramic image display system provided in an embodiment of this application;

[0044] Figure 4 is a structural schematic diagram of a vehicle according to an embodiment of this application;

[0045] Figure 5 is a structural schematic diagram of a vehicle according to an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] In one aspect, Figure 1 is a flowchart of a method for displaying a panoramic image of the front of a vehicle according to an embodiment of this application. As shown in Figure 1, this embodiment of the application provides a method for displaying a panoramic image of the front of a vehicle, including:

[0048] S1: Acquire the front view of the vehicle and generate a panoramic display image based on the front view.

[0049] S2: Acquire real-time vehicle speed and steering wheel angle data, calculate the predicted driving trajectory of the vehicle based on the real-time vehicle speed and steering wheel angle data, and generate the first panoramic image by combining the panoramic display image.

[0050] S3: Obtain information about obstacles in front of the vehicle. Combine the information with the first panoramic image to determine the location of the obstacles in front of the vehicle. Mark the location of the obstacles in front of the vehicle in the first panoramic image to generate a second panoramic image.

[0051] In this embodiment, an image acquisition module is used to acquire images of the front of the vehicle. The acquired images are preprocessed to generate a panoramic display image. Based on the real-time vehicle speed and steering wheel angle data, the predicted driving trajectory of the vehicle is calculated and combined with the panoramic display image to generate a first panoramic image. The first panoramic image includes the predicted driving trajectory in addition to the panoramic display image. The position of the obstacle in front of the vehicle is determined and marked in the first panoramic image to generate a second panoramic image. The second panoramic image includes the position of the obstacle in front of the vehicle in addition to the position of the obstacle in front of the vehicle in addition to the position of the obstacle in front of the vehicle. The second panoramic image is displayed to the driver using a PHUD display module.

[0052] The aforementioned panoramic display image refers to a wide-angle view of the vehicle's surrounding environment generated by synthesizing image data collected from multiple cameras (typically the front, rear, left, and right sides of the vehicle). The first panoramic image is generated by combining the panoramic display image with the vehicle's predicted driving trajectory. The predicted driving trajectory is based on the vehicle's real-time speed and steering wheel angle data, predicting the vehicle's possible path over a future period (e.g., a few seconds or minutes). The second panoramic image is an image based on the first panoramic image, with the positions of obstacles in front of the vehicle marked.

[0053] In some embodiments, the positions of obstacles in front of the vehicle are marked in the second panoramic image. Optionally, the positions of obstacles in front of the vehicle are displayed using different graphic symbols, symbols, or color codes. This application does not limit this. For example, the position of the first obstacle in front of the vehicle is marked as a triangle in the second panoramic image, and the position of the second obstacle in front of the vehicle is marked as a square in the second panoramic image.

[0054] As shown in Figure 2, the WHUD (Windshield Head-Up Display) technology, a related technology, floats above the vehicle's front cabin and displays at eye level, where the driver's normal line of sight is directly ahead. The image displayed by the WHUD is located in front of the driver's line of sight and may overlap with the actual road scene, interfering with the driver's normal vision and causing visual confusion and distraction. Furthermore, the information displayed by the WHUD may be affected by ambient light, reflections, and other factors, reducing image clarity and visibility.

[0055] Typically, a car's instrument panel is located in the center or front of the driver's cabin. When viewing the image ahead through the instrument panel, the driver needs to shift their gaze from the road directly in front to the instrument panel. This shift in gaze causes the driver's attention to be diverted from the road, reducing their observation and vigilance of the road conditions ahead, and increasing driving risks. This is especially true at high speeds or in complex road conditions, where a shift in the driver's gaze may lead to missing crucial information or causing dangerous situations.

[0056] Therefore, this solution utilizes PHUD (Panoramic Head-Up Display) technology to present a panoramic image of the vehicle's front to the driver. The PHUD image is projected between the WHUD display position and the instrument panel position, and the visible area of ​​the PHUD display technology completely covers the front cabin area of ​​the vehicle. The PHUD display module has a highly reflective black ink area printed on the lower side of the windshield. This design makes the image displayed on the windshield clearer and less susceptible to external light and interference. By using PHUD display technology, the driver can obtain key road information without shifting their gaze or losing focus, improving driving safety and convenience.

[0057] In some examples, the image in front of the vehicle is acquired, and a panoramic display image is generated based on the image. Specifically, this includes:

[0058] S11: Use the image acquisition module to capture the image in front of the vehicle, convert it into image data, and send the image data to the image processing module.

[0059] S12: The image processing module extracts the front sub-images captured by each sub-camera of the image acquisition module based on the front image data.

[0060] Specifically, the image acquisition module includes multiple sub-cameras used to capture images of the front of the vehicle from different angles. Therefore, the front sub-images captured by each sub-camera in the image acquisition module are extracted for subsequent determination of the panoramic display image. Here, the front sub-image refers to the independent viewpoint image captured by a single camera. For example, the image of the 120° field of view of the front of the vehicle covered by the forward-facing camera.

[0061] S13: Extract the front sub-images corresponding to the same frame of each front sub-image, perform image processing on each front sub-image, and determine the panoramic front image.

[0062] In some embodiments, it is ensured that all front sub-images are from the same moment to avoid stitching misalignment caused by vehicle movement. Here, the front panoramic image refers to the panoramic stitching result obtained from a single frame (or the same frame) of front sub-images.

[0063] S14: Extract the front sub-images corresponding to the same frame of each vehicle front sub-image frame by frame, determine the panoramic front image frame by frame, and synthesize the panoramic display image.

[0064] Specifically, the front-view image is split frame by frame, and each frame is processed. After processing each frame, a complete image, i.e., a panoramic display image, is synthesized. The panoramic display image includes blind spots that the driver cannot directly observe, such as the A-pillar of the car, thus effectively reducing blind spots for the driver during driving.

[0065] In some examples, the front sub-images corresponding to the same frame in each front sub-image are extracted, and image processing is performed on each front sub-image to determine the panoramic front image, specifically including:

[0066] S131: Perform distortion correction processing on the front image of the vehicle to correct image distortion caused by camera distortion.

[0067] Distortion correction refers to removing image distortion caused by the characteristics of camera lenses (such as wide-angle lenses, fisheye lenses, etc.). Distortion usually manifests as curved or distorted straight lines in the image. The purpose of distortion correction is to use mathematical models (usually camera intrinsic parameter models) to correct these distortions, making the image closer to the real scene.

[0068] S132: Adjust the brightness and contrast of each vehicle front sub-image in the same frame to make the lighting of each vehicle front sub-image consistent.

[0069] In some embodiments, during actual shooting, the brightness and contrast of the images captured by each camera may vary due to differences in the angle, position, or ambient light of each camera. Optionally, the brightness and contrast of each vehicle front sub-image in the same frame can be adjusted to ensure consistent lighting in each vehicle front sub-image, so that when stitching multiple images together, there will be no obvious differences between the images due to lighting differences.

[0070] S133: Use a feature detector to extract feature points and descriptors from each front sub-image of the vehicle to characterize the features in the front sub-image of the vehicle.

[0071] Feature points refer to points within stable regions of the vehicle front sub-image. These feature points can still be detected despite image rotation, scaling, and brightness changes. For example, feature points include corner points, edges, and textures in the vehicle front sub-image.

[0072] In this context, a descriptor is a numerical vector corresponding to a feature point. Descriptors describe the content of the region surrounding a feature point, thus helping to match the same feature point in different images.

[0073] S134: Use a matcher to match feature points between different front sub-images of vehicles and compare the similarity of descriptors to determine the correspondence between different front sub-images of vehicles.

[0074] Specifically, feature detectors include ORB (Oriented FAST and Rotated BRIEF) or SIFT (Scale-Invariant Feature Transform), and matchers include FLANN (Fast Library for Approximate Nearest Neighbors) or BFMatcher (Brute-Force Matcher).

[0075] S135: Based on the matching results of feature points of the front sub-images, calculate the homography matrix, align each front sub-image, and use the homography matrix to perform perspective transformation, mapping each front sub-image to the same coordinate system to correct the front sub-images.

[0076] The homography matrix is ​​a 3×3 linear transformation matrix used to map corresponding points of the front sub-image to the same coordinate system. Optionally, front sub-images from different perspectives (such as images captured by front-view and side-view cameras) can be geometrically transformed to align to the same coordinate system, eliminating distortion caused by differences in viewing angles.

[0077] S136: Stitch together the front sub-images of each vehicle in the same frame after correction, and process the overlapping areas after stitching the front sub-images to remove obvious seams at the stitching points.

[0078] In some examples, real-time vehicle speed and steering wheel angle data are acquired. Based on this data, the predicted vehicle trajectory is calculated. Combined with panoramic imagery, a first panoramic image is generated. Specifically, this includes:

[0079] S21: Based on the real-time vehicle speed and steering wheel angle data, a single-track model is used to calculate the vehicle's trajectory and generate a predicted trajectory.

[0080] S22: The predicted driving trajectory and the panoramic display image are fused to generate a second panoramic image.

[0081] In some examples, information about obstacles in front of the vehicle is acquired. This information is then combined with a first panoramic image to determine the location of the obstacles. The location of the obstacles is then marked in the first panoramic image to generate a second panoramic image. Specifically, this includes:

[0082] S31: Collect information on obstacles in front of the vehicle and send the information to the obstacle warning module.

[0083] S32: The obstacle warning module combines information about obstacles in front of the vehicle with the first panoramic image to determine the precise location and attributes of the obstacles.

[0084] Specifically, obstacle information in front of the vehicle includes distance, location, and attributes. Computer vision algorithms can be used to identify obstacles appearing in the first panoramic image. These algorithms can automatically analyze various objects and obstacles in the image in front of the vehicle and accurately identify their positions and features. By identifying obstacle features, the attributes of the obstacles in front of the vehicle can be determined. Alternatively, an obstacle recognition module can be used directly to collect obstacle information in front of the vehicle. The attributes of obstacles in front of the vehicle include: obstacle type (e.g., people, vehicles, cones), and corresponding height, width, and other information.

[0085] S33: Mark the position and attributes of obstacles in front of the vehicle using boxes and / or text in the first panoramic image to generate a second panoramic image.

[0086] In some examples, the method for displaying a panoramic view of the front of the vehicle also includes generating a third panoramic view based on the second panoramic view when the position of an obstacle in front of the vehicle overlaps with the predicted driving trajectory, and using this third panoramic view to issue a warning signal to the driver. In addition to displaying a clearer and more comprehensive panoramic view of the front of the vehicle, it is also necessary to provide the driver with a specific warning to remind them to slow down, stop, or avoid the obstacle.

[0087] Furthermore, when the position of an obstacle in front of the vehicle overlaps with the predicted driving trajectory, a third panoramic image is generated based on the second panoramic image to issue a warning signal to the driver, specifically including:

[0088] S41: Convert the data format of the obstacle information in front of the vehicle to represent the spatial coordinates of the obstacle in front of the vehicle in the form of a point cloud, and determine the point cloud of the obstacle in front of the vehicle.

[0089] S42: Determine whether there is a spatial overlap between the vehicle's predicted driving trajectory and the point cloud of obstacles in front of the vehicle.

[0090] S43: If there is a spatial overlap between the predicted driving trajectory and the point cloud of obstacles in front of the vehicle, a third panoramic image is generated based on the second panoramic image to issue a warning signal to the driver.

[0091] Example 1:

[0092] S101: Acquire the front view of the vehicle and generate a panoramic display image based on the front view.

[0093] The image acquisition module in this application includes multiple sub-cameras for capturing images from multiple directions and angles. The image acquisition module captures images of the front of the vehicle, converts them into front image data, and transmits the front image data to the image processing module for processing.

[0094] After receiving the front view data, the image processing module extracts the front sub-images corresponding to the same frame of each front sub-image, performs image processing on each front sub-image, and determines the panoramic front image. The following explanation uses the determination of a single panoramic front image frame from the same frame of front sub-images as an example:

[0095] Distortion correction is performed on the front sub-images to correct image distortion caused by camera distortion. The brightness and contrast of each front sub-image in the same frame are adjusted to ensure consistent lighting and a uniform visual effect. Feature detectors are used to extract feature points and descriptors from each front sub-image to characterize its features. A matcher is used to match feature points between different front sub-images and compare the similarity of descriptors to determine the correspondence between them. Feature detectors include ORB (Oriented Fast and Rotated BRIEF) or SIFT (Scale-Invariant Feature Transform), and matchers include FLANN (Fast Library for Approximate Nearest Neighbors) or BFMatcher (Brute-Force Matcher).

[0096] Based on the feature point matching results of the front sub-images, a homography matrix is ​​calculated, and the front sub-images are aligned. Perspective transformation is then performed using the homography matrix to map each front sub-image to the same coordinate system, thus correcting the front sub-images. The corrected front sub-images of the same frame are then stitched together, and overlapping areas are processed to remove obvious seams, ultimately determining the panoramic front image.

[0097] Using the above method, the front sub-images corresponding to the same frame of each vehicle front sub-image are extracted frame by frame, and the front panoramic image is determined frame by frame. The front panoramic images of consecutive frames are then synthesized into a panoramic display image.

[0098] S102: Calculate the predicted driving trajectory of the vehicle and combine it with the panoramic display image to determine the first panoramic image.

[0099] In this embodiment, the vehicle's trajectory is calculated based on real-time vehicle speed and steering wheel angle data. During vehicle movement, a single-track model is used to calculate the vehicle's trajectory using real-time speed and steering wheel angle data. The single-track model is a kinematic model of the vehicle, which can be used to accurately calculate the vehicle's forward trajectory. Using the vehicle's real-time speed, steering wheel angle, and wheelbase, the single-track model can calculate the vehicle's turning radius and update the vehicle's current position within continuous time steps, thereby generating a predicted driving trajectory. The predicted driving trajectory can be combined with panoramic imagery and clearly displayed in the driver's field of vision via a PHUD display module, improving the driver's visibility and understanding.

[0100] S103: Determine the location of the obstacle in front of the vehicle, mark the location of the obstacle in the first panoramic image, and generate the second panoramic image.

[0101] In this embodiment, an obstacle recognition module is used to collect information about obstacles in front of the vehicle. This information includes distance, position, and attributes. The exact location and attributes of the obstacles are marked in the first panoramic image using boxes and / or text, generating a second panoramic image. The obstacle recognition module can be an ADAS (Advanced Driver Assistance System).

[0102] S104: When the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, a third panoramic image is generated.

[0103] In this embodiment, an ADAS (Advanced Driver Assistance System) is used as the obstacle recognition module. Sensors in the ADAS system, such as LiDAR and cameras, acquire obstacle information about the environment surrounding the vehicle. This obstacle information is converted into a data format, representing the spatial coordinates of the obstacles as point clouds, thus determining the obstacle point cloud. Based on the spatial overlap between the vehicle's predicted driving trajectory and the obstacle point cloud, it automatically identifies whether the vehicle's predicted driving trajectory conflicts with the obstacle. If the vehicle's predicted driving trajectory overlaps with the obstacle point cloud, a third panoramic image is generated based on the second panoramic image. The third panoramic image highlights the obstacle and issues a warning signal to alert the driver of potential dangers.

[0104] Furthermore, taking a vehicle driving on the road as an example, the ADAS system first captures images of vehicles ahead using its front-facing camera. Then, the ADAS system's front-facing radar uses visual algorithms to determine the distance to obstacles or vehicles ahead, marking obstacles in front of the vehicle in real time. Based on the vehicle's speed, steering wheel angle, and wheelbase, a predicted driving trajectory is determined. When the predicted trajectory overlaps with a marked obstacle, the system highlights the overlapping area as an obstacle warning, generating a third panoramic image. This third panoramic image, containing warning information, is then displayed directly in the front cabin using the PHUD display module, effectively alerting the driver to potential obstacle risks.

[0105] In some embodiments, when the location of an obstacle in front of the vehicle overlaps with the predicted driving trajectory, the predicted collision time is calculated based on the obstacle type and / or the relative speed with respect to the obstacle. The obstacle type includes, but is not limited to, one or more of the following: people, vehicles (vehicles other than the user-driven vehicle), trees, rocks, animals, utility poles, repair signs, traffic cones, or other obstacles. Relative speed refers to the relative speed between the obstacle and the user-driven vehicle.

[0106] In some embodiments, risk levels are determined based on the predicted collision time. There is a negative correlation between predicted collision time and risk level; that is, the shorter the predicted collision time, the higher the risk level, and the longer the predicted collision time, the lower the risk level.

[0107] In some embodiments, if the predicted collision time falls within a first time interval, the risk level is classified as high-risk, corresponding to a first warning signal. If the predicted collision time falls within a second time interval, the risk level is classified as medium-risk, corresponding to a second warning signal. If the predicted collision time falls within a third time interval, the risk level is classified as low-risk, corresponding to a third warning signal.

[0108] For example, a predicted collision time of less than 3 seconds is considered a high-risk level, in which case a red light flashes on the screen and the vehicle's seats vibrate. A predicted collision time of 3 seconds or more but less than 5 seconds is considered a medium-risk level, in which case a yellow light flashes on the screen and the vehicle emits an audible alarm. A predicted collision time of 5 seconds or more is considered a high-risk level, in which case a blue light flashes on the screen.

[0109] In some embodiments, when generating the third panoramic image, the type of driving scenario of the vehicle is determined, and the information density displayed by the PHUD display module is dynamically adjusted. The information density displayed by the PHUD display module varies under different driving scenarios. Optionally, the driving scenario type includes, but is not limited to, highway driving scenario, parking driving scenario, congested driving scenario, or one of other driving scenario types.

[0110] For example, in a high-speed driving scenario, the PHUD display module has a higher information density, such as only showing the predicted collision time and hiding obstacle attribute information. Conversely, in a parking scenario, the PHUD display module has a lower information density, displaying information specific to the high-speed driving scenario, obstacle attribute information, and the precise distance to obstacles.

[0111] In this embodiment of the application, a driver monitoring system is used to collect the driver's eye position data. The eye position data includes eye coordinates and gaze point information. In order to ensure that the driver's eye position data can be obtained in real time, the driver monitoring system periodically acquires and updates the eye position data to ensure that the driver's eye position data is updated in real time when the driver's gaze changes.

[0112] Based on the eye position data obtained by the driver monitoring system, the PHUD display module adjusts the position and angle of the panoramic image displayed in front of the vehicle, so that the projected panoramic image is integrated with the driver's actual line of sight, avoiding visual conflict.

[0113] Figure 3 is a block diagram of a front panoramic image display system provided in an embodiment of this application. As shown in Figure 3, this embodiment of the application also provides a front panoramic image display system, which includes an image processing module, a trajectory prediction module, and an obstacle warning module. The image processing module is used to acquire front images and generate panoramic display images based on the front images. The trajectory prediction module is used to acquire real-time vehicle speed and steering wheel angle data, calculate the predicted driving trajectory of the vehicle based on the real-time vehicle speed and steering wheel angle data, and generate a first panoramic image by combining the panoramic display images. The obstacle warning module is used to acquire obstacle information in front of the vehicle, determine the position of the obstacle in front of the vehicle by combining the obstacle information with the first panoramic image, and mark the position of the obstacle in front of the vehicle in the first panoramic image to generate a second panoramic image; the obstacle warning module is also used to generate a third panoramic image based on the second panoramic image when the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory.

[0114] Furthermore, the front panoramic imaging display system also includes a PHUD display module, which displays a second panoramic image to the driver, and displays a third panoramic image to issue a warning signal to the driver when there is an overlap between the position of the obstacle in front of the vehicle and the predicted driving trajectory.

[0115] In this embodiment, the front panoramic imaging display system further includes an image acquisition module and an obstacle recognition module. The image acquisition module captures images of the area in front of the vehicle, converts them into front image data, and sends the front image data to the image processing module. The obstacle recognition module collects information about obstacles in front of the vehicle, including distance, position, and attributes.

[0116] When deploying a front panoramic imaging display system, the system modules are divided into a data layer, a business logic layer, and an application layer. The data layer includes an image acquisition module and an obstacle recognition module; the business logic layer includes an image processing module, a trajectory prediction module, and an obstacle warning module; and the application layer includes a PHUD display module, which is used to display fused panoramic images, predict driving trajectories, obstacle positions, and possible warning information.

[0117] Figure 4 is a structural schematic diagram of a vehicle provided in an embodiment of this application. The vehicle may include:

[0118] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0119] When the processor 502 executes the program, it implements the display screen control method provided in the above embodiments.

[0120] Furthermore, the vehicle also includes:

[0121] Communication interface 503 is used for communication between memory 501 and processor 502.

[0122] The memory 501 is used to store computer programs that can run on the processor 502.

[0123] The memory 501 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0124] If the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one thick line is used in Figure 5, but this does not indicate that there is only one bus or one type of bus.

[0125] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0126] Processor 502 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of this application.

[0127] Furthermore, referring to Figure 5, the memory, processor, and communication interface in the vehicle can be provided by a computer device deployed on the vehicle. The processor described in Figure 4 can be implemented as a Central Processing Unit (CPU) 701, and the memory 501 includes a system memory 704 comprising a Random Access Memory (RAM) 702 and a Read-Only Memory (ROM) 703. The bus is implemented as a system bus 705 connecting the system memory 704 and the CPU 701. The vehicle also includes a mass storage device 706 for storing an operating system 709, application programs 710, and other program modules 711.

[0128] The mass storage device 706 is connected to the central processing unit 701 via a mass storage controller (not shown) connected to the system bus 705. The mass storage device 706 and its associated computer-readable media provide non-volatile storage for the vehicle. That is, the mass storage device 706 may include computer-readable media (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0129] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 704 and mass storage device 706 described above can be collectively referred to as memory.

[0130] According to various embodiments of this disclosure, the vehicle can also operate by connecting to a remote computer on a network such as the Internet. That is, the vehicle can connect to network 708 via network interface unit 707 connected to system bus 705, or it can use network interface unit 707 to connect to other types of networks or remote computer systems (not shown).

[0131] The memory also includes at least one computer program stored in the memory, and the central processing unit 701 executes the at least one computer program to implement all or part of the steps in the methods shown in the above embodiments.

[0132] This application also provides a computer-readable storage medium storing one or more programs, which, when executed, can implement the aforementioned method for displaying panoramic images in front of a vehicle.

[0133] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method for displaying a panoramic image of the front of a vehicle.

[0134] This application also provides a computer program product, including: a computer program or instructions, which, when executed, implement the above-described method for displaying a panoramic image of the front of a vehicle.

[0135] This application also provides an electronic device, including a processor, a communication interface, the aforementioned computer-readable storage medium, and a communication bus. The processor, communication interface, and computer-readable storage medium communicate with each other via the communication bus. The processor is used to execute a program stored in the aforementioned computer-readable storage medium.

[0136] It should be noted that the electrical connections between the above-mentioned units do not necessarily represent the connections between lines. Indirect connections are applicable to the embodiments of this application as long as they achieve the purpose of this application.

[0137] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for displaying a panoramic image of the front of a vehicle, the method comprising: Acquire the front view of the vehicle and generate a panoramic display image based on the front view of the vehicle; The system acquires real-time vehicle speed and steering wheel angle data, calculates the predicted driving trajectory of the vehicle based on the real-time vehicle speed and steering wheel angle data, and generates a first panoramic image by combining the panoramic display image. Obstacle information in front of the vehicle is acquired. The obstacle information is combined with the first panoramic image to determine the position of the obstacle in front of the vehicle. The position of the obstacle in front of the vehicle is marked in the first panoramic image to generate a second panoramic image. The PHUD display module displays the second panoramic image.

2. The method according to claim 1, wherein, The method for displaying a panoramic image of the front of the vehicle also includes: When the location of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, a third panoramic image is generated based on the second panoramic image, and the PHUD display module displays the third panoramic image to issue a warning signal to the driver.

3. The method according to claim 1 or 2, wherein, The process of acquiring the front view of the vehicle and generating a panoramic display image based on the front view includes: The image acquisition module captures the image in front of the vehicle, converts it into image data, and sends the image data to the image processing module. The image processing module extracts the front sub-images captured by each sub-camera of the image acquisition module based on the front image data, and each sub-camera captures the front image from different angles. Extract the front sub-images corresponding to the same frame of each of the front sub-images, perform image processing on each of the front sub-images, and determine the panoramic front image; The front sub-images corresponding to the same frame of each of the aforementioned front sub-images are extracted frame by frame, and the front panoramic image is determined frame by frame to synthesize the panoramic display image.

4. The method according to any one of claims 1 to 3, wherein, The step of extracting the front sub-images corresponding to the same frame of each of the aforementioned front sub-images, performing image processing on each of the aforementioned front sub-images, and determining the panoramic front image includes: The image of the vehicle front is subjected to distortion correction processing to correct image distortion caused by camera distortion; Adjust the brightness and contrast of each of the vehicle front sub-images in the same frame to make the lighting of each vehicle front sub-image consistent; Using a feature detector, feature points and descriptors are extracted from each of the vehicle front sub-images to characterize the features in the vehicle front images; The feature points between different vehicle front sub-images are matched using a matcher, and the similarity of the descriptors is compared to determine the correspondence between the different vehicle front sub-images. Based on the matching results of the feature points of the front sub-images, a homography matrix is ​​calculated, the front sub-images are aligned, and a perspective transformation is performed using the homography matrix to map the front sub-images to the same coordinate system, thereby correcting the front sub-images. The corrected sub-images of the same frame are stitched together, and the overlapping areas of the stitched sub-images are processed to remove obvious seams at the stitching points.

5. The method according to any one of claims 1 to 4, wherein, The process of acquiring real-time vehicle speed and steering wheel angle data, calculating the predicted driving trajectory of the vehicle based on the real-time vehicle speed and steering wheel angle data, and generating a first panoramic image by combining the panoramic display image includes: Based on the real-time vehicle speed and steering wheel angle data, a single-track model is used to calculate the vehicle's trajectory and generate the vehicle's predicted trajectory. The predicted driving trajectory and the panoramic display image are fused together to generate the second panoramic image.

6. The method according to any one of claims 1 to 5, wherein, The step of acquiring obstacle information in front of the vehicle, combining the obstacle information with the first panoramic image to determine the location of the obstacle, and marking the location of the obstacle in the first panoramic image to generate a second panoramic image includes: Collect the obstacle information in front of the vehicle and send the obstacle information to the obstacle warning module; The obstacle warning module combines the obstacle information in front of the vehicle with the first panoramic image to determine the location and attributes of the obstacle in front of the vehicle; wherein, the obstacle information in front of the vehicle includes distance, location, and attributes; The location and attributes of the obstacle in front of the vehicle are marked in the first panoramic image using boxes and / or text to generate the second panoramic image.

7. The method according to any one of claims 1 to 6, wherein, When the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory, a third panoramic image is generated based on the second panoramic image to issue a warning signal to the driver, including: The data format of the obstacle information in front of the vehicle is converted to represent the spatial coordinates of the obstacle in the form of a point cloud, and the point cloud of the obstacle in front of the vehicle is determined. Determine whether there is a spatial overlap between the predicted driving trajectory of the vehicle and the point cloud of obstacles in front of the vehicle; If there is a spatial overlap between the predicted driving trajectory and the point cloud of obstacles in front of the vehicle, the third panoramic image is generated based on the second panoramic image and used to issue a warning signal to the driver.

8. A panoramic imaging display system for the front of a vehicle, the system comprising: Image processing module, trajectory prediction module, and obstacle warning module; The image processing module is used to acquire the front image of the vehicle and generate a panoramic display image based on the front image of the vehicle. The trajectory prediction module is used to acquire real-time vehicle speed and steering wheel angle data, calculate the predicted driving trajectory of the vehicle based on the real-time vehicle speed and steering wheel angle data, and generate a first panoramic image by combining the panoramic display image. The obstacle warning module is used to acquire obstacle information in front of the vehicle, combine the obstacle information with the first panoramic image to determine the position of the obstacle in front of the vehicle, and mark the position of the obstacle in front of the vehicle in the first panoramic image to generate a second panoramic image; it is also used to generate a third panoramic image based on the second panoramic image when the position of the obstacle in front of the vehicle overlaps with the predicted driving trajectory.

9. A computer-readable storage medium storing one or more programs that, when executed, can implement the method for displaying a panoramic image of the front of a vehicle as described in any one of claims 1 to 7.

10. An electronic device, comprising a processor, a communication interface, a computer-readable storage medium as described in claim 9, and a communication bus; wherein, The processor, communication interface, and computer-readable storage medium communicate with each other via a communication bus; The processor is used to execute programs stored in a computer-readable storage medium.

Images