Image processing method, system and apparatus, and medium, device and vehicle

By integrating AVM and SR image processing methods, the problems of redundant and cluttered driving information are solved, achieving clear visualization of driving information and improving safety.

WO2026232922A1PCT designated stage Publication Date: 2026-11-12CHENGDU THUNDER SOFTWARE TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/080514
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-06-05
Filing Date
2026-02-28
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

In existing technologies, the visual processing of driving information in surround reality technology leads to redundancy and clutter, increases the cognitive load on drivers, and poses a risk of obscuring key information.

Method used

By fusing images from the surround view monitoring system (AVM) and the surrounding display perception system (SR), image content outside the preset display range is retained, and image fusion is performed within the preset display range to avoid repeatedly visualizing the same driving information.

Benefits of technology

It effectively reduces visual redundancy and clutter in driving information, lowers the driver's cognitive load, and improves the visualization and safety of driving information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026080514_12112026_PF_FP_ABST
    Figure CN2026080514_12112026_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed in the embodiments of the present application are an image processing method, system and apparatus, and a medium, a device and a vehicle, which can solve the problem of visual redundancy and clutter in driving information. The method comprises: acquiring a first image and a second image, wherein the first image is a current image frame rendered by using a surrounding-reality (SR) system of a vehicle, and the second image is a current image frame rendered by using an around-view-monitor (AVM) system of the vehicle; and fusing the first image and the second image to obtain a fused image, wherein image content of the fused image comprises: image content of the first image or the second image outside a preset display range, and image content of the fused first image and second image within the preset display range.
Need to check novelty before this filing date? Find Prior Art

Description

An image processing method, system, apparatus, medium, device, and vehicle

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202510749416.1, filed on June 5, 2025, entitled "An Image Processing Method, System, Apparatus, Medium, Device and Vehicle", and Chinese Patent Application No. 202510594954.8, filed on May 9, 2025, entitled "An Image Processing Method, System, Apparatus, Medium, Device and Vehicle", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of graphics rendering technology, and in particular to an image processing method, system, apparatus, medium, device, and vehicle. Background Technology

[0004] Surround reality technology refers to the perception and visualization of driving information such as traffic participants and road facilities around a vehicle based on onboard perception capabilities. It can provide visual decision-making basis for advanced autonomous driving functions and help drivers observe the surrounding environment of the vehicle, thereby improving driving safety.

[0005] In related technologies, the real-world video stream rendered using an Around View Monitor (AVM) system is overlaid as an independent layer on top of a scene rendered using a Surrounding Reality (SR) system to achieve a visual display of driving information.

[0006] However, this image processing method of overlaying two layers of images can easily cause redundancy and clutter in the visual field of driving information, which in turn can lead to the risk of obscuring key driving information and increase the cognitive load on the driver. Summary of the Invention

[0007] This application provides an image processing method, system, apparatus, medium, device, and vehicle, which can solve the problem of visual redundancy and clutter in driving information. The technical solution is as follows:

[0008] Firstly, an image processing method is provided, the method comprising:

[0009] Acquire a first image and a second image. The first image is the current frame image rendered using the vehicle's Surround View Perception System (SR), and the second image is the current frame image rendered using the vehicle's Ambient Vision Monitoring System (AVM).

[0010] The first image and the second image are merged to obtain a merged image. The image content of the merged image includes: the image content of the first image or the second image outside the preset display range, and the image content of the first image and the second image after merging within the preset display range.

[0011] Secondly, an image processing system is provided, comprising an AVM module, an image rendering engine, and a frame buffer object (FBO). The FBO is communicatively connected to both the AVM module and the image rendering engine, wherein:

[0012] The AVM module is configured to render consecutive frames of images based on AVM and transmit the rendered current frame image to the FBO for storage.

[0013] The image rendering engine is configured to render consecutive frame images based on SR, and to fuse the current frame image stored in FBO with the current frame image rendered based on SR to obtain a fused image. The image content of the fused image includes: the image content of the current frame image rendered by the AVM module or the current frame image rendered based on SR outside the preset display range, and the image content of the current frame image rendered by the AVM module and the current frame image rendered based on SR within the preset display range.

[0014] Thirdly, an image processing system is provided, comprising an AVM algorithm unit, an AVM data source, and an image rendering engine. The AVM data source is communicatively connected to both the AVM algorithm unit and the image rendering engine, wherein:

[0015] The image rendering engine is configured to render consecutive frames of images based on SR;

[0016] The AVM algorithm unit is configured to render consecutive frame images based on AVM and send the rendered current frame image to the AVM data source.

[0017] The AVM data source is configured to send the current frame image it receives to the image rendering engine;

[0018] The image rendering engine is also configured to perform the following steps upon receiving the current frame image from the AVM data source:

[0019] The third and fourth images are merged to obtain a merged image;

[0020] The merged image is output to the vehicle's display device;

[0021] The third image is the current frame image received by the image rendering engine, the fourth image is the current frame image rendered by the image rendering engine, and the image content of the fused image includes: the image content of the third image or the fourth image outside the preset display range, and the image content of the third image and the fourth image after fusion within the preset display range.

[0022] Fourthly, an image processing apparatus is provided, the apparatus comprising:

[0023] The communication module is used to acquire a first image and a second image. The first image is a current frame image rendered using the vehicle's Surround View Perception System (SR), and the second image is a current frame image rendered using the vehicle's Panoramic View Monitoring System (AVM).

[0024] The processing module is used to fuse the first image and the second image to obtain a fused image. The image content of the fused image includes: the image content of the first image or the second image outside the preset display range, and the image content of the first image and the second image after fusion within the preset display range.

[0025] Fifthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction, the at least one instruction being executed by a processor to implement the image processing method of the first aspect.

[0026] In a sixth aspect, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image processing method of the first aspect.

[0027] In a seventh aspect, a vehicle is provided, which includes electronic equipment as in the fifth aspect, or an image processing system as in the second aspect.

[0028] In the image processing method provided in this application embodiment, by retaining the image content represented by the first image or the second image outside the preset display range, and fusing the image content represented by the first image and the second image within the preset display range, the same driving information perceived by the SR and AVM can be avoided by using the mutual complementarity of the perception range to ensure the comprehensiveness of the perceived driving information. This avoids redundant and cluttered driving information in terms of perspective, thereby reducing the risk of key driving information being obscured and significantly reducing the cognitive load on the driver. Attached Figure Description

[0029] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 is a schematic diagram of a driving information visualization display in related technologies;

[0031] Figure 2 is a structural block diagram of an electronic device provided in an embodiment of this application;

[0032] Figure 3 is a flowchart of an image processing method provided in an embodiment of this application;

[0033] Figure 4 is a schematic diagram of a near-field obstacle-triggered fusion scenario provided in an embodiment of this application;

[0034] Figure 5 is a schematic diagram of an intersection turning activation fusion scenario provided by an embodiment of this application;

[0035] Figure 6 is a schematic diagram of an automatic parking fusion scenario provided by an embodiment of this application;

[0036] Figure 7 is a schematic diagram of a fused image under a default view fusion scheme provided in an embodiment of this application;

[0037] Figure 8 is a schematic diagram of a fused image under a top-view fusion scheme provided in an embodiment of this application;

[0038] Figure 9 is a schematic diagram of a fused image under a forward-view fusion scheme provided in an embodiment of this application;

[0039] Figure 10 is a schematic diagram of a fused image under a left-view fusion scheme provided in an embodiment of this application;

[0040] Figure 11 is a schematic diagram of a fused image under a right-view fusion scheme provided in an embodiment of this application;

[0041] Figure 12 is a schematic diagram of a fused image under a close-up fusion scheme provided in an embodiment of this application;

[0042] Figure 13 is a schematic diagram of a fused image under a left rear view fusion scheme provided in an embodiment of this application;

[0043] Figure 14 is a schematic diagram of a fused image under a right-side rear view fusion scheme provided in an embodiment of this application;

[0044] Figure 15 is a schematic diagram of the sensing range distribution of SR, AVM and SD Map provided in an embodiment of this application;

[0045] Figure 16 is a schematic diagram of the sensing distance of SR, AVM and SD Map provided in an embodiment of this application;

[0046] Figure 17 is an architecture diagram of an image processing system provided in an embodiment of this application;

[0047] Figure 18 is an architecture diagram of another image processing system provided in an embodiment of this application;

[0048] Figure 19 is a timing diagram of real-time rendering of a system provided in an embodiment of this application;

[0049] Figure 20 is a schematic diagram of an opaque fusion effect provided in an embodiment of this application;

[0050] Figure 21 is a schematic diagram of a semi-transparent fusion effect provided in an embodiment of this application;

[0051] Figure 22 is an architecture diagram of another image processing system provided in an embodiment of this application;

[0052] Figure 23 is a block diagram of an image processing apparatus provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0054] Please refer to Figure 1 for a schematic diagram of the visualization of driving information in related technologies. In surround reality technology, the current main approach is to overlay the real-world video stream rendered by AVM onto the SR rendering scene as an independent layer. For example, as shown in Figure 1, an AVM window is overlaid on the full-screen SR screen to achieve a visualization of the driving information perceived by SR and AVM.

[0055] However, this dual-layer image processing method can easily lead to redundancy and clutter in the visual perspective of driving information, resulting in confusion in the visual priority of key driving information (such as lane lines and obstacles), posing a risk of occlusion, and increasing the cognitive load on the driver, potentially delaying decision-making in emergency situations. Furthermore, the spatial coordinate system deviation caused by the dual-layer image overlay can easily lead to inconsistencies in human-computer interaction logic, resulting in interaction contradictions (such as the difficulty in associating the predicted path displayed in the SR image with the AVM real-world view), thus affecting the user experience.

[0056] Based on the above analysis, and addressing the problem of redundant and cluttered driving information in related technologies from a visual perspective, this application provides an image processing method, system, device, medium, equipment, and vehicle. It performs deep data fusion on the driving information perceived by the SR and AVM to avoid repetitive visualization of the same driving information perceived by the SR and AVM. Compared to image processing methods that overlay two layers of images, this application can effectively avoid redundancy and clutter in the visual perspective of driving information, thereby reducing the risk of key driving information being obscured and significantly reducing the cognitive load on the driver.

[0057] First, to facilitate understanding of the technical solutions provided in this application, the main technical concepts involved in the embodiments of this application will be briefly explained below.

[0058] SR: This is an intelligent driving visualization system based on advanced in-vehicle perception systems and artificial intelligence (AI) algorithms. Through multi-sensor fusion (including cameras, millimeter-wave radar, and lidar), it constructs and dynamically renders a 360° view of the traffic environment surrounding the vehicle in real time. With high-precision 3D modeling technology at its core, the system digitizes driving information such as road infrastructure, traffic participants (vehicles, pedestrians, and non-motorized vehicles), lane markings, and traffic signals in real time. This information is then intuitively displayed through the in-vehicle central control screen, instrument panel, or augmented reality head-up display (AR-HUD), providing drivers with a "God's-eye view" of the overall road conditions and offering visualized decision-making support for advanced autonomous driving functions.

[0059] AVM, also known as a 360° panoramic imaging system, is a panoramic vision assistance technology for automobiles. This technology stitches together images from multiple perspectives using onboard cameras to provide the driver with a 360° overhead view of driving information, helping the driver to more clearly observe the surrounding environment and improving safety during parking, low-speed driving, and navigating narrow roads.

[0060] Standard Definition Map (SD MAP): This is a map type that is different from High Definition Map (HD Map) and is usually suitable for general navigation or basic geographic information query scenarios.

[0061] Image rendering engines, such as the Kanzi engine, are mainly used to render 2D / 3D models, textures, lighting and other data into high-fidelity images in real time or offline, based on computer graphics.

[0062] Please refer to Figure 2, which shows a structural block diagram of an electronic device provided in an exemplary embodiment of this application. This electronic device can be an in-vehicle central control unit, instrument cluster controller, smart cockpit, or multi-screen digital cockpit, capable of running applications. The electronic device in this application may include one or more of the following components: a processor 110, a memory 120, and an input / output device 130.

[0063] Processor 110 may include one or more processing cores. Processor 110 connects to various parts of the electronic device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. In some embodiments of this application, processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 110 may integrate one or more of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0064] The memory 120 may include random access memory (RAM) or read-only memory. In some embodiments of this application, the memory 120 includes a non-transitory computer-readable storage medium. The memory 120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described below, etc. The operating system may be an Android system (including systems deeply developed based on the Android system), an iOS system (including systems deeply developed based on the iOS system), or other systems. The data storage area may also store data created by the electronic device during use (such as audio and video data).

[0065] Please refer to Figure 3, which shows a flowchart of an image processing method according to an embodiment of this application. This method can be applied to the electronic device described above. The method may include the following steps:

[0066] Step S101: Acquire a first image and a second image. The first image is the current frame image rendered using the vehicle's Surround View Perception System (SR), and the second image is the current frame image rendered using the vehicle's Panoramic View Monitoring System (AVM).

[0067] In practice, a FrameBuffer Object (FBO) is created in advance, and the first image received from the SR algorithm module and the second image received from the AVM algorithm module are stored in the FBO, or the first image received from the SR algorithm module and the second image received from the AVM algorithm module are stored in separate FBOs for subsequent fusion.

[0068] In some embodiments of this application, the first image is the current frame image rendered using the vehicle's image rendering engine, which integrates an SR algorithm module and a map (such as SD MAP) algorithm module.

[0069] Step S102: Merge the first image and the second image to obtain a merged image. The image content of the merged image includes: the image content of the first image or the second image outside the preset display range, and the image content of the first image and the second image after merging within the preset display range.

[0070] In practice, the preset display range can be flexibly adjusted according to user instructions or the vehicle's driving status. For example, the preset display range can be adjusted to the full (or partial) perception range of the SR or AVM to support different interaction needs.

[0071] The image content represented by the second image within a preset display range is merged with the image content represented by the first image within a preset display range to combine two sets of driving information visualized using SR and AVM respectively into one set. This avoids the repeated visualization of the same driving information and effectively reduces the information pressure on the driver.

[0072] For example, for image content rendered using SR and AVM for the same driving information (i.e., the same driving information visualized, such as the same obstacle visualized), only the image content rendered by SR can be retained, or only the image content rendered by AVM can be retained.

[0073] For example, for image content rendered using SR and AVM for the same driving information, new image content can be generated based on the two image contents (using a generative model). This allows for a more accurate visualization of the relevant driving information by combining the three-dimensional contours, colors, and textures reflected in the respective image contents by SR and AVM.

[0074] If the image content to be included within and outside the preset display range comes from different images (for example, if the image content to be included outside the preset display range comes from the first image, and the image content to be included within the preset display range comes from the second image), then the corresponding image content within and outside the preset display range is processed by coordinate alignment and other methods to merge the two into a complete image, thereby obtaining a fused image.

[0075] It is understandable that if the preset display range does not include the overlapping area of ​​the perception range of SR and AVM, it can be regarded as only SR-rendered image content or only AVM-rendered image content exists within the preset display range.

[0076] If the image content existing within the preset display range and the image content required to be included outside the preset display range belong to the same image (such as both belonging to the first image and both belonging to the second image), the image fusion step can be skipped (such as directly displaying the first image or the second image), thereby achieving a full-screen display effect of SR or AVM screen.

[0077] If the image content within the preset display range and the image content required to be included outside the preset display range are not from the same image (i.e., the image content required to be included outside the preset display range comes from different images), then the corresponding image content within and outside the preset display range will be processed such as coordinate alignment to merge the two into a complete image, thereby obtaining a fused image.

[0078] Therefore, by deeply integrating SR virtual scenes with AVM real scenes (such as overlaying lane line prediction information displayed by SR visualization with obstacles displayed by AVM visualization), the cognitive load on drivers can be reduced, creating an immersive experience of "what you see is what you get".

[0079] In some embodiments of this application, blind spot risks (such as the risk of a child suddenly running into the image) can be marked in the fused image to transform the safety features of the Advanced Driving Assistance System (ADAS) into value points that users can perceive, supporting the upward pricing of vehicle models.

[0080] In summary, the technical solution provided in this application retains the image content represented by the first image or the second image outside the preset display range, and fuses the image content represented by the first image and the second image within the preset display range. Thus, while utilizing the mutual complementarity of the perception range of SR and AVM to ensure the comprehensiveness of the perceived driving information, it can avoid the repeated visualization of the same driving information perceived by SR and AVM, thereby avoiding redundancy and clutter in the perspective of driving information. This can reduce the risk of key driving information being obscured and significantly reduce the cognitive load on the driver.

[0081] In some embodiments of this application, the method further includes:

[0082] Check if the vehicle meets the preset activation conditions;

[0083] When a vehicle is detected to meet preset activation conditions, a fused image is displayed on the vehicle's display device;

[0084] If the vehicle does not meet the preset activation conditions, a first image or a second image is displayed on the vehicle's display device.

[0085] In some embodiments of this application, preset activation conditions are set so that the fusion rendering function of SR and AVM can be automatically turned on and off according to the specific scene in which the vehicle is located.

[0086] When the vehicle does not meet the preset activation conditions, the fusion rendering function of SR and AVM is automatically turned off. At this time, only the current frame image (i.e., the first image) rendered using SR (and map, such as SD MAP) or the current frame image rendered using AVM (i.e., the second image) is displayed on the vehicle's display device (such as the in-vehicle central control screen, instrument screen or AR-HUD). This can reduce computing overhead and ensure display efficiency.

[0087] When the vehicle meets the preset activation conditions, the fusion rendering function of SR and AVM is automatically activated. At this time, the fused image obtained based on the first image and the second image is displayed on the display device to ensure the comprehensiveness of driving information.

[0088] In some embodiments of this application, the preset activation conditions include at least one of the following:

[0089] The target object was detected within a preset range where the vehicle was located, such as a static obstacle, a moving pedestrian, or a low object.

[0090] The vehicle's speed is less than the set speed and the turn signal is turned on, for example, the vehicle's speed is less than 15 km / h and the turn signal is turned on.

[0091] The vehicle is in a parked state.

[0092] In some embodiments of this application, the image content after fusing the first image and the second image within a preset display range includes at least one of the following:

[0093] For the first driving information, the first image visualizes the image content within a preset display range; and for the second driving information, the second image visualizes the image content within a preset display range. The second driving information is different from the first driving information. For example, the image content of the first image included in the preset display range of the fused image can be a 3D scene rendered by SR (and map, such as SD Map), and the image content of the second image included in the preset display range of the fused image can be the image content rendered by AVM for driving information not perceived by SR (and map, such as SD Map) (corresponding to the second driving information, such as static obstacles, moving objects, etc.).

[0094] For the same driving information, the first image or the second image visualizes the image content within a preset display range; for example, within the preset display range, the fused image may include the image content rendered by SR (and map, such as SD Map) or AVM for the same obstacle.

[0095] In some embodiments of this application, the image content included in the fused image is flexibly adjusted according to the vehicle's driving state (which can be used to characterize different scenarios in which the vehicle is located).

[0096] For example, when a vehicle passes a pedestrian crossing, the fused image (such as the image content of the fused image within a preset display range) can include a 3D road scene map rendered by SR (and a map, such as SD Map) (including building models and road models, etc.) and pedestrian images rendered by AVM, so that the fused image can accurately describe moving pedestrians on the road based on AVM, thereby improving driving safety.

[0097] For example, when a vehicle is driving normally, the fused image (such as the image content of the fused image within a preset display range) can include the image content rendered by SR (and map, such as SD Map) for buildings and roads, so that the fused image can provide a more comprehensive description of the buildings and roads around the vehicle based on SR (and map, such as SD Map), thereby improving the driving experience.

[0098] For example, when a vehicle is in a parking scenario, the fused image (such as the image content of the fused image within a preset display range) can include the image content rendered by AVM for the ground, so that the fused image can accurately describe the ground of the parking lot based on AVM, thereby improving the parking experience.

[0099] In some embodiments of this application, in response to detecting that the vehicle's driving state conforms to a first preset state (e.g., parking state, low-speed driving state, or detection of a target object within a preset range of the vehicle), the fused image includes obstacles visualized in the second image within a preset display range. These obstacles are those not perceived by the SR (due to the lack of a pre-set related model, etc.), ensuring an accurate description of obstacles surrounding the vehicle. In some embodiments of this application, the target object includes at least one of static obstacles, moving pedestrians, and low-lying objects.

[0100] In some embodiments of this application, in response to detecting that the vehicle's driving state conforms to a second preset state (such as the vehicle passing through a turning intersection or pedestrian crossing), the fused image includes the moving objects visualized in the second image within a preset display range, so as to achieve enhanced feedback on moving objects around the vehicle.

[0101] For example, referring to the schematic diagram of a near-distance obstacle-triggered fusion scenario shown in Figure 4, if a target object is detected within a preset range of the vehicle, such as a static obstacle within 1.0 meter (adjustable from 0.3 to 1.0 meter), a moving pedestrian within 3.0 meter (adjustable from 1.0 to 3.0 meter), or a low-lying object within 0.5 meters, it indicates that the vehicle is in a near-distance obstacle-triggered fusion scenario. In this case, the obstacle visualized in the second image within a preset display range is used to supplement the obstacle information visualized in the first image within the preset display range. This allows AVM imagery to supplement the details of physical world obstacles that SR (and maps, such as SD Map) cannot perceive, making it easier for users to make driving decisions.

[0102] For example, referring to the schematic diagram of the intersection turning activation fusion scenario shown in Figure 5, when a vehicle turns at an intersection (for example, if the vehicle speed is detected to be less than the set speed and the turn signal is activated, it can be determined that it is turning at the intersection) and / or passing through a pedestrian crossing, the moving objects visualized in the second image within a preset display range are used to replace the moving objects visualized in the first image within the preset display range. For example, AVM images are used to display moving pedestrians in the 3D scene rendered by SR (and maps, such as SD Map). Thus, by combining AVM images with the scene rendered by SR (and maps, such as SD Map) to enhance the feedback on moving objects in the physical world, driving safety is improved.

[0103] For example, referring to the schematic diagram of the automatic parking fusion scenario shown in Figure 6, when the vehicle is in a parking state, it can be determined that the vehicle is in an automatic parking fusion scenario. At this time, the obstacles visualized in the second image within the preset display range are used to supplement the obstacles visualized in the first image within the preset display range. In this way, AVM images can be used in parking lots with relatively complex environments to supplement the details of physical world obstacles that cannot be perceived by SR (and maps, such as SD Map), thereby enhancing the sense of control and trust in parking and improving the user's parking experience.

[0104] In some embodiments of this application, the first image and the second image are fused, including:

[0105] Adjust the image content of the first image within the first target range and the image content outside the first target range to different transparency levels, and / or adjust the image content of the second image within the second target range and the image content outside the second target range to different transparency levels;

[0106] The first and second images are superimposed according to a preset superposition order.

[0107] In some embodiments of this application, at least one of the first target range, the second target range, and the preset overlay order is determined based on at least one of the user instruction, the vehicle's driving status, and the preset display range.

[0108] Taking the second target range as the preset display range as an example, the transparency of the image content of the second image within the preset display range can be adjusted to 1 so that the image content within the preset display range is fully displayed; and the transparency of the image content of the second image outside the preset display range can be adjusted to 0 so that the image content outside the preset display range is completely transparent; finally, the second image is superimposed on the first image to obtain a fused image, thereby enabling the fused image to include the image content of the first image outside the preset display range, as well as the image content of the second image within the preset display range.

[0109] In some embodiments of this application, image fusion is achieved by adjusting transparency in combination with image overlay. By introducing a clear range division (such as a first target range and a second target range), the degree of influence between different images within the corresponding range can be effectively controlled. For example, in the above example, interference of AVM images outside the preset display range on SR (and maps, such as SD MAP) images can be effectively avoided.

[0110] In some embodiments of this application, the first image and / or the second image are fully distorted before being superimposed, in order to ensure the clarity of the image (such as the image rendered after a viewpoint switch) and avoid image distortion.

[0111] In some embodiments of this application, an image within a preset display range is cropped from the fused image and output to a display device for display.

[0112] In some embodiments of this application, the image content after fusing the first image and the second image within a preset display range further includes at least one of the following:

[0113] Within a preset display range, at least one of the first image content of the first image and the second image content of the second image is used to visualize different driving information; for example, the first image content may be a 3D scene rendered by SR (and map, such as SD Map), and the second image content may be the image content rendered by AVM for driving information (such as static obstacles, moving objects, etc.) that is not perceived by SR (and map, such as SD Map).

[0114] Within the preset display range, either the third image content of the first image or the fourth image content of the second image is used to visualize the same driving information, thereby avoiding repeated visualization of the same driving information in the merged image.

[0115] In some embodiments of this application, at least one of the first image content, the second image content, the third image content, and the fourth image content is flexibly determined according to the vehicle's driving state (which can be used to characterize different scenarios in which the vehicle is located).

[0116] For example, when a vehicle passes a pedestrian crossing, the 3D road scene map (including building models and road models, etc.) rendered by SR (and maps, such as SD Map) can be identified as the first image content, and the pedestrian image rendered by AVM can be identified as the second image content. This allows the fused image to accurately describe moving pedestrians on the road based on AVM, thereby improving driving safety.

[0117] For example, when a vehicle is driving normally, the image content rendered by SR (and maps, such as SD Map) for buildings and roads can be identified as third image content, so that the fused image can provide a more comprehensive description of the buildings and roads around the vehicle based on SR (and maps, such as SD Map), thereby improving the driving experience.

[0118] For example, when a vehicle is in a parking scenario, the image content rendered by AVM for the ground can be determined as the fourth image content, so that the fused image can accurately describe the parking lot ground based on AVM, thereby improving the parking experience.

[0119] In some embodiments of this application, in response to detecting that the vehicle's driving state conforms to a first preset state (e.g., parking state, low-speed driving state, or detection of a target object within a preset range of the vehicle), the fused image includes second image content. The second image content includes obstacles visualized in the second image within a preset display range. These obstacles are those not visualized or perceived by the SR (due to the lack of a pre-set model, complex environment, etc.), ensuring an accurate description of obstacles surrounding the vehicle. In some embodiments of this application, the target object includes at least one of static obstacles, moving pedestrians, and low-lying objects.

[0120] In some embodiments of this application, in response to detecting that the vehicle's driving state conforms to a second preset state (such as the vehicle passing through a turning intersection or pedestrian crossing), the fused image includes fourth image content, which includes moving objects visualized in the second image within a preset display range, so as to achieve enhanced feedback on moving objects around the vehicle.

[0121] In some embodiments of this application, the method further includes:

[0122] Based on the vehicle's driving status (such as road conditions and vehicle speed, which can be determined based on the detection results of various onboard sensors to reflect the vehicle's environment), the preset display range can be adjusted, such as adjusting the shape, size, and angle range of the preset display range; or,

[0123] Based on the vehicle's driving status, target information is sent to the vehicle's SR and AVM. The target information indicates at least one of the following: the required image dimension to be rendered, the lens information for image rendering, and the viewpoint information for image rendering.

[0124] The image dimension can be two-dimensional or three-dimensional, the lens information can include the lens position, the viewpoint information can include the viewpoint direction (such as top view, left / right view, front view and default view), the viewpoint size (such as the magnification or reduction factor required for the viewpoint, i.e. the distance required to zoom in or out), the position of the viewpoint and the position to be focused on.

[0125] For example, when it is necessary to switch the fused image displayed by the vehicle to a normal view (i.e., the default view), after sending the corresponding target information to the SR and AVM, the system receives a first image rendered by the SR (and the map, such as SD Map) based on its own default view, and a second image rendered by the AVM based on its own default view. During the fusion phase, the original viewpoints of the first and second images are maintained, and the transparency of the second image is adjusted and superimposed on the first image to preserve the image content rendered by the AVM within a preset display range. This allows the AVM image to be used to emphasize and provide feedback on driving information within the preset display range.

[0126] When it is necessary to switch the fused image displayed on the vehicle to a top-down view, after sending the corresponding target information to the SR and AVM, the system receives the first image rendered by the SR (and a map, such as an SD Map) based on the top-down view, and the second image (a two-dimensional image) rendered by the AVM based on the top-down view. During the fusion stage, the transparency of the second image can be adjusted and superimposed on the first image to retain the image content rendered by the AVM within a preset display range, so as to use the AVM's two-dimensional real-scene imagery to emphasize and provide feedback on driving information within the preset display range.

[0127] When it's necessary to switch the fused image displayed on the vehicle to a forward-facing fused state, after sending target information to the SR (Signal Renderer) and AVM (Autonomous Rendering Machine), the SR (and the map, such as SD Map) zooms in on its own viewpoint based on the target information, simultaneously and quickly zooms in to the front of the vehicle, and changes its viewpoint to eye level to obtain the SR forward-facing view (i.e., the first image); the AVM renders its forward-facing AVM view based on the target information to obtain the second image. During the fusion phase, after fully correcting the distortion of the forward-facing AVM view, it is then fused with the SR forward-facing view to achieve a reasonable visualization of driving information in the area near the front of the vehicle.

[0128] When it's necessary to switch the fused image displayed by the vehicle to a left-facing view, after sending target information to the SR (Surveillance Renderer) and AVM (Autonomous Rendering Machine), the SR (and the map, such as SD Map) magnifies its own viewpoint based on the target information, simultaneously rotates clockwise to the left side of the vehicle, and changes its viewpoint to eye level, thus obtaining the SR left-facing view (i.e., the first image). The AVM renders the left-facing AVM view based on the target information to obtain the second image. During the fusion stage, after fully correcting the distortion of the left-facing AVM view, it is fused with the SR left-facing view. During the fusion process, the vehicle model is set to be transparent, and ground information is visualized using the SR (and the map, such as SD Map). This allows for a reasonable visualization of driving information in the area near the left side of the vehicle.

[0129] When it's necessary to switch the fused image displayed by the vehicle to a right-facing view, after sending target information to the SR (and map, such as SD Map), the SR (and map, such as SD Map) magnifies its own viewpoint based on the target information, simultaneously rotates counter-clockwise to the right side of the vehicle, and changes its viewpoint to eye level to obtain the SR right-facing view (i.e., the first image); the AVM renders the right-facing AVM view based on the target information to obtain the second image. During the fusion phase, after fully correcting the distortion of the right-facing AVM view, it is fused with the SR right-facing view. During the fusion process, the vehicle model is set to be transparent, and ground information is visualized using the SR (and map, such as SD Map). This allows for a reasonable visualization of driving information in the area near the right side of the vehicle.

[0130] When it is necessary to switch the fused image displayed by the vehicle to a close-up view, after sending target information to the SR (Surveillance Renderer) and AVM (Autonomous View Renderer), the SR (and map, such as SD Map) and AVM zoom in on the viewpoint based on the target information, focusing on the preset display range and rendering a first image and a second image (which are 3D images) that conform to their default viewpoints. During the fusion stage, the transparency of the second image is adjusted and superimposed on the first image to preserve the image content rendered by the AVM within the preset display range, so as to use the AVM's 3D real-world imagery to emphasize and provide feedback on driving information within the preset display range.

[0131] When it's necessary to switch the fused image displayed on the vehicle to a left rear view, after sending target information to the SR (Side Mirror) and AVM (Autonomous View Monitor), the SR (and the map, such as SD Map) zooms out and shrinks the viewpoint based on the target information, simultaneously rotating clockwise by a certain angle (e.g., about 100 degrees), and zooms the camera to the rearview mirror position (the specific coordinates can be flexibly determined) to obtain the SR left-facing view (i.e., the first image). The AVM renders the left-facing AVM view based on the target information to obtain the second image. During the fusion stage, the left-facing AVM view is fully distorted and then fused with the SR left-facing view. An image within a preset display range is cropped from the fused image and displayed through a display device. This allows for a reasonable visualization of driving information in the area near the left rear of the vehicle.

[0132] When it's necessary to switch the fused image displayed on the vehicle to a right-side rear view, after sending target information to the SR (Side Mirror) and AVM (Autonomous View Monitor), the SR (and the map, such as SD Map) zooms out and shrinks the viewpoint based on the target information, simultaneously rotating counterclockwise by a certain angle (e.g., about 100 degrees), and zooms the camera to the rearview mirror position (the specific coordinates can be flexibly determined) to obtain the SR right-side view (i.e., the first image). The AVM renders the right-side AVM view based on the target information to obtain the second image. During the fusion stage, the right-side AVM view is fully distorted and then fused with the SR right-side view. An image within a preset display range is cropped from the fused image and displayed through a display device. This allows for a reasonable visualization of driving information in the area near the right rear of the vehicle.

[0133] In some embodiments of this application, the view fusion scheme used by the vehicle is determined according to the vehicle's driving state, and the view information of the fused image to be displayed by the vehicle is determined accordingly. The view information may include information such as view direction, view size, and view position.

[0134] For example, when the vehicle uses the default view fusion scheme (which is used to obtain the fused image under the default view), the preset display range is a circular area centered on the center point of the vehicle's rear and conforming to a set radius. For example, the fused image under the default view fusion scheme can be seen in Figure 7, and the set radius can range from 3.13 meters to 6 meters.

[0135] For example, when a vehicle uses a top-down view fusion scheme (which is used to obtain a fused image from a top-down perspective), the preset display range is an area of ​​a first predetermined size where the vehicle is located, such as a circular area with a predetermined radius where the vehicle is located. For instance, the fused image under the top-down view fusion scheme can be seen in Figure 8.

[0136] For example, when a vehicle uses a forward-view fusion scheme (which is used to obtain a fused image from a forward view), the preset display range is an area that includes the area in front of the vehicle's front end and conforms to a second preset size. For instance, the fused image under the forward-view fusion scheme can be seen in Figure 9;

[0137] For example, when a vehicle uses a left-view fusion scheme (which is used to obtain a fused image from a left-view perspective), the preset display range is an area that includes the left side of the vehicle and conforms to a third preset size. For instance, the fused image under the left-view fusion scheme can be seen in Figure 10;

[0138] For example, when a vehicle uses a right-view fusion scheme (which is used to obtain a fused image from a right-view perspective), the preset display range is an area that includes the right side of the vehicle and conforms to a fourth preset size. For instance, the fused image under the right-view fusion scheme can be seen in Figure 11;

[0139] For example, if no new moving objects or static obstacles are detected around the vehicle (and the vehicle uses a close-up fusion scheme to obtain a fused image from a close-up perspective), the preset display range is a fan-shaped area centered on the vehicle's center point and conforming to a first angle range. For instance, the fused image under the close-up fusion scheme can be seen in Figure 12.

[0140] In some embodiments of this application, when a new moving object or static obstacle is detected around the vehicle (and the vehicle uses a near-field fusion scheme), the preset display range is a fan-shaped area centered on the vehicle's center point and conforming to a second angle range. The second angle range includes and is greater than the first angle range, thereby increasing the angle range of the fan-shaped area. This provides users with richer visual information when a new moving object or static obstacle appears, making it easier for drivers to avoid the newly appearing moving object or static obstacle, thus improving driving safety. For example, the first angle range can be 120° to 180°, the second angle range can be 120° to 360°, and the radius of the fan-shaped area can range from 3.13 meters to 6 meters.

[0141] In some embodiments of this application, when the vehicle uses a left rear view fusion scheme (which is used to obtain a fused image from the left rear view), the preset display range is an area that includes the left rear region of the vehicle and conforms to a fifth preset size. For example, the fused image under the left rear view fusion scheme can be seen in Figure 13;

[0142] In some embodiments of this application, when the vehicle uses a right rear-view fusion scheme (which is used to obtain a fused image from the right rear view), the preset display range is an area that includes the right rear region of the vehicle and conforms to a sixth preset size. For example, the fused image under the right rear-view fusion scheme can be seen in Figure 14.

[0143] In some embodiments of this application, the first image is the current frame image obtained by SR arranging and rendering a preset 3D model based on perception data and by standard definition map SD MAP arranging and rendering a preset 3D building model based on road surrounding data.

[0144] The pre-set 3D models include at least one of the following: vehicle model, adjacent vehicle models (such as sedans, SUVs, trucks, ambulances, motorcycles, electric vehicles, and bicycles), pedestrian models, road models (such as lanes, lane lines, parking spaces, and lane markings), facility models (such as toll stations, newsstands, trash cans, cones, guardrails, lawns, and trees), and traffic sign models (such as speed limit signs, traffic lights, parking lots, no-parking zones, gas / charging stations, and highway signs). The pre-set 3D building models include at least one of the following: residential building models, office building models, shop building models, city landmark building models, park building models, and bridge building models.

[0145] In practical implementation, SR (Sensitive Surface) performs real-time placement and rendering of the 3D model based on the perceived data; AVM (Advanced Visualization Machine) collects real-time image data of the physical world through cameras (typically capturing any object in the environment surrounding the vehicle), thus effectively supplementing the 3D model pre-set in SR; SD Map (Software Map) performs real-time placement and rendering of the 3D building model based on data from the surrounding roads. By merging and rendering the image content displayed by SR, AVM, and maps (such as SD Map) in real time on the same display device, the complementary perception ranges of the three products enable blind-spot-free environmental perception, making it more intelligent and reducing the need for multiple applications (APPs) to combine information, thus reducing the user's decision-making burden. For example, the perception range distribution and perception distance of SR, AVM, and SD Map are shown in Figures 15 and 16.

[0146] Understandably, under the influence of the trend of one-shot three-dimensional scenes and the combination of virtual and reality, the fusion rendering scheme of SR, AVM and map (such as SD MAP) provided in this application embodiment (also known as IntelliSphere Reality) can effectively enhance the visual impact of the combination of virtual and reality, thereby greatly enhancing product competitiveness.

[0147] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0148] Please refer to Figure 17, which shows an architecture diagram of an image processing system provided in one embodiment of this application. This system is used to implement the method embodiment of this application. For details not disclosed in the system embodiment of this application, please refer to the method embodiment of this application. The system includes an AVM module, an image rendering engine, and a frame buffer object (FBO). The FBO is communicatively connected to both the AVM module and the image rendering engine, wherein:

[0149] The AVM module is configured to render consecutive frames of images based on AVM and transmit the rendered current frame image to the FBO for storage.

[0150] The image rendering engine is configured to render consecutive frame images based on SR, and to fuse the current frame image stored in FBO with the current frame image rendered based on SR to obtain a fused image. The image content of the fused image includes: the image content of the current frame image rendered by the AVM module or the current frame image rendered based on SR outside the preset display range, and the image content of the current frame image rendered by the AVM module and the current frame image rendered based on SR within the preset display range.

[0151] In the technical solution provided in this application embodiment, the image rendering engine retains the image content represented by the current frame image rendered by itself or the AVM module outside the preset display range, and fuses the image content represented by the current frame image rendered by itself and the AVM module within the preset display range. Thus, while utilizing the mutual complementarity of the perception range of SR and AVM to ensure the comprehensiveness of the perceived driving information, it can avoid the repeated visualization of the same driving information perceived by SR and AVM, thereby avoiding redundancy and clutter in the perspective of driving information. This can reduce the risk of key driving information being obscured and significantly reduce the cognitive load of the driver.

[0152] In some embodiments of this application, the image rendering engine can also be configured to render consecutive frame images based on SR and maps (such as SD MAP).

[0153] In some embodiments of this application, the image rendering engine may also be configured to output the blended image to a display device of the vehicle so that the display device can display the blended image to a user.

[0154] In some embodiments of this application, the image rendering engine can also be configured to use the image rendering shader material class to overlay material effects on the current frame image rendered by the AVM module according to the material parameters of the current frame image rendered by itself.

[0155] In some embodiments of this application, the AVM module uses the current frame image it renders (such as multi-camera textures) as the input source for the image rendering ShaderMaterial class. This allows it to dynamically combine the image rendering material parameters (such as reflectivity and transparency) through UV mapping, thereby overlaying image rendering material effects (such as dynamic specular highlights and ambient occlusion) onto the image rendered by the AVM to enhance the 3D realism of the panoramic image. Furthermore, the image rendering engine overlays material effects onto the current frame image rendered by the AVM module based on the material parameters of the current frame image it renders, which can improve the integration of the AVM image and allow it to seamlessly blend with image rendering elements such as 3D vehicle models and HUDs.

[0156] Please refer to Figure 18, which shows the architecture of an image processing system provided in another embodiment of this application. The AVM module includes an AVM algorithm unit and an AVM data source. The AVM data source is communicatively connected to both the AVM algorithm unit and the image rendering engine, wherein:

[0157] The AVM algorithm unit is configured to render consecutive frame images based on AVM and transmit the rendered current frame image to the AVM data source.

[0158] The AVM data source is configured to transmit the current frame image it receives to the image rendering engine.

[0159] In practical implementation, the AVM algorithm unit executes AVM algorithm-related logic using a pre-configured AVM algorithm library (LIB) to generate an AVM video stream (i.e., consecutive frame images). In some embodiments of this application, the AVM algorithm unit also supports users integrating other related algorithms into the AVM algorithm library according to their actual needs.

[0160] In some embodiments of this application, the AVM algorithm unit and the image rendering engine communicate loosely through the AVM data source, thereby ensuring the scalability of the system's interaction.

[0161] In some embodiments of this application, the AVM algorithm unit is obtained by encapsulating the AVM algorithm module as a native plugin in the image rendering engine, so that it shares the same OpenGL rendering context with the image rendering engine.

[0162] In some embodiments of this application, the AVM algorithm module is deeply integrated as a native plugin into the image rendering engine to achieve sharing of OpenGL resources (i.e., OpenGL rendering context, also known as OpenGL context or OpenGL Context) between the AVM algorithm module and the image rendering engine. This eliminates the overhead of multiple context switching and improves the execution efficiency of the graphics pipeline. Tests show that by sharing the OpenGL context, GPU memory copy overhead can be reduced by more than 30%, and the frame rate can be increased to over 60 frames per second (FPS).

[0163] In some embodiments of this application, the AVM data source is established based on the data source (DataSource) data-driven architecture of the image rendering engine;

[0164] The AVM data source can also be configured to provide a dynamic view control interface to the user, and upon receiving a view switching command from the user through the dynamic view control interface, send target information to the AVM algorithm unit and the image rendering engine. The target information indicates at least one of the following: the required image dimension to be rendered, the lens information for image rendering, and the view information for image rendering.

[0165] In some embodiments of this application, the AVM data source is constructed using the data-driven architecture unique to the image rendering engine. This enables the AVM data source to provide a dynamic view control interface to the user to receive view switching commands from the user. In turn, it supports the user to control the vehicle to switch views through UI (User Interface) events (such as touch, voice commands, etc.) so as to achieve a smooth view transition of the fused image visually.

[0166] In some embodiments of this application, the image rendering engine includes an Output Texture module, an OpenGL rendering context, and a Swapbuffer, wherein:

[0167] The output texture module is configured to render consecutive frame images based on SR, and to fuse the current frame image rendered by the AVM module with the current frame image rendered based on SR to obtain a fused image, and to transmit the fused image to a memory block.

[0168] The memory block is configured to submit the fused image it receives to the vehicle's display device for display;

[0169] The OpenGL rendering context is configured to support rendering operations of the AVM module and the output texture module.

[0170] In practice, the output texture module uses pre-configured SR (and map, such as SD MAP) related plugins to render images related to 2D or 3D radar perception data, 2D or 3D vehicle models, and UI interfaces.

[0171] In some embodiments of this application, the image rendering engine and the AVM module can also be configured to render their current frame images sequentially according to a set rendering order to share the same rendering pipeline. This staged rendering approach avoids pipeline conflicts, ensuring that the shader, material, and lighting calculations of the image rendering engine and the AVM module are not interfered with. Furthermore, by synchronously processing AVM data using the SR rendering pipeline, the need for independent applications can be reduced, resulting in a 8-12% reduction in development time and costs.

[0172] In some embodiments of this application, the image rendering engine can also be configured to transmit the current frame image rendered by itself to the FBO for storage, and then fuse the different current frame images stored in the FBO (i.e. the current frame images rendered by the image rendering engine and the AVM module respectively) to obtain a fused image.

[0173] In some embodiments of this application, the image rendering engine may also be configured to render the current frame image in another pre-created FBO (i.e., another FBO different from the FBO used to store the current frame image), and to perform a frame clearing operation to clear the other FBO before rendering the current frame image.

[0174] In some embodiments of this application, the AVM module may also be configured to render the current frame image in another FBO, and to perform a clear frame operation to clear the other FBO before rendering the current frame image itself.

[0175] Taking a rendering order of first rendering the SR (Screen Representation) frame and then the AVM (Aspect-to-Virtual) frame as an example, please refer to Figure 19, which shows a real-time rendering timing diagram of a system provided in an embodiment of this application. The processing flow for consecutive frames may include: 1. Clear Screen; 2. SR frame rendering; 3. Buffering to FBO (Frame-Based Object); 4. Clear Screen; 5. AVM frame rendering; 6. Blending rendering; 7. Output frame (i.e., submitting the blended frame). The processing flow for each single frame (e.g., the current frame image) is as follows:

[0176] 1) Independent rendering stage of image rendering engine

[0177] The image rendering engine performs a frame clearing operation to clear the first frame blank (FBO).

[0178] The image rendering engine completes the drawing of 2D / 3D images in the first FBO according to its standard rendering pipeline to achieve SR image rendering, ensuring the correctness of its rendering logic such as materials, lighting, and shaders.

[0179] When the image rendering engine finishes rendering, the rendered current frame image is stored in the second FBO for subsequent fusion.

[0180] 2) AVM rendering stage

[0181] The AVM module performs a clear frame operation to clear the first FBO.

[0182] The AVM module executes the AVM algorithm rendering process in the first FBO to generate a panoramic view.

[0183] 3) Image blending and alignment

[0184] The image rendering engine fuses the panoramic view in the first FBO with the current frame image stored in the second FBO (i.e., the current frame image rendered by the image rendering engine) to obtain a fused image.

[0185] During the blending process, the image rendering engine can use additive blending mode, as well as independent control of color channels and alpha channels, to ensure the correct overlay of different images, thereby achieving a precise blending effect.

[0186] For example, first define an addition: chroma value: SR color value and AVM color value; transparency: SR transparency + AVM transparency. Then, define a multiplication of the color itself (overlaying the effect of the previous step): chroma value: C out =C src +(1-α src )*C dst Transparency: α out =α src +(1-α src )*α dst , where C out This represents the output result of the facial reflection value, α. out Indicates the transparency output, C src Represents the SR color value, C dst Represents the AVM color value, α src Indicates SR transparency, αd stThe blending effect of AVM transparency, opacity, and semi-transparency is shown in Figures 20 and 21.

[0187] 4) Final Output

[0188] After fusion is complete, the image rendering engine submits the fused image to the display device to form a complete AVM+UI interactive interface.

[0189] Based on the above embodiments, this application provides an image processing system that integrates an image rendering engine and an AVM module to achieve fused rendering of AVM, SR (and maps, such as SD MAP), and supports functions such as multi-view switching, dynamic material fusion, and OpenGL resource sharing. Furthermore, through frame buffer object (FBO) management and multi-stage rendering, it can ensure the correct overlay display of images rendered by SR (and maps, such as SD MAP) and images rendered by AVM.

[0190] Please refer to Figure 22, which shows an architecture diagram of an image processing system provided in one embodiment of this application. This system is used to implement the method embodiment of this application. For details not disclosed in the system embodiment of this application, please refer to the method embodiment of this application. The system includes an AVM algorithm unit, an AVM data source, and an image rendering engine. The AVM data source is communicatively connected to both the AVM algorithm unit and the image rendering engine, wherein:

[0191] The image rendering engine is configured to render consecutive frames of images based on SR;

[0192] The AVM algorithm unit is configured to render consecutive frame images based on AVM and send the rendered current frame image to the AVM data source.

[0193] The AVM data source is configured to send the current frame image it receives to the image rendering engine;

[0194] The image rendering engine is also configured to perform the following steps upon receiving the current frame image from the AVM data source:

[0195] The third and fourth images are merged to obtain a merged image;

[0196] The merged image is output to the vehicle's display device;

[0197] The third image is the current frame image received by the image rendering engine, the fourth image is the current frame image rendered by the image rendering engine, and the image content of the fused image includes: the image content of the third image or the fourth image outside the preset display range, and the image content of the third image and the fourth image after fusion within the preset display range.

[0198] In the technical solution provided in this application embodiment, the AVM algorithm unit and the image rendering engine achieve loosely coupled communication through the AVM data source, thereby ensuring the interactive scalability of the system. The image rendering engine retains the image content represented by the third or fourth image outside the preset display range and fuses the image content represented by the third and fourth images within the preset display range. Thus, while utilizing the mutual complementarity of SR and AVM in the perception range to ensure the comprehensiveness of the perceived driving information, it can avoid the repeated visualization of the same driving information perceived by SR and AVM, thereby avoiding redundancy and clutter in the perspective of driving information. This can reduce the risk of key driving information being obscured and significantly reduce the cognitive load on the driver.

[0199] In some embodiments of this application, the AVM algorithm unit is also configured to render consecutive frame images based on AVM and a map (such as SD MAP).

[0200] In some embodiments of this application, the image rendering engine is also configured to use the image rendering shader material class to overlay material effects on the current frame image rendered by the AVM algorithm unit according to the material parameters of the current frame image rendered by itself.

[0201] In some embodiments of this application, AVM uses the current frame image (such as multi-camera texture) rendered by itself as the input source of the image rendering ShaderMaterial class. Thus, it can dynamically combine it with image rendering material parameters (such as reflectivity and transparency) through UV mapping, thereby superimposing image rendering material effects (such as dynamic specular highlights, ambient occlusion, etc.) onto the image rendered by AVM to enhance the three-dimensional realism of the panoramic image. Furthermore, the image rendering engine superimposes material effects onto the current frame image rendered by the AVM algorithm unit based on the material parameters of the current frame image rendered by itself, which can improve the integration of the AVM image and make it seamlessly integrated with image rendering elements such as three-dimensional vehicle models and HUDs.

[0202] In some embodiments of this application, the AVM algorithm unit is obtained by encapsulating the AVM algorithm module as a native plugin in the image rendering engine, so that it shares the same OpenGL rendering context with the image rendering engine.

[0203] In some embodiments of this application, the image rendering engine further includes an Output Texture module, an OpenGL rendering context, and a Swapbuffer, wherein:

[0204] The output texture module is configured to render consecutive frame images based on SR, and, upon receiving the current frame image from the AVM data source, generate a blended image and send the blended image to a memory block;

[0205] The memory block is configured to submit the fused image it receives to the vehicle's display device for display;

[0206] The OpenGL rendering context is configured to support rendering operations of the AVM algorithm unit and the output texture module.

[0207] In practical implementation, the AVM algorithm unit executes AVM algorithm-related logic using a pre-configured AVM algorithm library (LIB) to generate an AVM video stream (i.e., consecutive frame images). In some embodiments of this application, the AVM algorithm unit also supports users integrating other related algorithms into the AVM algorithm library according to actual needs.

[0208] The output texture module uses pre-configured SR (and map, such as SD MAP) related plugins to render images related to 2D or 3D radar perception data, 2D or 3D vehicle models, and UI interfaces.

[0209] In some embodiments of this application, the AVM algorithm module is deeply integrated as a native plugin into the image rendering engine to achieve sharing of OpenGL resources (i.e., OpenGL rendering context, also known as OpenGL context or OpenGL Context) between the AVM algorithm module and the image rendering engine. This eliminates the overhead of multiple context switching and improves the execution efficiency of the graphics pipeline. Tests show that by sharing the OpenGL context, GPU memory copy overhead can be reduced by more than 30%, and the frame rate can be increased to over 60 frames per second (FPS).

[0210] In some embodiments of this application, the image rendering engine (or output texture module) and the AVM algorithm unit are also configured to render their current frame images sequentially according to a set rendering order to share the same rendering pipeline. This avoids pipeline conflicts through staged rendering, ensures that the shaders, materials, and lighting calculations of the output texture module and the AVM algorithm unit are not interfered with, and utilizes the rendering pipeline of SR to synchronously process AVM data, which can reduce the need for independent apps and reduce development time costs by 8%-12%.

[0211] In some embodiments of this application, the AVM data source is established based on the DataSource data-driven architecture of the image rendering engine;

[0212] The AVM data source is also configured to provide a dynamic view control interface to the user, and upon receiving a view switching command from the user through the dynamic view control interface, send target information to the output texture module and the AVM algorithm unit. The target information indicates at least one of the following: the image dimension to be rendered, the lens information for image rendering, and the view information for image rendering.

[0213] In some embodiments of this application, an AVM data source is constructed using a data-driven architecture specific to the image rendering engine, enabling the AVM data source to provide a dynamic view control interface to the user to receive view switching commands from the user, thereby supporting the user to trigger the switching of the view fusion scheme currently used by the vehicle through UI events (such as touch, voice commands, etc.).

[0214] For example, the AVM data source can synchronize the view parameters corresponding to the view switching command in real time, such as the view to be switched to (e.g., the default view, the top view, or the relevant view in the 3D surround view) or the magnification or reduction factor of the view to be switched, to the AVM algorithm unit and the output texture module, so as to indicate the view fusion scheme to be switched, thereby achieving a smooth view transition of the fused image visually.

[0215] In some embodiments of this application, the image rendering engine (or output texture module) is also configured to clear a pre-created first frame buffer object before each rendering of the current frame image, so as to use the first frame buffer object to cache the current frame image rendered by itself.

[0216] In some embodiments of this application, the AVM algorithm unit is further configured to clear a pre-created second frame buffer object before each rendering of the current frame image, so as to use the second frame buffer object to cache the current frame image rendered by itself.

[0217] In some embodiments of this application, the image rendering engine (or output texture module) and the AVM algorithm unit are further configured such that the one that completes the rendering of the current frame image first triggers the clear frame operation of the display device, which is used to delete the currently displayed image of the display device.

[0218] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0219] Please refer to Figure 23, which shows a block diagram of an image processing apparatus provided in one embodiment of this application. This apparatus has the functionality to implement the method example described above; the functionality can be implemented in hardware or by hardware executing corresponding software. The apparatus 220 may include:

[0220] The communication module 221 is used to acquire a first image and a second image. The first image is a current frame image rendered using the vehicle's surround view perception system (SR), and the second image is a current frame image rendered using the vehicle's panoramic surround view monitoring system (AVM).

[0221] The processing module 222 is used to fuse the first image and the second image to obtain a fused image. The image content of the fused image includes: the image content of the first image or the second image outside the preset display range, and the image content of the first image and the second image after fusion within the preset display range.

[0222] In summary, the technical solution provided in this application retains the image content represented by the first image or the second image outside the preset display range, and fuses the image content represented by the first image and the second image within the preset display range. Thus, while utilizing the mutual complementarity of the perception range of SR and AVM to ensure the comprehensiveness of the perceived driving information, it can avoid the repeated visualization of the same driving information perceived by SR and AVM, thereby avoiding redundancy and clutter in the perspective of driving information. This can reduce the risk of key driving information being obscured and significantly reduce the cognitive load on the driver.

[0223] In some embodiments of this application, the processing module 222 is further configured to perform the following steps:

[0224] Check if the vehicle meets the preset activation conditions;

[0225] When a vehicle is detected to meet preset activation conditions, a fused image is displayed on the vehicle's display device;

[0226] If the vehicle does not meet the preset activation conditions, a first image or a second image is displayed on the vehicle's display device.

[0227] In some embodiments of this application, the processing module 222 is further configured to perform the following steps:

[0228] Adjust the image content of the first image within the first target range and the image content outside the first target range to different transparency levels, and / or adjust the image content of the second image within the second target range and the image content outside the second target range to different transparency levels;

[0229] The first and second images are superimposed according to a preset superposition order.

[0230] In some embodiments of this application, the image content after fusing the first image and the second image within a preset display range includes at least one of the following:

[0231] Regarding the first driving information, the first image contains the image content visualized within a preset display range; and regarding the second driving information, the second image contains the image content visualized within a preset display range. The second driving information is different from the first driving information.

[0232] For the same driving information, the first or second image displays the image content within a preset display range.

[0233] In some embodiments of this application, the processing module 222 is further configured to adjust the preset display range according to the vehicle's driving status.

[0234] In some embodiments of this application, the processing module 222 is further configured to send target information to the vehicle's SR and AVM according to the vehicle's driving state. The target information is used to indicate at least one of the following: the required image dimension to be rendered, the lens information for image rendering, and the viewpoint information for image rendering.

[0235] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0236] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the image processing method disclosed in this application.

[0237] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the image processing method disclosed in this application.

[0238] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the image processing method disclosed in this application.

[0239] This application also provides a vehicle that includes electronic equipment or an image processing system.

[0240] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0241] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0242] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, systems, devices, storage media, and program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal equipment to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal equipment, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0243] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0244] These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable terminal equipment, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0245] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0246] The above provides a detailed description of an image processing method, system, apparatus, medium, device, and vehicle provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, wherein, The method includes: Acquire a first image and a second image, wherein the first image is a current frame image rendered using the vehicle's Surround Display Perception System (SR), and the second image is a current frame image rendered using the vehicle's All-Around View Monitoring System (AVM). The first image and the second image are merged to obtain a merged image. The image content of the merged image includes: the image content of the first image or the second image outside the preset display range, and the image content of the first image and the second image after merging within the preset display range.

2. The method according to claim 1, wherein, The method further includes: Detect whether the vehicle meets the preset activation conditions; When the vehicle is detected to meet the preset activation conditions, the fused image is displayed through the vehicle's display device; If the vehicle is detected to not meet the preset activation conditions, the first image or the second image is displayed on the vehicle's display device.

3. The method according to claim 1, wherein, The step of fusing the first image and the second image includes: Adjust the image content of the first image within the first target range and the image content outside the first target range to different transparency levels, and / or adjust the image content of the second image within the second target range and the image content outside the second target range to different transparency levels; The first image and the second image are superimposed according to a preset superposition order.

4. The method according to any one of claims 1-3, wherein, The image content obtained by fusing the first image and the second image within a preset display range includes at least one of the following: For the first driving information, the image content visualized by the first image within the preset display range; and for the second driving information, the image content visualized by the second image within the preset display range, wherein the second driving information is different from the first driving information; For the same driving information, the first image or the second image is the image content visualized within the preset display range.

5. The method according to claim 4, wherein, The image content after fusing the first image and the second image within a preset display range also includes at least one of the following: Within the preset display range, at least one of the first image content of the first image and the second image content of the second image is used to visually display different driving information; Within the preset display range, either the third image content of the first image or the fourth image content of the second image is used to visually display the same driving information.

6. The method according to any one of claims 1-5, wherein, The method further includes: Adjust the preset display range according to the vehicle's driving status; or, Based on the vehicle's driving status, target information is sent to the vehicle's SR and AVM, the target information indicating at least one of the following: the required image dimension to be rendered, the lens information for image rendering, and the viewpoint information for image rendering.

7. An image processing system, wherein, The system includes an AVM module, an image rendering engine, and a frame buffer object (FBO). The FBO is communicatively connected to both the AVM module and the image rendering engine. The AVM module is configured to render consecutive frame images based on AVM and transmit the rendered current frame image to the FBO for storage; The image rendering engine is configured to render consecutive frame images based on SR, and to fuse the current frame image stored in the FBO with the current frame image rendered based on SR to obtain a fused image; the image content of the fused image includes: the image content of the current frame image rendered by the AVM module or the current frame image rendered based on SR outside the preset display range, and the image content of the current frame image rendered by the AVM module and the current frame image rendered based on SR fused within the preset display range.

8. The system according to claim 7, wherein, The image rendering engine and the AVM module are also configured to render their current frame images sequentially according to a set rendering order to share the same rendering pipeline.

9. An image processing system, wherein, The system includes an AVM algorithm unit, an AVM data source, and an image rendering engine. The AVM data source is communicatively connected to both the AVM algorithm unit and the image rendering engine. The image rendering engine is configured to render consecutive frames of images based on SR; The AVM algorithm unit is configured to render consecutive frame images based on AVM and send the rendered current frame image to the AVM data source. The AVM data source is configured to send the current frame image it receives to the image rendering engine; The image rendering engine is also configured to perform the following steps upon receiving the current frame image from the AVM data source: The third and fourth images are merged to obtain a merged image; The fused image is then output to the vehicle's display device; Wherein, the third image is the current frame image received by the image rendering engine, the fourth image is the current frame image rendered by the image rendering engine, and the image content of the fused image includes: the image content of the third image or the fourth image outside the preset display range, and the image content of the third image and the fourth image after fusion within the preset display range.

10. The system according to claim 9, wherein, The image rendering engine and the AVM algorithm unit are also configured to render their current frame images sequentially according to a set rendering order to share the same rendering pipeline.

11. The system according to claim 9, wherein, The image rendering engine and the AVM algorithm unit are further configured such that the one that completes the rendering of the current frame image first triggers the clear frame operation of the display device, which is used to delete the current display image of the display device.

12. The system according to any one of claims 9-11, wherein, The image rendering engine and / or the AVM algorithm unit are further configured to clear a pre-created frame buffer object before each rendering of the current frame image, so as to use the frame buffer object to cache the current frame image rendered by itself.

13. An image processing apparatus, wherein, The device includes: The communication module is used to acquire a first image and a second image, wherein the first image is a current frame image rendered using the vehicle's Surround View Perception System (SR), and the second image is a current frame image rendered using the vehicle's Panoramic Surround View Monitoring System (AVM). The processing module is used to fuse the first image and the second image to obtain a fused image. The image content of the fused image includes: the image content of the first image or the second image outside a preset display range, and the image content of the first image and the second image after fusion within the preset display range.

14. A computer-readable storage medium, wherein, The storage medium stores at least one instruction, which is executed by a processor to implement the image processing method according to any one of claims 1-6.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the program, it implements the image processing method as described in any one of claims 1 to 6.

16. A vehicle, wherein, The vehicle includes the electronic device as claimed in claim 15, or the vehicle includes the image processing system as claimed in any one of claims 7-12.