Image processing method and apparatus, around view system and vehicle

Abstract

The present application relates to an image processing method and apparatus, an around view system and a vehicle. The image processing method comprises: acquiring a plurality of images to be processed; determining fusion regions of each of said images, and detecting regional pixel points in the fusion regions; calculating a regional equalization coefficient of the fusion regions in said images on each side on the basis of the regional pixel points, and calculating a transition equalization coefficient of each of said images on the basis of the regional equalization coefficient and position information of the pixel points in said images; and, on the basis of the transition equalization coefficient, stitching said images to generate a target panoramic image. The present application solves the problem of low quality of panoramic images generated by stitching.

Description

Image processing methods, devices, surround view systems, and vehicles Cross-references to related applications

[0001] This application claims priority to Chinese Patent Application No. 202411955256.8, filed with the Chinese Patent Office on December 27, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to, but is not limited to, the field of image processing, and in particular to, but is not limited to, image processing methods, apparatus, surround view systems, and vehicles. Background Technology

[0003] With the development of technology, in-vehicle surround view systems are gradually being integrated into more and more vehicle models. In-vehicle surround view is a common driver assistance technology. Cameras are installed at the front, rear, and side mirrors of the vehicle, and image processing is used to obtain a panoramic image of the target area. This provides the driver with a rich perspective of the surrounding environment, thereby improving driving safety. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] This application provides an image processing method, apparatus, surround view system, and vehicle.

[0006] In a first aspect, embodiments of this application provide an image processing method, the method comprising:

[0007] Acquire multiple images simultaneously captured by the multi-channel surround view image acquisition device in the vehicle surround view system;

[0008] Determine the fusion region of each image in the multiple images, and detect the regional pixels within the fusion region of each image in the multiple images;

[0009] For each of the multiple images, based on the regional pixels within the fusion region of the image, the region equalization coefficient of the fusion region of the image is calculated, and based on the region equalization coefficient and the position information of the pixels in the image, the transition equalization coefficient of the image is calculated.

[0010] Based on the transition equalization coefficient, the multiple images are stitched together to generate a target panoramic image.

[0011] In some embodiments, calculating the region equalization coefficient of the image's fusion region based on the region pixels within the fusion region includes:

[0012] From the plurality of images, determine the adjacent first and second images that include the fusion region;

[0013] Calculate the first average pixel value of the first pixel in the fusion region of the first image, and calculate the second average pixel value of the second pixel in the fusion region of the second image;

[0014] Calculate the total pixel value based on the first average pixel value and the second average pixel value;

[0015] Based on the first average pixel value and the total pixel value, calculate the region equalization coefficient corresponding to the first image, and based on the second average pixel value and the total pixel value, calculate the region equalization coefficient corresponding to the second image.

[0016] In some embodiments, calculating a first average pixel value of a first pixel in the fusion region of the first image and calculating a second average pixel value of a second pixel in the fusion region of the second image includes:

[0017] Obtain the chromaticity difference information between the first image and the second image;

[0018] Based on the chromaticity difference information, a first weight value is assigned to the first pixel; the pixel value of the first pixel is weighted and calculated based on the first weight value to obtain the first average pixel value.

[0019] Based on the chromaticity difference information, a second weight value is assigned to the second pixel; the pixel value of the second pixel is weighted and calculated based on the second weight value to obtain the second average pixel value.

[0020] In some embodiments, calculating the region equalization coefficient of the image's fusion region based on the region pixels within the fusion region includes:

[0021] Based on the pixels in the region, calculate the equalization coefficient of the current frame corresponding to the image;

[0022] The regional equalization coefficient is calculated based on the current frame equalization coefficient and the historical frame equalization coefficient.

[0023] In some embodiments, calculating the region equalization coefficient of the image's fusion region based on the region pixels within the fusion region includes:

[0024] Convert the image to a target color space where the luminance and chrominance channels are separated.

[0025] Calculate the region equalization coefficients of the luminance channel and / or the chrominance channel of the image in the target color space.

[0026] In some embodiments, the method further includes:

[0027] Detect the color difference information of the image;

[0028] If the color difference information is detected to be greater than a preset color difference threshold, the regional equalization coefficient of the image in the luminance channel and the regional equalization coefficient of the image in the chroma channel are calculated respectively.

[0029] In some embodiments, for each of the plurality of images, the fusion region of the image includes a first side region and a second side region; calculating the transition equalization coefficient of the pixels in the image based on the region equalization coefficient and the position information of the pixels in the image includes:

[0030] Calculate the coefficient comparison results between the regional equilibrium coefficient of the first side region and the regional equilibrium coefficient of the second side region;

[0031] Based on the coefficient comparison results, a corresponding transition equalization coefficient is assigned to each pixel in the image according to the position information of each pixel in the image.

[0032] In some embodiments, the step of assigning a corresponding transition equalization coefficient to each pixel in the image based on the coefficient comparison result and according to the position information of each pixel in the image includes:

[0033] Based on the position information of each pixel in the image, the first side pixel and the second side pixel in the image are determined, and the position comparison result between the first side pixel and the second side pixel is calculated;

[0034] Calculate the unit allocation coefficient based on the coefficient comparison results and the position comparison results;

[0035] Starting from the regional equalization coefficient of the first side region, the unit allocation coefficient is uniformly increased or decreased and allocated to each pixel according to the position difference between each pixel and the first side pixel to obtain the transition equalization coefficient.

[0036] In some embodiments, the step of stitching together the multiple images based on the transition equalization coefficient to generate a target panoramic image includes:

[0037] Obtain a preset mesh texture image;

[0038] Based on the position information of each pixel in the multiple images, the transition equalization coefficient is stored in the mesh texture image;

[0039] Using the mesh texture image, equalization adjustment parameters are calculated for the multiple images, and the multiple images are stitched together based on the equalization adjustment parameters to generate the target panoramic image.

[0040] In some embodiments, determining the fusion region of each image in the plurality of images includes:

[0041] A virtual panoramic image is determined based on the multiple images; the fusion region of each image in the multiple images is calculated in the virtual panoramic image.

[0042] In some embodiments, detecting region pixels within the fusion region of each of the plurality of images includes:

[0043] Based on a preset shrinkage coefficient, the virtual panoramic image is shrunk to obtain the pixel sampling area.

[0044] The pixels located within the fusion region in the pixel sampling area are detected to obtain the region pixels.

[0045] In some embodiments, detecting pixels located within the fusion region in the pixel sampling region to obtain the region pixels includes:

[0046] Detect the initial pixel in the pixel sampling area that is located within the fusion area;

[0047] Based on a preset sampling step size, the initial pixel points are sampled to obtain the region pixel points.

[0048] Secondly, embodiments of this application provide an image processing apparatus, including:

[0049] The acquisition module is used to acquire images collected simultaneously by the multi-channel surround view image acquisition device in the vehicle surround view system.

[0050] The fusion region sampling module is used to determine the fusion region of each image in the multiple images and to detect the region pixels within the fusion region of each image in the multiple images.

[0051] The equalization coefficient calculation module is used to calculate the region equalization coefficient of the fusion region of each of the multiple images based on the region pixels in the fusion region of the image, and to calculate the transition equalization coefficient of the pixel in the image based on the region equalization coefficient and the position information of the pixel in the image.

[0052] The generation module is used to stitch together the multiple images based on the transition equalization coefficient and generate a target panoramic image.

[0053] Thirdly, embodiments of this application provide a surround view system, including: an image acquisition device and a controller;

[0054] The image acquisition device is used to acquire multiple images to be processed;

[0055] The controller is connected to the image acquisition device and is used to acquire each of the images to be processed and execute the image processing method as described in the first aspect above.

[0056] Fourthly, embodiments of this application provide a vehicle including the surround view system described in the third aspect above.

[0057] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects, and advantages of this application more readily apparent. These other aspects will become clear upon reading and understanding the drawings and detailed description. Attached Figure Description

[0058] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0059] Figure 1 is a hardware structure block diagram of a terminal for an image processing method according to an embodiment of this application.

[0060] Figure 2 is a flowchart of an image processing method according to an embodiment of this application.

[0061] Figure 3 is a flowchart of another image processing method according to an embodiment of this application.

[0062] Figure 4 is a schematic diagram of the overall flow of an image processing method according to an embodiment of this application.

[0063] Figure 5 is a structural block diagram of an image processing apparatus according to an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0065] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. The embodiments described in this application may be combined with other embodiments without conflict.

[0066] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0067] The method embodiments provided in this example can be executed in a terminal, computer, or similar computing device. Taking the operation on a terminal as an example, FIG1 is a hardware structure block diagram of a terminal for an image processing method according to an embodiment of this application. As shown in FIG1, the terminal may include one or more (only one is shown in FIG1) processors 102 (processors 102 may include, but are not limited to, processing devices such as microprocessors (MCUs) or field-programmable gate arrays (FPGAs)) and a memory 104 for storing data. In some embodiments, the terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that the structure shown in FIG1 is only illustrative and does not limit the structure of the terminal. For example, the terminal may also include more or fewer components than shown in FIG1, or have a different configuration than shown in FIG1.

[0068] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the image processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0069] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0070] Some surround-view systems produce panoramic images with obvious stitching marks and clearly visible dividing lines, resulting in inconsistent display effects across different parts of the image. This severely impacts the visual quality of the panoramic image and can also affect the driver's judgment of road conditions. Currently, no effective solution has been proposed to address the low quality of panoramic images generated by stitching in related technologies.

[0071] To improve the quality of the panoramic image generated by stitching, this application provides an image processing method. Figure 2 is a flowchart of an image processing method according to an embodiment of this application. As shown in Figure 2, the process includes the following steps S210 to S240.

[0072] Step S210: Acquire multiple images acquired simultaneously by the multi-channel surround view image acquisition device in the vehicle surround view system.

[0073] In this system, multiple images (i.e., multiple images to be processed) can be acquired separately by multiple image acquisition devices, and each image to be processed typically faces a different direction of the acquisition environment. Taking a surround-view system applied to a vehicle as an example, this surround-view system includes multiple surround-view image acquisition devices deployed around the vehicle, which generally consist of four fisheye cameras. These four fisheye cameras acquire images of the environment around the vehicle from each direction. In practical applications, the fisheye images acquired at the same time are stitched together to form a panoramic image, so as to provide the driver with a comprehensive and accurate view of the road environment around the vehicle. Due to differences in vehicle body shape leading to variations in fisheye camera placement, complex lighting conditions during vehicle operation, and the influence of various camera algorithms, the panoramic image stitching effect in related technologies often exhibits differences in brightness and color, resulting in obvious "boundary lines" in the panoramic image, which seriously affects the visual effect of the panoramic image. Based on this, the embodiments of this application perform equalization processing on the currently acquired images to be processed through subsequent steps.

[0074] It should also be noted that the multi-channel image acquisition device mentioned above, used to acquire each image to be processed, can be applied not only in surround view systems, but also in other application scenarios that require stitching together images from various channels to generate a panoramic image, such as traffic intersections, etc., which will not be elaborated here.

[0075] Step S220: Determine the fusion region of each image in the multiple images, and detect the regional pixels within the fusion region of each image in the multiple images.

[0076] The fusion region specifically refers to the overlapping area between two adjacent images in each image to be processed. In other words, after stitching together the images to be processed, the area in each image that overlaps with other adjacent images is called the fusion region of that image. This fusion region can be calculated based on algorithms such as image features, geometric transformations, or pixel differences. Typically, each image to be processed has a fusion region on each side; for example, there is a fusion region on the left side of the i-th image, which overlaps with a portion of the (i-1)-th image, and another fusion region on the right side of the i-th image, which overlaps with a portion of the (i+1)-th image; where i is a positive integer greater than 1.

[0077] In this step, for each detected fusion region, the pixels in each image to be processed can be traversed to detect whether each pixel is located in the fusion region. Alternatively, a virtual panoramic image can be obtained by stitching together the images to be processed, and the coordinates of all pixels in the virtual panoramic image can be traversed to detect whether the coordinates of each traversed pixel are located in the fusion region, so as to obtain the regional pixels in each fusion region.

[0078] Step S230: For each of the multiple images, calculate the region equalization coefficient of the fusion region of the image based on the region pixels within the fusion region of the image, and calculate the transition equalization coefficient of the pixels in the image based on the region equalization coefficient and the position information of the pixels in the image.

[0079] The region equalization coefficient is a quantitative indicator that evaluates the degree of uniformity of pixel characteristics (such as brightness and contrast) within a specific fusion region of an image to be processed. The calculation process of the region equalization coefficient is explained below. First, the fusion regions of each image to be processed are determined according to the aforementioned steps. For each image, the pixel values ​​of pixels within each fusion region are extracted (pixel values ​​refer to the numerical color / brightness information stored by a single pixel in a specific color mode). Thus, for each fusion region, the pixel values ​​of the pixels within that region, as well as the pixel values ​​of the pixels in the fusion regions of adjacent images that overlap with it after stitching, can be determined. This allows for the determination of statistical measures of the pixels in the fusion regions of these two adjacent images, such as mean, variance, and standard deviation. These statistical measures are used to evaluate the brightness, contrast, and other characteristics of the fusion region of each image. Then, by combining the statistical measures of the two adjacent images, the region equalization coefficient of the fusion region of each image is calculated. This coefficient reflects the overall performance of the fusion region of the image, such as whether the brightness is uniform and whether the contrast is appropriate. The specific calculation method can be set according to actual needs; for example, the ratio of the weighted average to the standard deviation can be used as a measure of the equalization coefficient. By calculating the regional equalization coefficient of the fusion region as described above, the brightness of adjacent images can be taken into account.

[0080] Next, based on the region equalization coefficients of the fusion regions on both sides of each image to be processed, the transition equalization coefficient of each pixel in each image is calculated. The transition equalization coefficient is a quantitative indicator that evaluates the overall performance (such as brightness and contrast) of the image to be processed in the transition region between the two fusion regions, after further considering the pixel position information in the image. The transition region refers to the area between the two fusion regions of the image to be processed. For each pixel in each image to be processed, its position information is obtained, including its coordinates in the image and its position relative to the fusion region (such as its distance from the boundary of the fusion region). A weighting function is set based on the pixel position information. This function should reflect the importance of each pixel in the transition region; for example, pixels closer to the boundary of the fusion region should have a larger weight. Then, combining the region equalization coefficient and the weighting coefficient, the transition equalization coefficient of each image to be processed is calculated. This transition equalization coefficient can be seen as the overall performance of the image in the transition region, considering both the balance of the fusion region and the position information of the pixels. The specific calculation method can be to multiply the regional equalization coefficient by the weight of the corresponding pixel to obtain the transition equalization coefficient of the corresponding pixel, and then to sum the transition equalization coefficients of all pixels by weight to obtain the average transition equalization coefficient of the transition region.

[0081] As can be seen, the transition equalization coefficient of each pixel in the corresponding image to be processed exhibits a preset variation trend along a preset direction (usually the horizontal direction). This preset direction and preset variation area are determined based on the specific calculation method of the transition equalization coefficient. For example, considering that each image to be processed is usually captured by looking around in the horizontal direction, in this embodiment, the horizontal distance of each pixel in each image to be processed from the boundary of the fusion region can be calculated. Through weighted calculation, equalization coefficients are assigned to each pixel according to this distance, resulting in a uniformly increasing trend in the transition equalization coefficient of each pixel in the image to be processed from one side to the other. Alternatively, equalization coefficients can be assigned to each pixel according to the calculated horizontal distance, resulting in a uniformly decreasing trend in the transition equalization coefficient of each pixel in the image to be processed from one side to the other.

[0082] Furthermore, it should be noted that, due to algorithmic limitations, some images outside the fusion region may exist during stitching for each image to be processed; that is, the fusion region may not be at the very edge of the image to be processed. In this case, before stitching multiple images into the target panoramic image, the portions of each image that extend beyond the fusion region can be cropped. For example, for the i-th image, the horizontal coordinate of the leftmost pixel in the left fusion region is xl, and the horizontal coordinate of the rightmost pixel in the right fusion region is xR. However, this image also contains pixels with horizontal coordinates less than xl (i.e., pixels located to the left of the left fusion region) and / or pixels with horizontal coordinates greater than xR (i.e., pixels located to the right of the right fusion region). These pixels are considered to be outside the fusion region. To improve the efficiency and accuracy of image equalization processing, the transition equalization coefficient for these pixels can be uniformly set to 1, meaning that equalization compensation processing is not required for this portion.

[0083] Through the above steps, a region equalization coefficient can be calculated based on the regional pixels within the fusion area, and a transition equalization coefficient can be calculated based on the pixel location information. These coefficients can be used in subsequent image stitching and fusion processing to achieve a smoother, more natural transition effect.

[0084] Step S240: Based on the transition equalization coefficient, stitch together the processed images and generate a target panoramic image.

[0085] Specifically, based on the transition equalization coefficient of each pixel in each image to be processed, and the original pixel value of each pixel in the image to be processed, the compensation difference of each pixel in the image to be processed can be calculated. During the stitching process of each image to be processed, the original pixel value is equalized and compensated according to the compensation difference of each pixel, and finally equalized target panoramic image is generated.

[0086] In one embodiment, the target panoramic image can be displayed to the user via a display screen. Since the balanced target panoramic image can clearly and intuitively display the surrounding environment of the surround view system, the user can operate the relevant equipment correctly based on the target panoramic image. For example, the target panoramic image can be displayed to the driver via the vehicle's display screen, enabling the driver to operate the vehicle equipped with the surround view system correctly.

[0087] In the above image processing method, by calculating the region equalization coefficient of the fusion area on both sides of each image to be processed, and based on the region equalization coefficient and the position information of each pixel in the image to be processed, the transition equalization coefficient of each image to be processed is calculated. This allows the equalization coefficient of each pixel to gradually change in the direction from one side of each image to the other, thereby balancing the brightness value of the image. When the fisheye camera around the vehicle is in a complex scene with large differences in lighting, the display effects such as brightness between two adjacent stitches will transition evenly, and the overall stitching effect will be significantly improved. This effectively solves the problem of low quality of the panoramic image generated by stitching, making the image stitching look more natural.

[0088] In some embodiments, determining the fusion region of each image to be processed may further include the following steps: determining a virtual panoramic image based on each image to be processed, and calculating the fusion region in the virtual panoramic image. Specifically, during the stitching process, after preprocessing and coordinate transformation of each image to be processed, a virtual panoramic pixel sampling region is obtained. This region may not directly correspond to an actual image file, but is composed of transformed pixels. Each pixel in this virtual region can be traversed, and it can be determined whether these pixels are within the fusion region of the two image acquisition devices according to the selected fusion region determination method.

[0089] Through the above embodiments, the panoramic image stitching process and the equalization coefficient calculation process are synchronized and executed separately in separate threads, thereby completing image equalization processing during the stitching process and effectively improving image processing efficiency.

[0090] In some embodiments, the detection of region pixels within the fusion region may further include the following steps:

[0091] Based on a preset shrinkage coefficient, the virtual panoramic image is shrunk to obtain a pixel sampling region; pixels located within the fusion region in the pixel sampling region are detected to obtain the region pixels.

[0092] The aforementioned shrinkage coefficients can be preset according to actual conditions. Specifically, a horizontal shrinkage coefficient Sx and a vertical shrinkage coefficient Sy can be selected, where Sx, Sy ∈ [0,1]. The virtual panoramic image, i.e., the bird's-eye view (BEV) interface, is shrunk inward along the horizontal direction according to the horizontal shrinkage coefficient Sx and inward along the vertical direction according to the vertical shrinkage coefficient Sy. The shrunk area is used as the pixel sampling area. In this embodiment, the pixels within the pixel sampling area are traversed, and it is determined whether each pixel is located in the fusion area.

[0093] By sampling pixels only in the shrunken area, the impact of severe edge distortion in images acquired by image acquisition devices such as fisheye cameras can be effectively reduced, thus improving the accuracy of image processing.

[0094] In some embodiments, obtaining region pixels by detecting pixels located within the fusion region in the above-mentioned pixel sampling area may further include the following steps:

[0095] The initial pixel in the detection pixel sampling area is located within the fusion area; based on the preset sampling step size, the initial pixel is sampled to obtain the region pixel.

[0096] Specifically, after initially detecting whether each pixel in the pixel sampling area is located in the fusion region, a relatively dense set of pixels within the fusion region can be obtained. To improve image processing efficiency, in this embodiment, sparse sampling can be further performed on this pixel set. That is, a suitable fixed sampling step size is selected, and the pixels (xbev, ybev) in the pixel set are traversed at fixed intervals, where xbev∈[Wbev·Sx,Wbev·(1-Sx)], ybev∈[Hbev·Sy,Hbev·(1-Sy)]. If (xbev, ybev) is located in the fusion region BevMixAreaij, then the pixel coordinates of this point on the BEV interface are stored as (xbev,ij,ybev,ij), and the number of all points located in the fusion region BevMixAreaij is Nij. Here, Wbev represents the width of the virtual panoramic image, and Hbev represents the height of the virtual panoramic image. Through the above embodiments, a portion of pixels are selected for processing by sparse point sampling, thereby significantly reducing the amount of computation and correspondingly improving the image processing speed.

[0097] In some embodiments, the calculation of the region equalization coefficient of the fusion region of each image to be processed based on region pixels may further include the following steps:

[0098] Convert the image to be processed to a target color space where the luminance and chrominance channels are separated; calculate the region equalization coefficients of the luminance and / or chrominance channels of the image to be processed in the target color space.

[0099] Common color spaces that separate luminance and chrominance channels include YUV, YCbCr, and Lab. If the image to be processed is not in one of these color spaces, a color space conversion is performed to transform the image from the original color space (e.g., RGB) to the target color space. Taking the YUV color space as an example, this conversion process can be represented by the following formula:

[0100] After converting the image to be processed to the target color space using the above formula, the equalization coefficients for the Y, U, and / or V channels can be calculated separately. Based on these calculated equalization coefficients, brightness and / or color difference can be balanced individually. Compared to calculating the overall color equalization coefficient in other color spaces, this embodiment converts the image to the target color space and calculates the equalization coefficients for each channel. This allows for convenient individual equalization of different channels such as brightness and color difference, making the image processing method more suitable for various application scenarios.

[0101] Taking the calculation of the region equalization coefficient in the Y channel as an example, for a fusion region, a first image and a second image, which include the fusion region, are determined from multiple images. The first average brightness value of the first pixel in the fusion region of the first image is calculated, and the second average brightness value of the second pixel in the fusion region of the second image is calculated. Based on the first and second average brightness values, the total brightness value is calculated. Based on the first and total brightness values, the region equalization coefficient corresponding to the Y channel of the first image is calculated, and based on the second and total brightness values, the region equalization coefficient corresponding to the Y channel of the second image is calculated. The calculation method for the region equalization coefficient in the U or V channels is similar to that in the Y channel and will not be repeated here.

[0102] In some embodiments, calculating a first average brightness value of a first pixel in the fusion region of a first image and calculating a second average brightness value of a second pixel in the fusion region of a second image includes: obtaining chromaticity difference information between the first image and the second image; assigning a first weight value to the first pixel based on the chromaticity difference information; performing a weighted calculation on the brightness value of the first pixel based on the first weight value to obtain a first average brightness value; assigning a second weight value to the second pixel based on the chromaticity difference information; and performing a weighted calculation on the brightness value of the second pixel based on the second weight value to obtain a second average brightness value.

[0103] Compared to related technologies that are usually limited to adjusting only the Y channel in the YUV color space, which often makes it difficult to accurately reproduce color differences under complex lighting conditions, resulting in deviations in the overall color of the image and affecting visual quality, the embodiments of this application, through the above steps, can not only equalize the Y channel, but also extend to the U and V channels, realizing comprehensive optimization of image color information. It can perform gradient adjustments on all three YUV channels, effectively improving the color difference problem caused by uneven brightness under complex lighting conditions.

[0104] In some embodiments, the above image processing method may further include the following steps:

[0105] The process involves detecting color difference information in the image to be processed. A suitable color difference detection algorithm is selected based on the characteristics of the image and the application requirements. Color difference detection algorithms can be based on distance metrics in color space (such as Euclidean distance, Manhattan distance, etc.) or on specific color difference formulas. The color difference detection algorithm is applied to the image to be processed to calculate the color difference information of each pixel or region in the image; this color difference information can be represented as the magnitude or degree of color difference, typically a numerical value or a vector.

[0106] Next, based on application requirements, one or more color difference thresholds are set. These thresholds are used to determine whether the color difference in the image is significant and whether further equalization processing is needed. The calculated color difference information is compared with the preset color difference threshold. If the detected color difference information is greater than the preset threshold, the color difference in the image to be processed is considered significant. In this case, in addition to calculating the region equalization coefficient in the luminance channel to equalize the image brightness, the region equalization coefficient in the chroma channel is also calculated separately. In subsequent steps, the transition equalization coefficient for each channel is calculated separately based on the region equalization coefficients in the luminance and chroma channels, thereby achieving consistent equalization processing of image brightness and color difference. Conversely, if the color difference is not significant, it can be considered that the color difference in the image is not significant, and color difference equalization is not necessary. In this case, only the region equalization coefficient in the luminance channel needs to be calculated, and the image brightness needs to be equalized.

[0107] Through the above embodiments, the color difference information of the image to be processed is calculated to determine whether color difference equalization processing is still needed, thereby further optimizing the equalization processing method for each channel in the target color space and improving processing efficiency.

[0108] In some embodiments, the above-described calculation of the region equalization coefficient of each image to be processed within the fusion region based on region pixels may further include the following steps:

[0109] For the fusion region, corresponding neighboring images are determined from each image to be processed; the neighboring images include a first image and a second image; based on the region pixels (i.e., the pixels in the fusion region of the first image), a first average value of the fusion region of the first image is calculated, and based on the region pixels (i.e., the pixels in the fusion region of the second image), a second average value of the fusion region of the second image is calculated; based on the first average value and the second average value, the total pixel value is calculated; based on the first average value and the total pixel value, the region equalization coefficient corresponding to the first image is calculated, and based on the second average value and the total pixel value, the region equalization coefficient corresponding to the second image is calculated.

[0110] First, from all the images to be processed, we determine which images are adjacent based on the location of the fusion region. Taking a certain fusion region as an example, this fusion region has two corresponding adjacent regions, referred to as the first image and the second image, respectively. That is, the fusion region of the first image and the fusion region of the second image overlap after stitching.

[0111] All pixels belonging to the fusion region are extracted from the first image and the second image respectively. The first average value (i.e., the first average pixel value) of all pixels in the fusion region of the first image and the second average pixel value (i.e., the second average pixel value) of all pixels in the fusion region of the second image are calculated. Then, the first average value and the second average value are added together to obtain the total pixel value; this total pixel value represents the overall brightness or color level of all pixels in the fusion region of the first image and the second image after stitching.

[0112] Next, the region equalization coefficient of the fusion region of the first image is calculated using the ratio between the first average value and the total pixel value; this region equalization coefficient reflects the relative brightness or color contribution of the first image during the fusion process. Similarly, the region equalization coefficient of the fusion region of the second image is calculated using the ratio between the second average value and the total pixel value.

[0113] Taking the calculation of the regional equilibrium coefficient under the Y channel as an example, the calculation process of the above regional equilibrium coefficient can be expressed by the following formula:

[0114] in, This is used to represent the first average value of the i-th image to be processed in the Y channel. Used to represent the second average value of the j-th image to be processed in the Y channel; the i-th image to be processed and the j-th image to be processed are adjacent images. Used to represent the region equalization coefficient of the i-th path of the image to be processed in the Y channel. This is used to represent the region equalization coefficient of the j-th path of the image to be processed in the Y channel; and

[0115] Through the above embodiments, in the actual image stitching process, the brightness or color of two adjacent images in the fusion area can be adjusted by calculating the region equalization coefficient of each adjacent image, so as to ensure a more natural and smooth transition, thereby further improving the image processing quality.

[0116] In some embodiments, the above-described calculation of the first average value of the first image within the fusion region based on region pixels, and the calculation of the second average value of the second image within the fusion region based on region pixels, may further include the following steps:

[0117] The process involves: acquiring chromaticity difference information between a first image and a second image; assigning a first weight value to a first pixel in the first image corresponding to a region pixel (i.e., a pixel in the fusion region of the first image) based on the chromaticity difference information; performing a weighted calculation on the pixel values ​​of the first pixel based on the first weight value to obtain a first average value; assigning a second weight value to a second pixel in the second image corresponding to a region pixel (i.e., a pixel in the fusion region of the second image) based on the chromaticity difference information; and performing a weighted calculation on the pixel values ​​of the second pixel based on the second weight value to obtain a second average value.

[0118] Specifically, firstly, the first pixel corresponding to the region pixel in the first image and the second pixel corresponding to the region pixel in the second image are determined. For example, the region pixel is detected and determined based on the aforementioned virtual panoramic image, and the coordinates of the region pixel within the fusion region can be represented as (x... bev,ij ,y bev,ij According to (x) bev,ij ,y bev,ij The coordinates (x, y) of the first pixel of the i-th path of the image to be processed can be calculated. cam,i～ij ,y cam,i～ij ) and the coordinates (x) of the second pixel of the j-th path image to be processed cam,j～ij ,y cam,j～ij ).

[0119] For each pixel in the first and second images corresponding to the fusion region, their differences in color space are calculated. This can be achieved by calculating the difference in the chromaticity components of the two pixels. Next, for each pixel in the first image corresponding to the fusion region (denoted as the first pixel), a weight value (denoted as the first weight value) is assigned based on its chromaticity difference with the corresponding pixel in the second image. The smaller the chromaticity difference, the closer the two pixels are in color, and therefore a higher weight can be assigned. Similarly, for the corresponding pixel in the second image (denoted as the second pixel), a weight value (denoted as the second weight value) is also assigned based on the chromaticity difference. In the actual image stitching process, these weighted averages can be used to adjust the pixel values ​​in the fusion region of the two images to ensure a more natural and smooth transition. For example, the pixels in the fusion region can be weighted and averaged based on these averages to obtain the final stitching result.

[0120] It should also be noted that a weighting function can be defined to convert chromaticity differences into weight values. This function can be linear, non-linear, or based on a specific rule. For example, an inverse proportional function can be used, such that the smaller the chromaticity difference, the larger the weight; or a threshold function can be used, where the weight is 1 when the chromaticity difference is less than a certain threshold, and 0 (or a small value) otherwise.

[0121] Finally, for each first pixel point in the first image, the pixel value is weighted using its corresponding first weight value. Then, all the weighted pixel values are averaged to obtain the first average value. Similarly, for each second pixel point in the second image, the pixel value is weighted using its corresponding second weight value. Then, all the weighted pixel values are averaged to obtain the second average value.

[0122] Taking the Y channel as an example, the calculation process of the weighted average value of the i-th image to be processed and the j-th image to be processed is shown in the following formula:

[0123] In the above formula, is the value of the Y channel, U channel, and V channel of the i-th fisheye camera at (x cam,i～ij , y cam,i～ij ); is the weighted average value of the Y channel of all points (x ij , y bev,ij , y bev,ij ) in the fusion area BevMixArea mapped to the i-th fisheye camera, ij is the weighted average value of the Y channel of all points (x bev,ij , y bev,ij ) in the fusion area BevMixArea

[0124] mapped to the j-th fisheye camera. ColorDelta is the sum of the color value differences of the U channel and the V channel, and DeltaThreshold is the set color value difference threshold. When ColorDelta < DeltaThreshold, Weight = 0. In addition, the calculation methods of the weighted average values of the U channel and the V channel are the same as those of the Y channel.

[0125] In some embodiments, the above method of calculating the regional equilibrium coefficient of each image to be processed in the fusion area based on regional pixel points may further include the following steps:

[0126] Based on regional pixel points, calculate the current frame equilibrium coefficient corresponding to the image to be processed; according to the current frame equilibrium coefficient and the obtained historical frame equilibrium coefficient, calculate the regional equilibrium coefficient.

[0127] In this embodiment, the region equalization coefficient calculation method yields the equalization coefficient of the current frame's image to be processed. Considering that during the acquisition of a series of frames by the image acquisition device, issues such as brightness jumps in a particular frame may occur, this embodiment, to ensure the accuracy of image processing, also combines the region equalization coefficients of N frames of images to be processed acquired at historical moments for comprehensive calculation; N is a positive integer, and the range of N values ​​can be adjusted according to the application scenario. Based on the current frame's equalization coefficient and the historical frame's equalization coefficient, an average or weighted average calculation can be performed to obtain the final region equalization coefficient.

[0128] The aforementioned historical frame equalization coefficients can be calculated from the historical frames of the image to be processed in the same way as the current frame equalization coefficients. These historical frame equalization coefficients may be stored in a data structure, such as an array, queue, or database. Taking an array as an example, the current frame equalization coefficient of the current frame image to be processed, calculated above, will be stored in the coefficient storage array of historical frame equalization coefficients; when the length of the historical frame equalization coefficient array exceeds a set length, the oldest equalization coefficient is discarded and the latest equalization coefficient is stored. It should also be noted that the region equalization coefficients calculated from the current frame image to be processed, combined with the historical frame equalization coefficients, will not be stored in the coefficient storage array to ensure that the historical frame equalization coefficients are not affected by the output of the brightness adjustment algorithm.

[0129] Through the above embodiments, the region equalization coefficient of the image to be processed is calculated by combining the equalization coefficient of historical frames, thereby avoiding the problem of image processing failure or error caused by sudden changes in brightness of a certain frame due to environmental factors, and effectively improving the accuracy of image processing.

[0130] In some embodiments, the fusion region in each image to be processed includes a first side region and a second side region; the first side region in the image to be processed may be the fusion region on the right side of the image, and the second side region may be the fusion region on the left side; or vice versa.

[0131] The above calculation of the transition equalization coefficient for each image to be processed based on the region equalization coefficient and the position information of pixels in the image to be processed may further include the following steps:

[0132] Calculate the region equalization coefficient of the first side region and compare the coefficients between the region equalization coefficients of the second side region; based on the coefficient comparison results, assign a corresponding transition equalization coefficient to each pixel according to the position information of each pixel in the image to be processed.

[0133] The process involves comparing the difference between the region equalization coefficients of the first and second side regions to determine the inconsistency in brightness or contrast within the fusion regions of the image to be processed. Based on this coefficient comparison and the positional information of each pixel in the image to be processed (such as coordinates and distance from the fusion region boundary), a corresponding transition equalization coefficient is assigned to each pixel. This assignment process reflects the changing trend of pixel values ​​in the transition region and ensures that the transition equalization coefficient exhibits a smooth changing trend in the image.

[0134] The above-mentioned method for assigning equalization coefficients to each pixel can be as follows: The transition equalization coefficient is calculated using linear interpolation based on the distance of each pixel from the boundary of the fusion region. For example, if the equalization coefficient of the first side region is less than that of the second side region, the transition equalization coefficient can be gradually increased from the first side region to the second side region. Alternatively, the transition equalization coefficient can be calculated based on the pixel's position information using a specific nonlinear function (such as an exponential function or a logarithmic function). Another approach is to use weight allocation; that is, a weight function is set, which assigns a weight to each pixel based on its position information (such as distance, coordinates, etc.). Then, the region equalization coefficient is multiplied by the corresponding pixel's weight to obtain the transition equalization coefficient.

[0135] It's important to note that when calculating the transition equalization coefficient, for pixels outside the fusion region (i.e., pixels closer to the image edge than the fusion region), their corresponding transition equalization coefficient can be uniformly set to 1 (or some other appropriate value), meaning that no equalization compensation processing is performed on this part. This can improve the efficiency and accuracy of image processing.

[0136] Through the above embodiments, the transition equalization coefficient of each image to be processed can be calculated based on the regional equalization coefficient and the position information of the pixels. The calculated transition equalization coefficient can be applied to the image to be processed to adjust the brightness or contrast and other characteristics of the image in the transition area, so as to achieve a smoother and more natural transition effect of the panoramic image.

[0137] In some embodiments, the above-mentioned assignment of corresponding transition equalization coefficients to each pixel based on the coefficient comparison results and according to the position information of each pixel in the image to be processed may further include the following steps:

[0138] Based on the positional information of each pixel, the first-side pixels and the second-side pixels in the image to be processed are determined, and the positional comparison result between the first-side pixels and the second-side pixels is calculated. Specifically, determining the first-side pixels and the second-side pixels in the image to be processed is typically achieved by examining the coordinates of the pixels or their relative positions to the boundaries of the fusion region. Generally, the difference in the x-coordinates between the leftmost and rightmost pixels in the image to be processed is calculated as the positional comparison result.

[0139] Based on the coefficient comparison results and the position comparison results, the unit allocation coefficient is calculated; taking the regional equalization coefficient of the first side region as the starting point, the unit allocation coefficient is uniformly increased or decreased and allocated to each pixel according to the position difference between each pixel and the first side pixel, and the transition equalization coefficient is obtained.

[0140] The aforementioned unit allocation coefficient is used to determine the increment or decrement of the transition equalization coefficient allocated to each pixel. The magnitude and direction (incrementing or decreasing) of the unit allocation coefficient depend on the comparison between the region equalization coefficients of the first and second side regions, as well as the positional relationship between the pixel and the first side pixels. Then, starting from the region equalization coefficient of the first side region, the unit allocation coefficient is uniformly incremented or decremented among the pixels according to the positional difference (such as distance, coordinate difference, etc.) between each pixel and the first side pixels. Therefore, each pixel receives a transition equalization coefficient that reflects its equalization within the transition region. Finally, the calculated transition equalization coefficient is applied to the image to be processed to adjust characteristics such as brightness or contrast within the transition region. This can be achieved by multiplying the transition equalization coefficient by the image pixel value (or other appropriate operations).

[0141] To facilitate understanding, the calculation process is explained below with a practical application scenario. The aforementioned transition equalization coefficients can be stored in the mesh texture image. For the i-th path image, the x-coordinates of the leftmost and rightmost pixels in that image can be mapped to the mesh texture image, obtaining their horizontal coordinates on the mesh texture image. That is, the pixel coordinates of the first side and the pixel coordinates of the second side mentioned above.

[0142] Taking the Y channel as an example, the transition equilibrium coefficient The calculation method is shown in the following formula:

[0143] in, The pixel coordinates of the equalization coefficients assigned by the current computation of the image to be processed are mapped to the pixel coordinates under the mesh texture image. For the i-th path of the image to be processed in the fusion region BevMixArea ij The application equalization coefficient of the Y channel, which is also the regional equalization coefficient of the first side region mentioned above; For the i-th path of the image to be processed, in the BevMixArea fusion region on the other side ij′ The application equalization coefficient of the Y channel is also the region equalization coefficient of the second side region mentioned above. j and j′ represent the two image acquisition devices deployed on both sides of the i-th image acquisition device that acquires the i-th image to be processed, namely the j-th image acquisition device and the j′-th image acquisition device.

[0144] Through the above embodiments and steps, the transition equalization coefficient of each pixel in each image to be processed can be calculated more accurately, and the transition equalization coefficient can be applied in subsequent image processing to achieve a smoother and more natural transition effect.

[0145] In some embodiments, the above-mentioned process of stitching together the processed images based on the transition equalization coefficient and generating the target panoramic image may further include the following steps:

[0146] Obtain the preset mesh texture image. Specifically, create n M×N×Channel mesh textures; where the value of n is determined by the number of images to be processed. For example, if there are four images to be processed, then the value of n is set to 4. The values ​​of M and N can be preset based on experience and actual conditions. For example, both M and N can be set to 32, then the mesh texture image consists of a 32×32 mesh. The number of channels is Channel = 3, and the transition equalization coefficients of each channel are stored respectively (since the mesh texture is an image, the transition equalization coefficient can be mapped from [0,2] to [0,255] in the texture).

[0147] Based on the positional information of each pixel in the image to be processed, the transition equalization coefficients are stored in the mesh texture image. It should also be noted that, in order to accurately store the transition equalization coefficients, the coordinates in the image to be processed need to be mapped to the mesh texture image.

[0148] For example, for the i-th path of the image to be processed, the x-coordinates of its leftmost and rightmost sampling points are x and x, respectively. cam,imaxL x cam,imaxR , for x cam,imaxL x cam,imaxR The coordinates are normalized to obtain normalized coordinates. Will Mapped to the horizontal coordinates of an M×N mesh texture The calculation method is as follows:

[0149] Here, `floor()` represents rounding down. Taking the Y channel as an example, iterates through each point of an M×N mesh texture, and the horizontal coordinate of the M×N mesh texture is... or When the transition equalization coefficient of the YUV channel is set to 1, the horizontal coordinate of the M×N mesh texture is... Then the Y channel is in Transition equilibrium coefficient at the point It can be calculated using the formula for the transition equilibrium coefficient mentioned above.

[0150] Finally, using the mesh texture image, equalization adjustment parameters are calculated for the image to be processed, and the images are stitched together based on these parameters to generate the target panoramic image. The equalization adjustment parameter is the difference between the transition equalization coefficient stored in the mesh texture image and the channel values ​​of the image to be processed. During BEV stitching, the same texture coordinates are used to sample the mesh texture when sampling the image to be processed using texture coordinates. Therefore, for each channel of each sampling point in the image to be processed, a corresponding transition equalization coefficient can be found in the mesh texture. The detailed adjustment process is as follows:

[0151] Taking the conversion of the image to be processed to the target color space as an example, the RGB values ​​of the i-th image texture at each texture coordinate are obtained and converted to the YUV color space. The values ​​of the three channels of the M×N mesh texture at the same texture coordinate are obtained respectively, i.e., the transition equalization coefficients of the YUV channels of the M×N mesh texture at the same texture coordinate are obtained. The differences ΔY, ΔU, and ΔV between the YUV channel values ​​at the transition equalization coefficients and the YUV channel values ​​in the original image are calculated. ΔY, ΔU, and ΔV are then converted to the differences ΔR, ΔG, and ΔB in the RGB color space. This conversion process is shown in the following formula:

[0152] The equalized RGB values ​​R of the image are obtained through the above method. dst G dst B dst They are respectively;

[0153] Through the above embodiments, the transition equalization coefficients are stored in the created mesh texture to complete the equalization processing of the image to be processed and the panoramic image stitching, thereby facilitating unified processing when using professional graphics program interfaces such as OpenGL. Compared with storing the transition equalization coefficients in other storage formats such as matrices, which requires converting matrix data into image data during image processing, this embodiment directly stores the transition equalization coefficients in the mesh texture image, eliminating the need for additional data format conversion and improving the convenience and efficiency of image processing.

[0154] The embodiments of this application are described and illustrated below with reference to specific application scenarios. Taking an application in a surround-view system as an example, the surround-view system consists of multiple fisheye cameras arranged around the vehicle. Figure 3 is a flowchart of another image processing method according to an embodiment of this application. As shown in Figure 3, the process includes the following steps S301 to S307.

[0155] Step S301: Initialize parameters, determine the fusion region in the BEV sampling area, and sample the fusion region at fixed intervals.

[0156] This involves obtaining the width and height of the BEV interface, relevant vehicle parameters, and initializing related variables. A horizontal contraction coefficient S is selected. x and the coefficient of contraction in the vertical direction S y The BEV interface is sized according to the horizontal shrinkage coefficient S. x Contracting inward along the horizontal direction, and according to the vertical contraction coefficient S y The contracted area, extending vertically inward, is designated as the BEV pixel sampling region. Then, a fusion region determination method is chosen to define the fusion region. Each pixel within the BEV pixel sampling region is traversed, and it is determined whether each pixel falls within the fusion region of the two fisheye cameras.

[0157] Choose a suitable fixed step size, and traverse the pixels in the BEV pixel sampling area at fixed step intervals. If the pixel is located in the fusion area BevMixAreaij, then store the pixel coordinates of the point on the BEV interface and record the total number of all points located in the fusion area.

[0158] Step S302: Based on the pixel coordinates sampled in the fusion region, the sampling point coordinates of the two fisheye camera image textures corresponding to the fusion region can be calculated.

[0159] Step S303: Based on the pixels of the two fisheye cameras corresponding to the pixels of the BEV fusion region obtained in step S302, calculate the weighted average of the Y channel, U channel, and V channel, and calculate the equalization coefficients of the Y channel, U channel, and V channel of the fusion region based on the weighted average of the Y channel, U channel, and V channel.

[0160] Step S304: The equalization coefficient calculated in step S303 is the equalization coefficient of the current frame. The application equalization coefficient of the current frame is solved by combining the equalization coefficients of the historical frames. Then, the equalization coefficient of the current frame is stored in the equalization coefficient array of the historical frames. The application equalization coefficient of the current frame is not stored in the equalization coefficient array of the historical frames to ensure that the equalization coefficient of the historical frames is not affected by the output result of the brightness adjustment algorithm. In addition, when the length of the equalization coefficient array of the historical frames exceeds the set length, the oldest equalization coefficient is discarded and the latest equalization coefficient is stored.

[0161] Step S305: For any fisheye camera image texture in the horizontal direction, the YUV channel equalization coefficient is set to 1 for points other than the leftmost and rightmost sampling points. For points between the leftmost and rightmost sampling points, the YUV channel equalization coefficient increases or decreases arithmetically from the equalization coefficient corresponding to the leftmost point to the equalization coefficient corresponding to the rightmost point.

[0162] Step S306: Based on the number of fisheye cameras, create a corresponding number of 3-channel M×N mesh textures, where the equalization coefficients for the Y, U, and V channels are stored respectively. Specifically, the transition equalization coefficients obtained in step S305 are stored in the M×N mesh textures. All fisheye camera images are processed in the same manner to obtain the same number of M×N mesh textures as the number of fisheye cameras. These mesh textures are used for color balance in the BEV stitched image.

[0163] In step S307, when performing BEV stitching, the same texture coordinates are used to sample the M×N mesh texture when sampling the fisheye camera image using texture coordinates. Therefore, for each sampling point in the fisheye image, the YUV channel can find the corresponding transition equalization coefficient in the M×N mesh texture, and finally the equalized RGB value is calculated.

[0164] Additionally, please refer to Figure 4, which provides a schematic diagram of the overall workflow of an image processing method. In this overall processing flow, the original images captured by each fisheye camera are input to the brightness and color difference equalization module for processing; the brightness and color difference equalization module performs fusion region sampling; the weighted average of each channel at the sampling points is calculated; a mesh texture is created, the texture content being the equalization coefficients after smooth transition of each channel; and the original fisheye camera image is then equalized using the mesh texture. Furthermore, the BEV / 3D stitching module completes the stitching process of the images from each channel, ultimately generating a panoramic image with brightness and color difference equalization.

[0165] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0166] This embodiment also provides an image processing apparatus for implementing the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0167] Figure 5 is a structural block diagram of an image processing apparatus according to an embodiment of this application. As shown in Figure 5, the apparatus includes: an acquisition module 51, used to acquire multiple images to be processed; a fusion region sampling module 52, used to determine the fusion region of each image to be processed and detect regional pixels within the fusion region; an equalization coefficient calculation module 53, used to calculate the regional equalization coefficient of each image to be processed within the fusion region based on the regional pixels, and to calculate the transition equalization coefficient of each image to be processed based on the regional equalization coefficient and the position information of the pixels in the image to be processed; and a generation module 54, used to stitch the images to be processed based on the transition equalization coefficient and generate a target panoramic image.

[0168] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination. Specific examples in this embodiment can be found in the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0169] This embodiment also provides a surround view system, including: an image acquisition device and a controller; the image acquisition device is used to acquire multiple images to be processed; the controller is connected to the image acquisition device and is used to acquire each image to be processed and execute the image processing method as described in any of the above embodiments.

[0170] This embodiment also provides a vehicle including the surround view system as described in the above embodiments.

[0171] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0172] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0173] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image processing method, comprising: Acquire multiple images simultaneously captured by the multi-channel surround view image acquisition device in the vehicle surround view system; Determine the fusion region of each image in the multiple images, and detect the regional pixels within the fusion region of each image in the multiple images; For each of the multiple images, based on the regional pixels within the fusion region of the image, the region equalization coefficient of the fusion region of the image is calculated, and based on the region equalization coefficient and the position information of the pixels in the image, the transition equalization coefficient of the pixels in the image is calculated. Based on the transition equalization coefficient, the multiple images are stitched together to generate a target panoramic image.

2. The image processing method according to claim 1, wherein, The calculation of the region equalization coefficient of the fusion region based on the region pixels within the fusion region of the image includes: From the plurality of images, determine the adjacent first and second images that include the fusion region; Calculate the first average pixel value of the first pixel in the fusion region of the first image, and calculate the second average pixel value of the second pixel in the fusion region of the second image; Calculate the total pixel value based on the first average pixel value and the second average pixel value; Based on the first average pixel value and the total pixel value, calculate the region equalization coefficient corresponding to the first image, and based on the second average pixel value and the total pixel value, calculate the region equalization coefficient corresponding to the second image.

3. The image processing method according to claim 2, wherein, Calculating the first average pixel value of the first pixel in the fusion region of the first image, and calculating the second average pixel value of the second pixel in the fusion region of the second image, includes: Obtain the chromaticity difference information between the first image and the second image; Based on the chromaticity difference information, a first weight value is assigned to the first pixel; the pixel value of the first pixel is weighted and calculated based on the first weight value to obtain the first average pixel value. Based on the chromaticity difference information, a second weight value is assigned to the second pixel; the pixel value of the second pixel is weighted and calculated based on the second weight value to obtain the second average pixel value.

4. The image processing method according to claim 1, wherein, The calculation of the region equalization coefficient of the fusion region based on the region pixels within the fusion region of the image includes: Based on the pixels in the region, calculate the equalization coefficient of the current frame corresponding to the image; The regional equalization coefficient is calculated based on the current frame equalization coefficient and the historical frame equalization coefficient.

5. The image processing method according to claim 1, wherein, The calculation of the region equalization coefficient of the fusion region based on the region pixels within the fusion region of the image includes: Convert the image to a target color space where the luminance and chrominance channels are separated. Calculate the region equalization coefficients of the luminance channel and / or the chrominance channel of the image in the target color space.

6. The image processing method according to claim 5, further comprising: Detect the color difference information of the image; If the color difference information is detected to be greater than a preset color difference threshold, the regional equalization coefficient of the image in the luminance channel and the regional equalization coefficient of the image in the chroma channel are calculated respectively.

7. The image processing method according to any one of claims 1 to 6, wherein, For each of the multiple images, the fusion region of the image includes a first side region and a second side region; the step of calculating the transition equalization coefficient of the pixels in the image based on the region equalization coefficient and the position information of the pixels in the image includes: Calculate the coefficient comparison results between the regional equilibrium coefficient of the first side region and the regional equilibrium coefficient of the second side region; Based on the coefficient comparison results, a corresponding transition equalization coefficient is assigned to each pixel in the image according to the position information of each pixel in the image.

8. The image processing method according to claim 7, wherein, Based on the coefficient comparison results, the process of assigning corresponding transition equalization coefficients to each pixel in the image according to the position information of each pixel in the image includes: Based on the position information of each pixel in the image, the first side pixel and the second side pixel in the image are determined, and the position comparison result between the first side pixel and the second side pixel is calculated; Calculate the unit allocation coefficient based on the coefficient comparison results and the position comparison results; Starting from the regional equalization coefficient of the first side region, the unit allocation coefficient is uniformly increased or decreased and allocated to each pixel according to the position difference between each pixel and the first side pixel to obtain the transition equalization coefficient.

9. The image processing method according to any one of claims 1 to 8, wherein, The step of stitching together the multiple images based on the transition equalization coefficient to generate a target panoramic image includes: Obtain a preset mesh texture image; Based on the position information of each pixel in the multiple images, the transition equalization coefficient is stored in the mesh texture image; Using the mesh texture image, equalization adjustment parameters are calculated for the multiple images, and the multiple images are stitched together based on the equalization adjustment parameters to generate the target panoramic image.

10. The image processing method according to any one of claims 1 to 9, wherein, Determining the fusion region of each image in the plurality of images includes: A virtual panoramic image is determined based on the multiple images; Calculate the fusion region of each image in the multiple images in the virtual panoramic image.

11. The image processing method according to claim 10, wherein, The detection of region pixels within the fusion region of each of the multiple images includes: Based on a preset shrinkage coefficient, the virtual panoramic image is shrunk to obtain the pixel sampling area. The pixels located within the fusion region in the pixel sampling area are detected to obtain the region pixels.

12. The image processing method according to claim 11, wherein, The step of detecting pixels located within the fusion region in the pixel sampling area to obtain the region pixels includes: Detect the initial pixel in the pixel sampling area that is located within the fusion area; Based on a preset sampling step size, the initial pixel points are sampled to obtain the region pixel points.

13. An image processing apparatus, comprising: The acquisition module is used to acquire multiple images captured simultaneously by the multi-channel surround view image acquisition device in the vehicle surround view system. The fusion region sampling module is used to determine the fusion region of each image in the multiple images and to detect the region pixels within the fusion region of each image in the multiple images. The equalization coefficient calculation module is used to calculate the region equalization coefficient of the fusion region of each of the multiple images based on the region pixels in the fusion region of the image, and to calculate the transition equalization coefficient of the pixel in the image based on the region equalization coefficient and the position information of the pixel in the image. The generation module is used to stitch together the multiple images based on the transition equalization coefficient to generate a target panoramic image.

14. A surround view system, comprising: Image acquisition equipment and controller; The image acquisition device is used to acquire multiple images to be processed; The controller is connected to the image acquisition device and is used to acquire each of the images to be processed and execute the image processing method as described in any one of claims 1 to 12.

15. A vehicle comprising the surround view system as claimed in claim 14.

Images